Skip to content

Release v1.0.2 (2025-12-01)

Highlights

  • Progress logging helpers (create_progress_tracker, log_progress) are now exported from spark_fuse.utils.progress, making it easy to add elapsed-time bars to notebooks.
  • The data-processing template uses DataFrame.write.change_tracking (with schema evolution) by default and shows Delta history before and after writes for quick debugging.

Upgrade Notes

  • No code changes required for existing pipelines. Notebook users can import the new progress helpers and adopt the refreshed template path (/tmp/spark_fuse/orders_enriched_ct) if desired.

Detailed Changes

Added

  • Shared progress helpers for consistent step/timing logging across notebooks.
  • Pre- and post-write Delta log inspection cells in the template notebook.

Changed

  • Template write target renamed to /tmp/spark_fuse/orders_enriched_ct and now prefers write.change_tracking with schema evolution enabled on Delta targets, falling back to Parquet when Delta is unavailable.