zhao-dbt-plan fixes silent data corruption in dbt microbatch rolling-window models
A developer has released zhao-dbt-plan, an open-source tool that addresses a gap in dbt's native microbatch incremental strategy, where rolling-window downstream models can silently compute on insufficient data after an upstream backfill. The tool reads dbt's compiled manifest, traverses the DAG, and calculates the correct expanded time window each downstream model actually needs — without executing any dbt commands itself. Users receive a reviewable plan that can then be run through their existing orchestration setup, such as Dagster, Airflow, or raw dbt. The tool also flags models whose expanded windows exceed a configurable threshold, preventing unnoticed over-computation. The developer notes that SQLMesh handles similar cascading restatement natively, but zhao-dbt-plan targets teams already committed to dbt who have no equivalent built-in option.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in