How a Silent Databricks Pipeline Burned $2,300 in 11 Days Without a Single Error

A senior engineer deployed a Databricks MERGE pipeline to production after successful testing on a 40 GB sample, unaware the production gold table was 3.8 TB with no Z-ordering or partition pruning on the merge key. The job ran every 30 minutes, completed without errors, and showed a green status in the Workflows UI — yet scanned the entire 3.8 TB table on every execution. Over eleven days, the pipeline quietly accumulated $2,300 in compute costs against an expected spend of roughly $40, going unnoticed until a monthly DBU report was reviewed. The incident highlights a broader challenge in production data engineering, where pipelines can pass every technical check while simultaneously inflating costs, silently dropping records, or producing subtly incorrect results. Unlike traditional software failures, data pipeline issues rarely trigger loud alerts, meaning engineers must proactively monitor cost, data quality, and latency rather than relying on exit codes alone.