How a 4TB-to-500GB table join triggered a $4,000/hour cloud bill and outage

A data engineering team incurred roughly $4,000 in a single hour after a production join between a 4TB events table and a 500GB user_metadata table caused out-of-memory failures on both BigQuery and Databricks. The query had completed in 42 seconds on a 10% staging sample but hung for 45 minutes in production, spiking BigQuery slot usage above 10,000 and crashing a Databricks SQL Warehouse with a Java heap error. The root cause was each engine attempting a broadcast join on a table far too large to fit in a single worker's memory, compounded by a Databricks configuration that had raised the auto-broadcast threshold to 1GB from its 10MB default. The team resolved the BigQuery issue by rewriting the query to select specific columns and pre-filter join keys, cutting shuffle volume from 12TB to 800GB. On Databricks, they overrode the optimizer with a MERGE hint to force a Shuffle Sort-Merge Join, which spilled to disk rather than crashing, completing the query in 12 minutes.