Race Condition Behind TypeError Crashed Three Benchmark Pipelines at 3 AM

A TypeError triggered by a race condition between an asynchronous data fetch and a synchronous pipeline evaluator crashed three benchmark pipelines on a developer eval platform. The root cause was that evaluatePipeline() assumed inputData.metrics would always be populated, but at over 100 requests per second, the async fetchBenchmarkData() often had not resolved in time. Engineers resolved the issue by adding early null-checks, processing metrics in 100-item chunks to prevent out-of-memory crashes, and introducing a semaphore-based worker pool capped at four concurrent workers. Immutable data objects and a three-second fetch timeout were also added to eliminate mutable-state race conditions and hanging requests. The fixes addressed TypeError crashes, memory exhaustion risks, and thread pool starvation with minimal performance overhead.