Duplicate Retry Layers in LangGraph Pipeline Caused Silent Cost and Latency Overruns

A LangGraph AI pipeline was silently executing up to seven retries per step despite a configured maximum of three, causing token costs and slow-run durations to exceed projections without triggering any errors. The root cause was two independently written retry mechanisms — one at the step level and one in the orchestrator graph — that had no visibility into each other's activity. The fix involved introducing a shared RetryBudget object passed to every node and the orchestrator alike, making the total and per-step retry counts visible across both layers. Instrumentation revealed that the 'resolve' step was consuming most of the retry budget because an upstream API was returning HTTP 200 responses with malformed JSON for certain inputs, making retries futile. The team ultimately fixed the problem through input normalization rather than retry configuration, and added a 'complete' flag with a list of unfinished steps to the result object so downstream aggregations could handle partial runs correctly.