Developer discovers CI quality gate had silently failed on every PR for weeks
A software developer writing on DEV Community revealed that their CI pipeline's content quality gate had been failing on every pull request for an extended period, without anyone realizing it was broken. The gate, powered by an LLM-based grading job, was rejecting requests before any scoring occurred due to unsupported JSON Schema constraints — such as minimum and maximum on integers — being sent to the Anthropic API. Because the error message only flagged the first failing constraint, a quick fix would have triggered a second 400 error, then a third, masking the full scope of the problem. The developer ultimately audited all schemas against a documented list of rejected keywords, uncovering sixteen violations in a single schema alone. The incident highlighted that popular Python and TypeScript SDKs silently strip unsupported constraints, a safeguard that disappears when developers call the API directly without dependencies.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)
Log in to join the discussion and vote.
Log in