How to Turn AI Judge Findings Into Deterministic Eval Gates

·1 views

A software engineering post on DEV Community argues that teams make a costly mistake by leaving recurring model-as-judge findings unresolved, re-running expensive, non-deterministic checks on every evaluation cycle. The author proposes a 'ratchet' system that classifies evidence into three tiers: unforgeable proof (Tier 1), statistical signals against an independent baseline (Tier 2), and model-as-judge opinions (Tier 3). When a judge repeatedly flags the same class of failure, engineers are advised to identify an equivalent deterministic check and promote it to Tier 1 or Tier 2, making it fast, free, and capable of blocking a pipeline run. For example, a judge complaint about non-existent citations can be replaced with a simple string-membership function that verifies cited section headers exist verbatim in the source document. The core argument is that Tier 3 should serve as a discovery tool for finding gaps in lower-tier checks, not as a permanent, recurring evaluation gate.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Python OOP Day 3: Inheritance, Polymorphism, and Duck Typing Explained

A tutorial series on Python object-oriented programming has reached its third installment, focusing on inheritance and polymorphism. The guide explains three types of inheritance — single, multilevel, and multiple — showing how child classes can reuse and extend parent class behavior. It also covers method overriding and the built-in super() function, which allows child classes to call parent methods without duplicating code. Polymorphism and duck typing are demonstrated through a practical payroll system where different employee types respond to the same method call in their own way. The article includes working code examples and diagrams to illustrate how these concepts apply in real-world Python projects.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

All 236 Tests Passed Locally, Yet the Release Was Broken — Here's Why

A developer maintaining a Claude Code notification tool fixed a timezone bug, saw all 236 local tests pass, but discovered the release was broken only after pushing to CI. The root cause was that both the code and the tests silently assumed the host machine's local timezone — Asia/Hong_Kong — which matched the embedded timezone in the data, making errors invisible locally. When CI ran on a UTC-based Ubuntu runner, the symmetry broke and exposed not just the original bug but two additional hidden flaws in the test logic itself. The tests had effectively been verifying that a value equalled itself, not that the timezone logic was correct, meaning they could never fail regardless of the underlying bug. The incident highlights how a fully green local test suite can be misleading when tests share the same flawed assumption as the code they are meant to validate.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer finds Claude Code usage-limit hook fires before transcript is fully written

A developer building a Telegram notification tool for Claude Code discovered a subtle timing bug that caused usage-limit alerts to silently fail. When Claude Code hits an account usage limit, it triggers a StopFailure hook, but the session transcript is written asynchronously, meaning the hook can fire before the rate-limit data is flushed to disk. In one logged incident, the developer's code read the transcript just 19.7 milliseconds before Claude Code finished writing it, causing the limit-detection logic to find nothing and fall back to a generic error ping. Because a fallback notification still arrived each time, the failure was invisible until the developer noticed the expected limit-specific message was absent. The finding highlights that the hook payload's transcript path only guarantees the file exists, not that the current turn's data has been fully written to it.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How JavaScript Handles Async Tasks: The Event Loop and Concurrency Model Explained

JavaScript is a single-threaded language, meaning it can execute only one operation at a time, yet it manages asynchronous tasks like API calls, timers, and user interactions without freezing the browser. This is made possible by JavaScript's concurrency model, which delegates time-consuming operations to browser Web APIs or Node.js APIs while the main thread continues running. Once an asynchronous task completes, its callback is placed in the Callback Queue and later picked up by the Event Loop for execution. The Call Stack tracks active function calls using a Last In, First Out structure, and works alongside the Microtask Queue and Macrotask Queue to determine execution order. Understanding these components helps developers write more efficient, non-blocking JavaScript code for modern web applications.

0 comments Read more at DEV Community