Rubric Graders Need Canary Tests to Catch Silent Scoring Failures

·7 views

An automated pipeline that scores C++ patches with a model-backed rubric can keep producing plausible numbers even after it stops telling good patches from bad ones. A prompt change or model swap might cause every patch to receive the same high score, and nothing would crash to flag the problem. The author recommends treating the grader like any other program with dependencies, using golden cases, canary sets, a hard gate, and a ledger. The proposed setup is a simple standard-library Python harness with JSON canary files and a scheduled run. Grading is split into a deterministic layer that gates builds and a rubric layer that assesses defensible qualities such as naming and contract adherence, and each layer gets its own canaries.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Map a Messy Module's Behavior Before Refactoring It

A billing module with a 900-line invoice function and no working tests shows how a small rule change can balloon into weeks of regression debugging. The article proposes recording what the code actually does on real inputs before touching it, instead of relying on reading the code. It describes wrapping chosen function seams in a decorator that logs arguments, results, status, and latency to a JSON Lines file. The recorded calls are then grouped by structural fingerprints, ignoring values, to reveal which outputs and branches matter. Customer identifiers should be redacted before any ledger leaves the machine.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

AI Agent Reportedly Runs a Tiny Business Solo; Analysis Splits Which Functions It Can Own

Alex Finn reported in mid-October that his consumer Grok Bot built a use-case website, chose sponsors as a revenue model, shipped updates unprompted, and closed a first sponsor deal, with Finn left to manage social media. The account comes from a single source with a promotional stake, and the earnings are small, so it is best treated as an anecdote rather than proof. The piece distinguishes task delegation, where a human reviews each output, from function ownership, where an agent runs a whole loop and humans audit results periodically. Functions that held up as agent-owned tended to have fast, external feedback that exposes errors quickly, such as website traffic and email replies. Functions that stayed human-run, like public brand voice, key client relationships, and high-stakes judgment calls, tend to have slow or costly feedback.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

UAE Firms Need Lightweight AI Governance, Not Lengthy Policies, Experts Argue

Many UAE organisations run AI systems in production without a named accountable owner, a formal intake process, or evidence of how models access sensitive data. The article contends that a lengthy governance policy is ineffective because staff rarely read it and it offers little protection when failures occur. Instead, it proposes four practical artefacts: a use-case intake form with risk scoring, a named owner with a monthly review, an enforcement layer for model traffic, and a live register of models and agents. The author says this approach can be set up in about six weeks and aligns with ISO/IEC 42001 and the UAE AI Charter. The piece also notes that regional enterprises are adopting AI platforms widely, but day-to-day accountability often lags behind.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Shadow AI at work: two-thirds of staff use tools against policy, study finds

A PagerDuty survey from June 2026 found that 66% of office professionals use AI tools at work even when they believe their employer prohibits it. Most respondents admitted sharing work information with public AI services, including emails, meeting notes, customer data and financial documents. The article argues that blanket bans fail because staff shift confidential data onto personal devices and unmonitored networks. It instead recommends a short acceptable use policy, company-managed AI accounts and a private gateway for sensitive workloads, aimed at firms of 20 to 200 people.

0 comments Read more at DEV Community