Google Engineers Share Four Rules for Writing Better LLM Evaluation Rubrics

·1 views

A Google engineering team working on Agent Skills for Google products has published guidance on evaluating AI-generated responses at scale. The team uses an 'LLM-as-a-judge' approach, where a model grades responses against structured rubrics made up of true/false questions. They found that vague or compound rubric questions introduce inconsistency and noisy data, wasting token budgets on unreliable metrics. To improve reliability, they recommend splitting compound questions into atomic checks, avoiding overlapping criteria, and using strict objective language such as RFC 2119 terms like MUST and MUST NOT. The approach also enables the use of smaller, faster models for grading since evaluating clear boolean facts is a less complex task.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Baseline: A Production-Ready FastAPI Starter Kit with JWT Auth and Async SQLAlchemy

A developer has released Baseline, an open-source FastAPI starter kit designed to eliminate repetitive project setup for production applications. The kit follows a strict four-layer architecture — Router, Service, Repository, and Schema — keeping business logic, persistence, and HTTP handling cleanly separated. It includes JWT authentication with short-lived access tokens, a consistent error response shape, and uses Argon2 password hashing instead of the commonly recommended bcrypt wrapper. Tests are run against real PostgreSQL rather than SQLite to better reflect production conditions. The project is intentionally minimal, excluding features like Stripe integration, multi-tenancy, or an admin panel, focusing solely on a reliable, scalable foundation.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Fails to Automate Medium Publishing via Chrome DevTools Protocol

A developer spent two hours attempting to programmatically publish an article to Medium using Chrome DevTools Protocol (CDP) and various browser automation techniques. Medium's story editor relies on a ProseMirror-like architecture whose internal state model cannot be updated through standard DOM manipulation methods such as innerHTML assignment or the deprecated execCommand API. Lower-level CDP methods like Input.insertText and character-by-character keyboard simulation also failed to trigger Medium's internal transaction system, causing the editor to treat the document as empty despite visible text. Clipboard-based workarounds were blocked because navigator.clipboard.writeText requires both explicit user permissions and a genuine user gesture, neither of which CDP can fully replicate. The experience highlights how modern rich-text editors actively resist automation by decoupling their internal state from raw DOM and input events.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer details browser automation workaround for Gumroad's login and product creation

A developer has documented a step-by-step browser automation workflow to create and publish products on Gumroad, which offers no public API for the task. The process required overcoming several technical hurdles, including bypassing React's input tracking using native DOM setters to register form field changes correctly. Gumroad's email-based two-factor authentication added further complexity, requiring a timed polling delay of at least 10 seconds to avoid retrieving stale verification tokens. A persistent Chrome profile with a dedicated user data directory was used to preserve session cookies across script runs, avoiding repeated login cycles. The guide also covers handling a contenteditable description editor and a numeric price field, both of which behave unexpectedly with standard automation approaches.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Harness Engineering Applies to Human Operators, Not Just AI Agents

A concept gaining traction in agentic software development argues that the standard equation 'Agent = Model + Harness' omits a critical counterpart: the human operator who directs what agents build across sessions. Just as an AI agent relies on a harness — loops, tools, context, and constraints — to turn reasoning into useful output, the human directing those agents needs equivalent supporting machinery. This parallel framework, termed the 'human harness,' is defined as a system enabling safe and effective execution by both transmitting a worker's capability into productive work and securing that work against failure states. The concept draws on analogies from physical harnesses, such as those used in horse-drawn carts and rock climbing, to distinguish between transmission and securing functions. Proponents argue that before automating execution through agents, human operators must first orient their work within a structured harness of their own.

0 comments Read more at DEV Community