Baize Scores AI Agents on Tool-Call Behavior Using Frozen, Replayable Production Runs

·2 views

Baize evaluates AI agents by examining the sequence of tool calls they make rather than the wording of their final replies. It builds evaluation corpora by sampling real production runs from historical event logs, filtered by criteria such as time window and run status. Sampling is deterministic, using a seeded hash so the same batch is reproduced each time, with defaults of about 10 cases per batch and a cap of 20. Each sampled run is frozen into an evaluation case with its full event stream and input snapshot, and cases are scored only after their expected behavior is confirmed. Assertions cover required reads before writes, mandatory or forbidden tool calls, argument checks, and downstream state comparisons, and are also usable through a skill-level EVAL.yaml file and an offline CI script.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

A 28-day Search Console review: 1,510 impressions but only 6 clicks for a new site

A small English-language site of about 70 posts, aimed at foreigners handling everyday matters in China, examined its 28-day Search Console data after impressions rose sharply over roughly two weeks. Most impressions came from pages already ranking in the top 10, yet those pages drew only 5 clicks from 1,131 impressions, a click-through rate under half a percent. The author attributes the gap largely to weak titles and descriptions that read as topic labels rather than answers to the questions users actually search. Query-level data was incomplete because Search Console applies privacy thresholds at low volume, so the page-level view was used instead. The author's planned response is to rework titles and descriptions rather than publish more posts.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

pgbrake: an undo layer for Postgres when AI agents run SQL

The developer describes pgbrake, an open-source MCP server that lets AI coding agents such as Claude Code and Codex work against Postgres databases without direct access. Each statement is parsed with pglast, and before UPDATE or DELETE operations the affected rows are saved locally so they can be restored with an undo command, which refuses if those rows changed afterward. DDL, statements without WHERE clauses, and changes touching more than 100 rows require human approval through the client or a terminal command that cannot run without a TTY. All statements and decisions are recorded in a local audit log, and nothing leaves the user's machine. The tool is MIT-licensed, written in Python, and has been tested only on Postgres 17, with its author noting that undo does not cover TRUNCATE or DDL and is no substitute for backups.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Checklist for safely switching PHP versions on WordPress: audit, staging clone, and rollback

A developer guide explains that changing a WordPress site's PHP version from a host dropdown is quick, but confirming the site still works can take much longer. The recommended process starts with inventorying installed plugins, themes, must-use plugins, and drop-in files, since drop-ins like object-cache.php can cause white screens on incompatible PHP versions. It also advises verifying that the WP-CLI runtime matches the web-facing PHP-FPM version, because command-line tests otherwise may not reflect what visitors experience. Compatibility scanning with PHPCompatibility via PHP_CodeSniffer is suggested to flag removed functions and deprecated constructs, though it cannot detect behavioural changes or missing extensions. Finally, the guide recommends comparing loaded PHP extensions between the old and new versions, since missing modules such as gd or intl can break sites in ways code scans miss.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Haunted Shell: A Terminal Ghost That Comments on Your Commands Offline

A developer built Haunted Shell, a small tool that adds a ghostly voice to the terminal and occasionally comments on commands just run. It uses a zsh hook to capture each command and its exit code, then passes them to Gemma 3 (1B), an open-weight Google model served locally through Ollama. The ghost speaks after roughly one in three commands, running in the background so it never blocks the prompt. The project runs entirely offline with no cloud calls, which the author argues keeps shell history private and costs nothing per use. It was submitted to the Hacktoberfest Open-Source AI Challenge, entering the Best Use of Gemma category.

0 comments Read more at DEV Community