How to Build a Python Eval Harness That Catches AI Agent Failures Before Users Do

·1 views

A developer tutorial on DEV Community outlines how to build an automated evaluation harness for AI agents after a real-world incident where a travel agent booked the wrong flight for three days without detection. The author argues that manual, happy-path testing is fundamentally flawed because it cannot account for non-determinism, silent regressions, or the unpredictable prompts real users submit. The proposed solution is a Python-based testing repo that grades an agent across four dimensions: whether it called the correct tools, whether its reasoning path was efficient, whether it achieved its goal judged by an LLM, and whether recent changes degraded overall performance. The harness is demonstrated using a small travel-support agent with three tools and is designed to integrate with CI pipelines to automatically flag regressions. The pattern is framework-agnostic and compatible with LangChain, LangGraph, raw MCP, or custom agent setups.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Kotro Proxy Engine Aims to Cut AI Coding Costs With Local Token Optimization

As AI coding assistants like Cursor and Devin become standard developer tools, engineers and CTOs are reporting runaway cloud API costs driven by long sessions, large codebases, and repetitive error loops. Kotro is an open-source, Rust-based sidecar tool designed to sit between a developer's IDE and LLM providers, intercepting and optimizing outgoing AI traffic locally. Its key features include a circuit breaker that halts infinite retry loops, an intelligent router that redirects simple prompts to free local models instead of paid cloud APIs, and AST-aware code pruning that strips irrelevant context before sending requests upstream. The tool also embeds a vector-based semantic cache to match near-duplicate prompts without additional API calls. Kotro's developers claim these combined optimizations can reduce token consumption by up to 50–60%, lowering costs without requiring manual workflow changes.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

PostureGuard: Open-Source Tool Uses MediaPipe and Electron to Detect Bad Posture

A developer tutorial on DEV Community introduces PostureGuard, a real-time desktop posture monitoring application built with MediaPipe, OpenCV, Electron, and Vue.js. The tool uses a webcam to track key body landmarks — specifically the ear, shoulder, and nose — and calculates head tilt angles to detect forward head posture, commonly called 'tech neck.' A Python-based vision engine runs pose estimation on a standard CPU using MediaPipe's lightweight BlazePose model, making it accessible without dedicated GPU hardware. The vision engine communicates with the Electron frontend via a WebSocket bridge, triggering on-screen alerts when the user's head tilt exceeds a 25-degree threshold. The project is aimed at developers seeking a lightweight, background utility to improve posture during long computing sessions.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Why Strong Coding Skills Alone Won't Land You Freelance Clients

A developer-focused guide argues that technical proficiency and freelance marketability are fundamentally different skill sets, with the latter rarely taught in formal training. The core advice is to reframe self-introductions around specific client problems rather than listing technologies, since business owners pay to have problems solved, not for elegant code. Instead of starting on competitive platforms like Upwork with no reviews, new freelancers are urged to tap existing networks with targeted asks and approach local businesses whose site issues are already visible. Trust, not technical ability, is identified as the primary barrier clients face when hiring a freelancer. The guide also outlines a structured pricing method that includes a 30–50% time buffer and delays quoting until project scope is clearly defined.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer publishes free Spanish study notes for GitHub Agentic AI Developer certification

A developer named Matías Beltramone created comprehensive Spanish-language study notes for the GH-600 (GitHub Certified: Agentic AI Developer) certification after finding official Microsoft Learn materials insufficient, estimating they cover only 35–45% of the actual exam content. The self-made notes span all six exam domains, including agent architecture, tool permissions, memory management, evaluation, multi-agent orchestration, and guardrails. Each domain section includes dense theory, real code examples, and a hands-on lab, designed to be self-contained without requiring additional sources. Beltramone used Claude as a research and editing assistant to verify concepts and improve clarity, while drawing exam-specific content from official sources. The complete notes are available for free on his blog and GitHub repository.

0 comments Read more at DEV Community