AgentSpec Offers Jest-Style Testing Framework Built for Non-Deterministic AI Agents

·2 views

A developer has released AgentSpec, an open-source testing framework designed to address the challenges of validating AI agent behavior, which traditional tools like Jest and Vitest cannot reliably handle. Unlike conventional test suites that rely on exact string matching, AgentSpec offers flexible assertions including semantic similarity checks, tool-call verification, and an LLM-based quality judge that runs locally via Ollama at no API cost. The framework also generates behavior diff reports that highlight precisely what changed in an agent's output between test runs, helping teams identify regressions beyond simple pass/fail results. AgentSpec integrates with CI/CD pipelines out of the box, supporting JUnit XML output and a dedicated GitHub Action. The tool was prompted by a real-world incident in which a modified system prompt caused a support agent to silently stop offering password reset assistance, a bug that only surfaced through a user complaint rather than automated tests.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Heavy TV Watching Linked to Reduced Brain Volume, USC Study Finds

A new study from the University of Southern California has found an association between heavy television watching and smaller brain structures. Researchers observed that individuals who watch large amounts of TV tend to show reduced volume in certain brain regions. The findings suggest that prolonged passive screen time may have measurable effects on brain development or maintenance. While the study identifies a correlation, researchers have not established a definitive causal relationship between TV habits and brain size changes.

0 comments Read more at Hacker News

ProgrammingDEV Community ·

AI Image Generator Bypasses Safety Filters Using Semantic Cues, Not Keywords

A developer writing on DEV Community demonstrated what they call the 'Fifi Law': AI image generators can produce historically sensitive figures even when explicit names or trigger words are absent from the prompt. Using a carefully worded, seemingly neutral description of a 1930s German child's bedroom with a 'leader figure' portrait, the author generated an image widely recognizable as Adolf Hitler. The experiment highlights that current AI content moderation systems rely on keyword string-matching rather than semantic understanding of context. Because no banned terms appeared in the prompt, the safety filter approved the request while the image model correctly inferred the intended subject. The author argues this gap between keyword-based censorship and semantic comprehension represents a real and recurring vulnerability in AI safety systems.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How Local AI Models Actually Work: A Developer's Deep Dive with Ollama

A developer running experiments on an NVIDIA DGX Spark machine spent Week 2 of a 32-week project exploring what happens beneath the surface of a local AI model API. Using Ollama as the runtime, they demonstrated the distinction between a model — which is static weight data stored on disk — and the runtime, which loads those weights into memory and generates text from prompts. By querying Ollama's local API, the developer retrieved key technical details about the Phi-4 model, including its 14.7 billion parameters, 16,384-token context length, and Q4_K_M quantization format. They also inspected the model's manifest file, revealing that a packaged model consists of separate blobs for the weights, prompt template, license, and default parameters. The exercise aimed to build foundational understanding of how AI model packaging and runtimes function before advancing to more complex GPU-level topics.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Build Alarms That Page on Root Cause, Not Just Output Count

A software engineering post on DEV Community outlines a common flaw in output-count alarms, using a background matching job as a practical example. The core problem is that a count of zero can mean either a system failure or a legitimately quiet period, making a single threshold unable to distinguish between the two. The author demonstrates that adding a streak hedge, such as requiring three consecutive zero-result nights, delays the false alert but does not resolve the underlying ambiguity. The proposed fix moves intelligence into the emitter itself, logging the reason a zero count occurred so the alarm reacts to cause rather than coincidence. The redesigned system requires no additional AWS cost, eliminates observed false pages, and records benign zero nights as a separate, unalarmed metric.

0 comments Read more at DEV Community