Developer finds 8 of 30 AI agent test calls failed due to flawed prompts, not the model

·1 views

A software developer building an AI intake agent for US home-service contractors discovered that 8 out of 30 scripted test calls failed during evaluation. In one critical case involving a suspected gas leak, the agent correctly advised the caller to evacuate but logged a record with no address or callback number, making a dispatcher response impossible. Investigation revealed the failures stemmed from prompt design errors: the agent was given a file path to a JSON schema it could not access, causing it to invent its own data structure. A safety guardrail instructing the agent to stop collecting information during emergencies also lacked instructions to resume, leaving dispatchers with incomplete records. The developer fixed both issues by injecting the actual schema text directly into the prompt and adding explicit post-emergency follow-up instructions for collecting a caller's address and phone number.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

ML Agent Reported Job Canceled While GPU Cluster Kept Running

An ML operations agent case study highlights a critical gap between agent-level cancellation and actual job termination on an external GPU scheduler. When an operator instructed the agent to cancel a fine-tuning run, the agent halted its own orchestration and reported success — but the job request had already been handed off to the scheduler. The scheduler subsequently accepted the job, which began consuming reserved compute despite the operator believing no allocation was active. The agent also risked losing the job ID needed to locate and properly cancel the workload. Experts recommend agents treat such cancellations as pending until the external scheduler confirms the job's final state, and actively retrieve the job ID to request and verify termination.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

TypeSafe Jev Offers a Lightweight Decision Layer to Streamline AI Agent Workflows

Most AI agent systems rely on a single large language model for every task, including planning, tool selection, and risk scoring, which makes them costly and hard to test. TypeSafe Jev is a purpose-built tool that handles decision-making over a closed set of outcomes using three typed primitives: Choice, Score, and Noul. These primitives let developers separate fast, frequent decisions — such as tool routing, moderation triage, and permission gates — from open-ended generative tasks, keeping the latter in more capable models. Under this architecture, deterministic application code retains control over permissions, side effects, and rollback, preventing probabilistic components from silently driving destructive actions. The approach makes individual models replaceable while keeping the overall decision protocol stable and observable.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

VoiceLoop: Open-Source Browser Voice Agent Library Rivals OpenAI Realtime on Latency

TodoForAI has open-sourced VoiceLoop, a zero-dependency JavaScript library that runs a full voice agent pipeline — VAD, speech-to-text, LLM, and text-to-speech — entirely in the browser. The library is designed for low latency, with first audio responses delivered in under a second by streaming TTS output while the LLM is still generating. A key feature is real barge-in detection based on transcribed words rather than microphone energy, which prevents the agent from cutting itself off due to speaker echo. Independent benchmarking via the team's own voice-agent-bench tool showed VoiceLoop with Deepgram and ElevenLabs matching OpenAI Realtime at roughly 862ms voice-to-voice latency while recording zero self-interruptions across 30 echo-coupled turns. The library is available on npm and supports any OpenAI-compatible LLM endpoint, with local WASM-based options requiring no API keys.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

n8n 2.0 Brings Breaking Security Changes That Could Disrupt Existing Workflows

n8n 2.0 is a significant upgrade that changes core defaults around Code nodes, task runners, Python execution, database handling, OAuth, and workflow publishing. Task runners are now enabled by default, and Code nodes can no longer access environment variables or use the $evaluateExpression() function as they previously could, which may silently break existing workflows. Python execution has also been overhauled, replacing the Pyodide-based model with an external task-runner approach that drops support for some previously available built-in variables. These changes are primarily security-driven, meaning workflows that ran without issue on n8n 1.x may behave differently after upgrading even without any manual edits. Developers are advised to test Code nodes with the new task-runner flag enabled and migrate secrets from environment variables to n8n credentials before attempting the upgrade.

0 comments Read more at DEV Community