Detecting silent failures in AI agents with traces and evaluation events

·1 views

AI agents can return a successful HTTP 200 with no exception and no error span, yet still give a wrong or incomplete answer, which keeps monitoring dashboards green. The article proposes tracing every agent step, separating technical errors from content errors, and attaching an evaluation result to each trace. It relies on OpenTelemetry GenAI conventions, including agent and tool-call spans and a gen_ai.evaluation.result event that records quality scores. Capturing tool arguments and results is opt-in because it may expose sensitive data, so it is recommended only on a masked sample. A broken evaluator is recorded as an error rather than treated as a correct answer.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

OpenAI's AI pipeline reports 948 distinct Linux kernel root vulnerabilities for $1.2M

In a mid-October run, an unreleased OpenAI cyber model generated 948 distinct local privilege escalation flaws in the Linux kernel, each with a path to root access. The total API cost was about $1.21 million, roughly $1,280 per vulnerability. The pipeline narrowed the kernel to code reachable by unprivileged users, then had agents validate flaws in virtual machines and attempt root exploits, with common protections like KASLR, SMEP, and SMAP enabled. The bugs are local rather than remote, so an attacker would still need an initial foothold, and the findings are counted as vulnerabilities rather than merged fixes. OpenAI says human engineers review exploits and patches before disclosure, starting with the most severe cases and sending reports to kernel maintainers gradually.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Freezing design tokens so AI agents can't edit shared styling scales

An agent asked to adjust spacing in a checkout header also changed a shared spacing token, which unintentionally affected other pages like settings. The author now freezes token and theme files by default for agent-driven UI work, allowing edits only to named component paths. Token changes require an explicit unlock, such as a human prompt or a separate pull request. When a design needs a new token, the agent is instructed to stop and propose the change in writing rather than editing the shared scale. The approach is paired with PR template checks, CI warnings, and review guidelines to keep token changes separate from feature work.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Using Power Logger Data and pandas to Size an Industrial Diesel Generator

A DEV Community guide explains how a plant engineer with basic Python can size a diesel generator using data from a power logger clipped to the incoming supply. It recommends logging for at least a full production week, including shift changes, weekend slowdowns and restarts after load-shedding, and noting seasonal peaks such as June cooling demand. The author suggests logging individual feeders as well as the incomer, then modeling restart surges by grouping large loads into steps. Site derating for high ambient temperature and altitude, plus expected future load growth, should be checked against each generator's load acceptance. Harmonic distortion from VFDs and UPS systems should also be logged because it affects alternator sizing.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Local speaker diarization on a MacBook Air M3 runs 4.4x faster on the GPU

The author transcribes meetings and interviews on a MacBook Air M3 using whisper.cpp for speech recognition and pyannote.audio for speaker diarization, keeping audio off third-party services for privacy reasons. An initial run of a 42-minute meeting took about 2 hours 9 minutes and heavily loaded the laptop. Process monitoring showed pyannote.audio running on the CPU at around 343% usage, because it was never explicitly assigned to the GPU, even though whisper.cpp already used Metal. The author moved processing into a background job launched with nohup and changed the device setup, achieving a 4.4x speedup for diarization and a 2.5x speedup for the whole pipeline. The changes were not measured individually, and the initial baseline was not a controlled comparison.

0 comments Read more at DEV Community