Monitoring Deployed AI Agents Requires Tracking Behaviour, Reliability, Cost and Safety

·2 views

A DEV Community guide argues that AI agents cannot be monitored like conventional APIs, since they can report success while making wrong decisions or changing state incorrectly. It recommends tracking four dimensions (behaviour, reliability, cost and safety) rather than only HTTP status and latency. Per-run tracing using OpenTelemetry's gen_ai conventions, with secrets and personal data redacted, is presented as the foundation for aggregation, alerting and debugging. The guide highlights the rate of disagreement between an agent's self-reported success and independent outcome checks as a key signal, alongside cost per successful outcome and golden-set evaluations. It also cautions that LLM judges must be calibrated against human judgment before being trusted, and recommends deterministic circuit breakers to stop runaway loops.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Context Engineering: Treating the Model's Input as a Budget to Design

As context windows expanded to hundreds of thousands of tokens, developers shifted from crafting one perfect prompt to deliberately managing everything a model receives. This covers system instructions, retrieved documents, tool outputs, conversation history, and persistent memory. The article warns that stuffing a large window with irrelevant material dilutes attention and can reduce accuracy even when the needed information is present. It recommends selecting relevant retrieved chunks instead of full documents, pruning accumulated tool output and history in long agent loops, and keeping memory curated. The guidance is aimed at builders of retrieval-augmented generation systems and agents beyond the demo stage.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Preserving the Evidence Behind Procurement Decisions, Not Just the Working Comparison

A supplier comparison changes over time as revised offers, quantities, and delivery terms come in, so the working document must stay flexible. The decision record, however, needs to capture exactly what was compared when the choice was made. The article describes an anonymized procurement design that separates the original supplier document, its normalized interpretation, the working comparison, and a frozen decision snapshot. Using an invented example, it shows how a per-pack price and a per-unit price can be misleading without an equivalence rule covering specification, unit, and conditions. The author argues that snapshots should be taken at the point a human acts on a comparison, with later corrections recorded as amendments rather than silent rewrites. The piece makes no claims about measured savings or new deployments.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Turkey's Pardus Linux: History and Philosophy of a Homegrown Open-Source OS

Pardus is a Linux-based operating system distribution developed in Turkey and built on free and open-source software principles. Its goals include meeting user needs, promoting open-source use, and reducing the country's dependence on foreign technology. The system is widely used in public institutions and schools, including on interactive whiteboards. Its name comes from the scientific name of the Anatolian leopard, giving the project a local identity. Development began in 2003 at TÜBİTAK's BİLGEM research center, with the decision made to create a GPL-licensed Linux distribution, and implementation started in early 2004 under UEKAE.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Mirra: A Telegram Bot That Adds Light Activities to Couple and Family Chats

A developer has built Mirra, a Telegram bot aimed at couples, families and friends who want more shared fun within their existing chats. It offers conversation starters and small joint activities, such as a question answered privately and revealed together, or families collecting memories and photos into a shared chronicle. Couples start the bot in a private chat and invite a partner by link, while family mode runs inside a group chat. The core features are free, with optional paid extras, and the couple and family features are available in English, while the friends mode is still being translated beyond Russian. The developer is seeking feedback on whether the partner invitation and family group setup are clear.

0 comments Read more at DEV Community