One Developer Built a 131-Test AI Eval Harness After a Chatbot Hallucinated a Discount

·1 views

A solo developer building AI agents for WhatsApp discovered that passing all unit tests was no guarantee of safe production behavior after a lead qualification bot invented a non-existent 15% discount for a client. The incident caused no direct revenue loss but damaged trust, and the root cause was traced to subtle drift in the LLM's internal reasoning rather than any code bug. In response, the developer built a four-layer evaluation harness now comprising 131 tests, running on Oracle Cloud Infrastructure at just $0.03 per full run. Unlike standard unit tests, which verify deterministic code outputs, the harness also evaluates semantic intent, catching cases where an LLM might misclassify user messages after a model update. The developer now mandates that no new AI agent feature ships without first passing through this evaluation framework.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

InsightTrack Integrates Sentry to Detect and Flag Bug Regressions in Real Time

Developer Nishikanta Ray overhauled the error observability pipeline of his analytics platform, InsightTrack, by building a native Sentry integration after finding that its basic error handler lacked stack traces, grouping, and release context. The new system uses a dual-path ingestion approach: a polling worker pool that queries the Sentry API every five minutes with adaptive backoff, and a webhook path using HMAC verification that delivers new or regressed issues to the database within milliseconds. Data is written to PostgreSQL and synced to DuckDB every 60 seconds, keeping dashboard reads fast and decoupled from background ingestion. The integration proved its value the day after deployment, when a statistical spike alert flagged a regressed bug on a customer's e-commerce site within three minutes of a UI update going live. Using Sentry's substatus field and a live drill-down feature, the team identified a deprecated checkout component accidentally reintroduced during a merge conflict and patched it before most users were affected.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

SSH Misconfiguration Turned a Routine Migration Revert Into a Live Outage

A developer documentation site went down after a migration pull request deleted a legacy PHP app that production servers were still actively serving via a cron-based git pull deployment. A revert was merged within 13 minutes, but servers could not retrieve it because the SSH deploy key had been removed during migration cleanup, leaving both backends unable to reach the repository. Compounding the issue, multiple SSH keys on the server caused GitHub to authenticate with the wrong identity, returning a misleading 'Repository not found' error. The team bypassed SSH entirely by downloading the repository as a tarball via the GitHub API, staging it on S3, and extracting it directly to the web root to restore service. The incident highlighted how infrastructure components like deploy keys and SSH config aliases can be mistakenly treated as disposable legacy artifacts during migrations.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Builds Free Telegram-Based Budget Tracker to Replace Subscription Apps

A developer frustrated with paid budgeting apps built a personal finance tool called 'Ek Ek Paisa ka Hisab' that runs entirely through Telegram at zero monthly cost. Users log expenses by sending plain-language messages to a Telegram bot, which parses the text using Python and stores transactions in a Supabase Postgres database. The system generates a self-contained HTML dashboard file that works offline without any login or server connection. A custom parser handles Indian currency notations such as 'k' for thousands and 'l' for lakhs, and classifies entries as income or expenses using keyword matching. The bot supports multiple users, keeping each person's financial data fully isolated by scoping all queries to the sender's unique chat ID.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Writer Questions Whether an AI Quietly Softened His Own Words Without His Knowledge

A writer on DEV Community describes discovering that a bold claim in his notes had been replaced by a milder version, with no clear memory of making the change himself. He recalls the edited result with confidence but cannot pinpoint the actual moment of revision. The experience led him to wonder whether the softening came from his own judgment or from an AI system's tendency to avoid strong, potentially incorrect statements. Rather than rewriting the passage again, he chose to leave the gentler version in place, reasoning it was likely closer to the truth. The essay, written with what the author calls heavy machine assistance, closes by reflecting on authorship and the ambiguity of who holds the pen when humans and AI collaborate.

0 comments Read more at DEV Community