How a 750-Line Python Checker Catches Silent Failures in AI Prompt Systems

·2 views

A developer building a 15-skill AI agent system discovered that prompt failures can go undetected for weeks because language models continue producing plausible-looking output even when structured data is quietly corrupted. The system silently dropped a required field, causing downstream skills to process null values and deliver slightly incorrect final scores in documents that appeared completely normal. To address this, the developer identified three testable contract layers: deterministic arithmetic checks, JSON schema validation for output structure, and prose rules converted into measurable constraints like word counts and URL citations. A 750-line, dependency-free Python checker runs as a file-write hook and validates outputs against a single JSON contract file, catching real regressions faster than any other method. The checker is itself verified using deliberate fixture failures and mutation testing, after which the developer found that one of their own assertions had been passing for the wrong reason.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Build vs. Buy AI: Why SMBs Must Account for the Full Cost of Ownership

Adopting AI tools for business operations carries hidden long-term costs that go far beyond licensing fees, including ongoing maintenance, model deprecation, and connector upkeep. Research from McKinsey, BCG, and Gartner highlights that while AI use is widespread, fewer than 6% of enterprises generate measurable value at scale, and over 40% of agentic AI projects may be canceled by 2027. AI systems differ from traditional software in that they drift over time — accuracy can silently degrade and outputs carry legal weight, as demonstrated when Air Canada was held liable in a 2024 tribunal ruling for a chatbot's incorrect policy advice. A support agent handling 50,000 chats monthly can require over $500,000 per year just to maintain accuracy, while typical enterprise AI rollouts see only a fraction of licensed seats used regularly. For small and mid-sized businesses, the core question is not which tool has the best features, but whether the organization can sustain the full operational burden of ownership over a 24-month horizon.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Handover App Helps Small Charities Identify Missing Roles After Reach Volunteering Closes

A developer has built a free web tool called Handover, designed to help small charities identify the volunteer or trustee roles they actually need — not just the ones they think they need. The tool prompts organisations to describe what is going wrong, then diagnoses the underlying capability gap and generates a ready-to-use role advert. The project was inspired by the closure of Reach Volunteering, a UK charity that over 45 years placed nearly 6,000 volunteers and trustees annually across 2,440 organisations, contributing an estimated £60 million in expertise. Reach is shutting down despite record impact, as funding for charity capacity-building infrastructure has dried up. Handover runs entirely in the browser with no backend, requires no sign-up, and supports a free Gemini API key for live, personalised results.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Local vs. API Embeddings: Why sentence-transformers Won for This RAG Pipeline

Retrieval-Augmented Generation (RAG) pipelines require converting text into vectors, and developers must choose between running embedding models locally or via an external API. A developer building a local RAG pipeline selected the sentence-transformers/all-MiniLM-L6-v2 model, integrated through ChromaDB, citing zero cost, no data privacy risks, and no API dependencies as key reasons. The model is lightweight at 22 million parameters and 384 dimensions, making it fast enough on standard laptop CPUs for development-scale document ingestion. However, the author acknowledges trade-offs, including a lower accuracy ceiling compared to larger API-hosted models, which becomes more pronounced with domain-specific content such as medical or legal text. The guidance offered is that local embeddings suit privacy-sensitive or cost-constrained projects, while API-based embeddings may be preferable when higher semantic accuracy is required at scale.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Karibu Give Lets Feature Phone Users Donate via USSD With No Internet or App

Karibu Give is a USSD-based micro-donation platform built for feature phone users in Kenya and across Africa who lack smartphones or internet access. Users dial a shortcode, select a cause, and complete a donation of as little as 50 KES through an M-Pesa STK push in under two seconds. The platform was developed for the International Day of Charity to address two barriers to giving: device accessibility and donor trust. Donor data is protected using hashed phone numbers, while all transactions are logged to both a local SQLite database and Snowflake for audit purposes. An AI-generated impact summary powered by Google Gemini provides donors with transparent, plain-language reports on how funds have been used.

0 comments Read more at DEV Community