SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

Solo dev discovers all 155 tests passed while every real AI request silently failed

0
·1 views

A solo developer building GigLegal, a contract tool for freelancers on Next.js 15, found that all 155 Playwright end-to-end tests were passing while every real AI request was failing in production. The root cause was a provider resolver that automatically switched to a mock AI service during tests, meaning the test suite never interacted with the actual API. Behind this masking issue lay three stacked bugs: an invalid structured output configuration, a schema with 66 optional fields that exceeded the vendor's undocumented 24-field limit, and a timeout problem on a 20-field policy step. Splitting the large single AI call into smaller, focused requests ultimately resolved the failures, with the developer noting that mocks reflect assumptions rather than real API behavior.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

How a Per-Project Token Quota System Stopped One Batch Job From Starving Others

Three projects sharing a single LLM gateway and a 10-million-token allowance ran into trouble on day nineteen when a batch summarization job consumed 7.1 million tokens in four hours, causing all other projects to fail with quota errors. The root cause was a shared token pool with no per-project attribution, meaning any single consumer could exhaust the entire allowance unchecked. To fix this, a per-project quota layer was built in Python, assigning each project its own daily token budget and per-minute request limit, with all limits summing to 9 million to preserve a 10 percent safety buffer. The system runs a pre-check before each upstream request using a conservative token estimate, rejecting over-budget requests with a 429 response, then reconciles actual usage from the API response afterward. This two-step approach ensures that a single bad estimate causes only a minor overshoot rather than a runaway consumption event.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Developer Builds UrbanLease Appliance Rental Platform, Shares Key Technical Lessons

A developer built UrbanLease, a web platform allowing users to rent home appliances with features including rental history, payments, and separate user and vendor portals. Early in development, the project was migrated from SQLite to MySQL to better handle growing data complexity, requiring fixes to tables, columns, and database relationships. Feeling the initial version lacked differentiation, the developer added a rule-based smart recommendation system that filters and ranks appliances based on user inputs like budget and rental duration. A second feature, an appliance condition history log, was introduced to show prospective renters details such as past rentals, maintenance records, and service history. The developer noted that building a functional product is only part of the process, emphasizing that meaningful features should solve real problems rather than simply add more pages.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Study Shows Full-File AI Code Rewrites Consume 11x More Tokens Than Simple Generation

A developer burned through a 10-million-token free AI coding allowance in two days by repeatedly requesting full-file rewrites, prompting a structured investigation into token costs. Using MonkeyCode, an open-source AI coding platform offering 10 million free tokens and a free server, a reproducible benchmarking harness was built to measure real token consumption across common coding tasks. Results showed that a full-file rewrite costs roughly 3,300 tokens per run, compared to just 300 for a simple code generation task — an 11x difference. The benchmark covered five task types: code generation, debugging, test writing, refactoring, and full-file rewriting, with token estimates based on a four-characters-per-token heuristic. The findings highlight that deliberate task selection, rather than defaulting to rewrites, can stretch a 10-million-token allowance to cover tens of thousands of meaningful coding operations.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

AI-Written Tests Can Pass While Missing Critical Bugs, Mutation Testing Reveals

A developer merged an AI-generated 14-test suite for an ISO date parser without review, only to discover the tests failed to catch broken new timestamp formats added a week later. The tests repeatedly validated the same happy-path scenarios with minor input variations, leaving key edge cases completely uncovered. Experts warn that a fully green AI-generated test suite does not guarantee meaningful coverage, and that false confidence from such suites can be more harmful than having no tests at all. Mutation testing — which introduces deliberate code bugs to check whether tests detect them — offers a measurable way to assess suite quality, with a score below 50% indicating largely decorative tests. A workflow using the open-source MonkeyCode platform, which currently offers free server access and a 10-million-token allowance, is proposed to automate this audit on every agent-generated suite.

0 comments Read more at DEV Community
Solo dev discovers all 155 tests passed while every real AI request silently failed · ShortSingh