Developer's 0% test score was a false alarm caused by an empty API credit account

·4 views

A developer using Claude Code discovered that a test report showing 0 out of 96 passing was not due to actual test failures, but because the account had run out of API credits. The incident exposed a fundamental flaw in how many evaluation scripts handle infrastructure failures — they report a zero score instead of signalling that measurement was impossible. Further investigation revealed a second bug: the scoring tool only captured stdout while warnings went to stderr, meaning it had never actually evaluated pass or fail results on any run. The developer also found that a handoff document incorrectly stated no test cases existed for 24 AI skills, when in fact all 24 had full test coverage. The episode highlights the importance of distinguishing between a failed test and a failure to run the test at all.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

AI Can Write Code, But Understanding Why It Exists Remains a Hard Problem

As AI tools make writing code faster and cheaper, software engineers are facing a growing challenge: understanding why existing code was written in the first place. Critical context is often scattered across old commits, pull requests, incident reports, and the memories of engineers who have since left a company. Removing a seemingly redundant piece of code can pass all tests yet cause production failures hours later, because the reasoning behind it was never formally documented. Experts argue that engineering organizations already possess this historical knowledge, but it remains fragmented across disconnected tools and systems. Structuring these relationships into a connected 'engineering graph' could allow AI agents to provide safer, context-aware recommendations rather than acting on code alone.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Size PTU Deployments for GPT-5.6 Luna on Microsoft Foundry

Microsoft Foundry's Provisioned Throughput Units (PTUs) offer dedicated, fixed processing capacity for model deployments, providing tenant-exclusive resources and latency SLAs unlike standard pay-as-you-go plans. PTU sizing requires calculating Normalized TPM by combining effective input and weighted output token volumes, then dividing by a model-specific Input TPM per PTU constant. For GPT-5.6 Luna on a Global Provisioned deployment, that constant is 30,000 Input TPM per PTU, with a minimum allocation of 15 PTUs. A sample workload of 1,000 peak RPM with 1,200-token prompts and 200-token responses requires 80 PTUs without caching, dropping to 60 PTUs when a 50% prompt cache rate is applied. PTU quota is allocated per subscription and region, meaning capacity in one region or deployment type cannot be shared with another.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Flutter State Management Showdown: Provider, Riverpod, and Bloc Compared

A Flutter developer with production experience across all three major state management solutions — Provider, Riverpod, and Bloc — has outlined a framework for choosing between them based on six practical criteria. The criteria include learning curve, boilerplate, testability, ecosystem tooling, refactor cost, and team fit, ranked by how severely each affects production apps. Provider is highlighted as the simplest entry point, requiring minimal ceremony and readable code, but it lacks enforced structure and can lead to tangled logic as apps scale. The author argues that the real question is not which solution is objectively best, but which is most appropriate given the app's complexity and the team maintaining it. All three tools are described as actively maintained and capable of powering real production applications.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Builds Private AI Journaling App Using Gemini, Firebase and Cloud Run

A developer has created Personal Gemini Journal, a privacy-focused AI-powered journaling application, as part of the Google Cloud Run AI Challenge. The app allows users to write journal entries and hold multi-turn conversations with Google Gemini, which then generates structured insights including key themes, emotional patterns, and actionable next steps. Journal sessions and interactions are stored in Cloud Firestore using Firebase UID-based paths, ensuring each user can only access their own data. Authentication is handled via Firebase with Google Sign-In, while the Gemini API key is kept server-side and managed through Google Cloud Secret Manager to prevent exposure. Users can search past reflections, organize entries by mood, export their data as a JSON archive, and resume previous sessions after signing back in.

0 comments Read more at DEV Community