Why Your Best AI Config in Testing Often Underperforms in Production

·1 views

A software developer has demonstrated a statistical phenomenon called best-of-K selection bias, which causes the top-performing configuration chosen from multiple candidates to appear better than it truly is. When evaluating K configurations on a single test set, the winner is likely the one most flattered by random noise rather than genuinely superior. A simulation using eight identical configurations — all with a true 50% pass rate — showed the selected winner scoring as high as 54%, purely due to chance. The bias grows logarithmically with the number of candidates tested, meaning running more comparisons inflates the apparent performance gap further. This explains why a configuration that wins a pre-deployment bake-off can consistently underperform in production, a drop often mistaken for data drift or bad luck.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Developer discovers dashboard took 7.6 seconds to load due to three separate bugs

A developer at agency platform DevLabs found their client-facing dashboard was taking 7.6 seconds to render just fifteen data points — four stat tiles, three donut chart slices, six bars, and four invoices. The performance issue had gone undetected for months because the local development database contained only four invoices, where load time was a near-instant 111 milliseconds. Using Sentry's performance monitoring tools, the developer identified that MongoDB and Mongoose database spans are not included in Sentry's default integrations and must be manually configured, which had been masking the true source of slowness. Once instrumentation was correctly set up, a single oversized database query span was immediately visible as the primary bottleneck, alongside two additional separate issues. To measure fixes accurately, the developer built a tagged data-volume generator and benchmark harness against a realistic dataset, ensuring results were reproducible without risking real client records.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Replace Repetitive AI Prompts With Autonomous Daily Loops

A tutorial on DEV Community teaches readers to build structured AI loops instead of manually typing repeated prompts each day. The loop design includes four key components: an activation trigger, a numeric stop condition, an independent evaluator, and a default failure action. The approach draws on insights from Anthropic's Boris Cherny, who says he now writes loops rather than prompts, and Addy Osmani's concept of 'loop engineering.' Anthropic's own experiments reportedly showed strong autonomous performance when loops were correctly designed, though the results have not been independently reproduced. The article notes that the method is best suited for repetitive, well-defined tasks and warns that a misconfigured evaluator without a failure policy could allow low-quality outputs to pass through.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Builds Open-Source Monero Mobile App With NFC Payments After Outsourcing Fails

The creator of MyZubster, a decentralized marketplace for tokenizing real-world assets using Monero, has announced plans to build a React Native mobile app after an outsourced development effort with an Italian firm proved too costly and misaligned with the project's vision. The developer paid for external app development through a company called PianetaItalia based in Rimini, but abandoned the effort when the budget ran out and the codebase proved difficult to maintain. The self-built app will be open-source, cross-platform for iOS and Android, and will feature NFC-based Monero payments, allowing buyers to complete transactions by tapping their phone rather than scanning QR codes or entering wallet addresses manually. The existing backend already includes a Node.js payment gateway, Monero subaddress generation, escrow functionality, and AI-assisted dispute resolution. The developer is publicly sharing the journey to invite community collaboration and demonstrate that decentralized applications can be built independently using open-source tools.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Test Your API Against Untrusted Inputs Before Attackers Find the Gaps

Most API test suites only verify that endpoints work with valid, well-formed data, leaving behavior under hostile inputs untested. Security-focused testing treats every input field — request bodies, query parameters, headers, file uploads, and AI-generated payloads — as a potential attack surface. In July 2026, Hugging Face disclosed a security incident where the entry vector was malicious data rather than stolen credentials, with a crafted dataset triggering a remote code loader via template injection. Strict schema validation using constraints like additionalProperties: false, enums, and length limits can act as a security boundary, rejecting malformed inputs before business logic is ever reached. Running adversarial input tests automatically in CI on every code change helps catch these vulnerabilities consistently, regardless of which framework or tooling is used.

0 comments Read more at DEV Community