Why AI test systems need a failure ledger, not just a pass/fail score

·2 views

Software engineer Derek Wang argues, in an essay published on DEV Community, that AI harness testing should be modeled on philosopher Karl Popper's principle of falsification — advancing knowledge by eliminating wrong answers rather than accumulating right ones. Wang contends that a test system capable of recognizing and classifying failures is far more valuable than one that simply returns a pass or fail verdict. He describes a regression suite his team built, called fulltest, which maintains a structured ledger cataloguing each class of failure along with its root cause and known repair strategies. The system distinguishes between known failures — previously documented debts — and unknown failures, treating the latter as the most critical signal for genuine improvement. Wang concludes that fast, trustworthy feedback is what enables AI agents to make bold correct changes while remaining cautious about potentially harmful ones.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Why Shipping Broken Software Early Beats Waiting for Perfection

A software developer argues that releasing imperfect products early is more effective than polishing them in isolation before launch. Drawing on the philosophy of a colleague named Marek, the piece contends that live, production software exposes real bugs and user behavior that internal reviews simply cannot replicate. Delaying a release to refine a product risks optimizing for untested assumptions, often resulting in larger, harder-to-fix errors at launch. The author advocates for an iterative, public-facing build cycle — ship early, observe failures, fix specifically, and repeat. The core argument frames building in public not as a marketing tactic but as an epistemological discipline that grounds development in real-world feedback rather than speculation.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer builds image copy-detection tool using search engines instead of web crawling

Software developer Harman Singh built Sealify in 2023, a copy-detection tool that identifies edited or redistributed versions of images and videos across the web without running its own crawler. Instead of indexing the web independently, Sealify uses existing reverse image search from Google, Bing, and Yandex as a broad first stage, then applies local verification on a single Mac mini to filter out false matches. The approach was driven by both practical and environmental concerns, as building a proprietary web index requires continuous crawling, large-scale storage, and significant energy and water consumption. Singh notes that data centers consumed roughly 66 billion liters of water in the US alone in 2023, and that duplicating existing infrastructure produces no new value. By reusing indexes that already exist, Sealify aims to deliver accurate copy detection while avoiding the hardware, energy, and e-waste costs of a redundant system.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

SQL Queries Don't Run in the Order You Write Them — Here's Why It Matters

Although SQL queries are written starting with SELECT, databases process them in a different sequence that affects both results and performance. The actual execution order follows these steps: FROM, WHERE, GROUP BY, HAVING, SELECT, DISTINCT, ORDER BY, and finally LIMIT or OFFSET. Early stages like FROM and WHERE handle table selection and row filtering, which can significantly reduce the data processed in later steps. HAVING differs from WHERE in that it filters grouped results rather than individual rows. Understanding this execution order helps developers write more efficient queries and troubleshoot unexpected outputs.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How and Why to Split a PDF Into Pages or Sections Safely

Splitting a PDF allows users to extract specific pages or sections—such as a single chapter or signature page—into separate files without affecting quality. Common methods include splitting by page range, extracting selected pages, or generating one file per page, depending on the use case. Users should note that printed page numbers may not match file positions, and internal links or form field connections can break when pages are separated into different files. Encrypted PDFs typically require the password to be removed before splitting can proceed. When sharing extracted pages from sensitive documents like contracts or IDs, it is advisable to scroll through the output file first to confirm no unintended content is included.

0 comments Read more at DEV Community