Developer finds same scoring flaw across five self-improving AI loops built in one evening

·1 views

A developer built five small AI self-improvement loops in a single evening using Claude Code, each following a mutate-judge-keep-if-better structure applied to different artifacts such as prompts, graphs, and guardrail harnesses. All five loops showed score gains over 10 rounds, appearing successful on paper. However, after reviewing all five project READMEs side by side, the developer identified a shared flaw: every judge measured whether the correct keywords or structural shapes were present, not whether the artifact actually functioned as intended. For example, a graph judge would score a non-functional 'verifier' node identically to a working one, simply because the keyword appeared. The developer flagged this as a cautionary finding for anyone building optimize-against-a-judge pipelines, noting the bug was only caught by comparing all five projects together rather than moving straight to a sixth prototype.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

How to Build a UAT Environment That Gives You Reliable Test Results

User acceptance testing (UAT) frequently produces misleading results not because of faulty code, but because the environment it runs in is poorly configured or misunderstood. Many teams default to wherever a spare environment exists — often dev, QA, or staging — each of which introduces its own sources of noise and false failures. A proper UAT environment should be dedicated, stable, isolated from production data, and as close to production configuration as possible. Common pitfalls include configuration drift, missing third-party integrations, unrealistic test data, and incorrect user permissions that hide real bugs or surface fake ones. Addressing these environment-level problems before testing begins is what separates actionable stakeholder feedback from reports that teams cannot reproduce or act on.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

SmartSuite Dropped Its Free Plan, Highlighting Risks of Hosted Work OS Tools

In January 2026, SmartSuite removed its free plan, leaving teams that relied on it scrambling without prior warning. The incident illustrates a broader pattern in the work-OS software category, where vendors have quietly repriced plans, restricted features to higher tiers, or folded products after acquisitions. The author argues the real risk of hosted tools is not vendor misconduct but the inherent dependency on a third party's commercial decisions. As an alternative, the author promotes Appdor, a self-hosted work OS built on standard Postgres, where data remains accessible via standard database tools regardless of the vendor's choices. The key question posed to teams is not whether their vendor is trustworthy, but what a sudden change in terms would actually cost them to fix.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Builds Open Asset Store for Limn Engine Game Framework

A developer has begun building an Asset Store for Limn Engine, a game development framework, with the aim of creating a central hub for game assets. The store is planned to support a wide range of asset types, including 2D sprites, backgrounds, UI elements, audio, textures, VFX, and developer tools. A 2D Asset Editor is currently under development, allowing users to import images and classify them by asset type. The interface is being built using HTML, CSS, and JavaScript. The project is being developed in public and is intended to eventually form a broader ecosystem around the Limn Engine platform.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Design a Distributed Logging Platform from Scratch

Most applications can generate logs easily, but managing millions of log events across hundreds of machines turns logging into a distributed systems challenge. A well-designed logging platform must simultaneously handle high-speed ingestion and fast, flexible querying — two workloads that require careful architectural balance. The proposed architecture layers log collectors, an ingestion layer, a buffer queue, indexing nodes, storage nodes, and a query coordinator to separate concerns cleanly. Using an e-commerce platform as a case study, the design supports queries such as filtering errors by service, searching by status code, and aggregating failures over time. The article argues that building such a system from scratch is one of the most effective ways to understand how production-grade tools like Elasticsearch work under the hood.

0 comments Read more at DEV Community