Developer Withholds Final AI Test Despite Perfect Score to Prevent Benchmark Contamination

·1 views

A software developer building an AI-powered operating system called Eterna Clarity achieved a perfect 24/24 score on an internal benchmark but chose not to advance the model or expose it to the final test set. The decision stemmed from concerns that the benchmark had become part of the development loop, meaning strong performance could reflect familiarity with test data rather than genuine improvement. An earlier training attempt illustrated the risk: fixing one failing case caused the model's broader performance to drop from 23/24 to 18/24, as it became overconfident in situations where abstaining was the correct response. The developer identified two key lessons — that targeted improvements must be measured against retained behaviors, and that a benchmark loses its evaluative integrity once it influences training decisions. To restore a truly blind final evaluation, a separate 60-case test set was created and kept sealed until candidate models were fully frozen.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

How to Prevent Linux Memory Thrashing Using zram and systemd-oomd

When a Linux system runs low on RAM, it often enters a thrashing state where disk-based swap causes severe slowdowns before the kernel's OOM killer intervenes. Two tools address this problem: zram creates a compressed in-RAM swap device to keep memory reclaim fast, while systemd-oomd monitors cgroup v2 pressure stall information (PSI) to proactively kill memory-hungry processes before the system locks up. Together, they replace slow disk-backed swap with a more responsive, in-memory alternative supported on modern distros including Debian, Ubuntu, Fedora, and Arch. Setup involves configuring zram-generator with a small config file and enabling systemd-oomd, both of which integrate cleanly with systemd without requiring custom boot scripts. The approach requires kernel PSI support (available since Linux 4.20) and cgroup v2, both of which are defaults on most current Linux distributions.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Argues Intent-Based RPA Architecture Can Cut Maintenance Costs by 82%

A software developer writing on DEV Community argues that traditional robotic process automation systems fail because they are built around specific UI paths rather than desired outcomes, making them brittle whenever interfaces change. To address this, they designed a modular automation architecture using a 636-byte WebAssembly proof module injected at the browser boundary, separating intelligence from authority. The design enforces a strict lifecycle — intent, lease, observe, decide, act, verify, artifact — where an AI model can propose actions but cannot grant itself permissions or cross security boundaries. Using a planning tool called agent-calc, the author modeled a hypothetical scenario of 100 workflows and estimated monthly maintenance costs could drop from $112,000 to around $20,000, an 82% reduction, though they caution the figures are illustrative assumptions rather than measured results. To stress-test the approach beyond controlled demos, the team also built a synthetic web application called Component Gym, designed to resist memorization by randomizing UI elements between runs.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

LaunchSignal aggregates Product Hunt, Hacker News, and GitHub into one ranked feed

A developer built LaunchSignal, a single dashboard that combines new product and repository launches from Product Hunt, Hacker News, and GitHub into one ranked list. Because the three platforms use incompatible scoring systems, the tool relies on a time-decay formula — score multiplied by exp(-hours/18) — to surface fresher, more active entries above older stagnant ones. The 18-hour half-life was chosen to ensure a recent Show HN post with real votes ranks higher than a brand-new empty repository. GitHub search queries were constrained to a short keyword list covering AI-related terms after longer queries returned errors. The public board does not personalize results, and a paid keyword alert tier is available at $9 per month, while the ranking itself carries no editorial curation or classification layer.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer's Claude Code Cost Tracker Silently Logged $0 for 52 Days Due to Missing Payload Field

A solo developer building an automated Claude Code environment discovered that their token cost-tracking script had been silently recording zero values for 52 consecutive days across 2,340 logged rows. The error stemmed from a misunderstanding of Claude Code's Stop hook, which fires after every assistant response but does not include usage or model fields in its payload. Because the script attempted to read non-existent fields, it converted undefined values to NaN and ultimately logged $0.00 for every session without throwing any errors. The Stop hook payload contains only four fields — session_id, transcript_path, cwd, and hook_event_name — a detail not clearly documented, making the silent failure easy to miss. The correct approach, the developer explains, is to follow the transcript_path pointer and extract token usage data directly from the session transcript file.

0 comments Read more at DEV Community