Developer runs 170 AI agent goals for $0.49, uncovers 10 bugs unit tests missed

·2 views

A developer building PlannerCritic, an open-source engine where one LLM writes plans and a second reviews them, ran field tests across 157–170 goals costing as little as $0.30–$0.49 in API fees. The v0.1.0 field test uncovered 10 issues — including harness bugs, design flaws, and prompt gaps — that traditional unit tests failed to catch because 57 of 65 assertion files were silently malformatted. Subsequent releases v0.2.0 and v0.2.1 each recorded zero field-test failures after code review caught all 41 bugs before the LLM ever ran. The project illustrates how a zero-failure result can mean opposite things: in v0.1.0 it signaled a broken harness, while in v0.2.1 it reflected a robust regression gate. The author argues that real-goal field testing against a live LLM surfaces system failures that hand-crafted unit test inputs are structurally unable to anticipate.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Why MCP Protocol Cannot Enforce Business Logic or Authorization Decisions

The Model Context Protocol (MCP) provides a standardized way for AI agents to discover and invoke tools across systems, solving a real interoperability problem in connecting applications to APIs and databases. However, MCP's design does not address whether a requested business action is actually authorized under organizational policy — only whether the technical connection and input schema are valid. In a practical example, an agent executing an employee offboarding request may successfully call the correct tool with valid arguments, yet still act prematurely if it cannot verify who holds authority over the termination time. Production workflows typically involve multiple distinct identities — requester, actor, subject, and approver — and collapsing these roles leads to misleading audit trails regardless of whether OAuth tokens are used correctly. Business rules, policy enforcement, and multi-party authorization must therefore be layered on top of MCP by the surrounding system, not assumed to be handled by the protocol itself.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How Hibernate's L1 Cache Can Make Failing Spring Data Tests Appear Green

A technical deep-dive from the Evolutionary Architecture series explains how Spring Data's @DataJpaTest can produce false-positive results due to Hibernate's first-level cache. When a test saves an entity and immediately retrieves it using findById(), Hibernate returns the cached in-memory object rather than querying the database, meaning no real persistence round-trip occurs. This behavior masks bugs such as missing column constraints or broken entity mappings, which only surface in production. The article identifies the find()/findById() lookup path as the specific vulnerability, noting that custom JPQL or native queries do force a real database round-trip. The proposed fix involves explicitly flushing Hibernate's write-behind queue and clearing the persistence context before any read assertion, ensuring tests validate actual database persistence rather than in-memory state.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

10GB Home Lab Networks: High Cost, Limited Gains for Most Users

Tech enthusiasts continue to debate whether upgrading to a 10GB network is genuinely necessary for home lab setups. For most users, existing 1GB or 2.5GB configurations comfortably handle tasks like 4K streaming, NAS management, and smart home devices. The real bottleneck typically lies with ISP speed limits, which a faster local network cannot overcome. Upgrading to 10GB also demands costly hardware changes — including Cat6a cabling, new switches, and compatible NICs — adding significant financial and logistical burden. Analysts conclude that unless a user has specific high-demand needs like large-scale data transfers or professional media production, the investment is unlikely to deliver proportionate returns.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

SolonCode Web's Rerun and Continue Buttons Serve Distinct Roles in AI Chats

SolonCode's web interface includes two action buttons — Rerun and Continue — displayed alongside every AI response to address the two most common problems in AI-assisted coding sessions. The Rerun button discards the AI's last response and any subsequent messages, then replays the original user prompt from scratch, allowing the agent to take a fresh approach. In contrast, the Continue button preserves the existing response and resumes generation from where the agent stopped, appending new content to the same output bubble. Under the hood, each button triggers a distinct backend command — /rerun and /continue — that manipulates the session's message history or internal execution trace differently. Developers can use Rerun when an answer is incorrect or took the wrong direction, and Continue when a response was cut short before completion.

0 comments Read more at DEV Community