AI Model Followed Provider Schema Over Harness Instruction, Exposing Test Logic Flaw

·2 views

A developer discovered that an AI model's tool-call arguments were flagged as a mismatch by their test harness, but the root cause was a conflict between two competing authorities: the harness expected a single-key JSON object, while the provider's runtime schema required additional fields. The model complied with the provider's schema rather than the harness instruction, which was technically correct behavior since the harness was demanding something the schema did not permit. The developer noted that a mismatch only proves a difference exists, not which side holds the correct expectation. Rather than loosening the comparison to hide the failure, the fix added the required intent field as a harness-authored constant to the expected object, preserving the integrity of the control. Simply copying the model's own output into the expected value was rejected as it would have made the test verify the model against itself.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Developer Builds Charity-Linked Card Game Using Pledge Donation API

A developer created Give Garden, a card-collection garden game that integrates real charitable donations as a core gameplay mechanic, submitted for DEV Community's Weekend Challenge: Generosity Edition. The game uses the Pledge API, a charitable giving platform, to process donations without the developer handling any funds directly. Players collect plant and tree cards of varying rarity — from common to legendary — and strategically place them in a grid garden to earn in-game currency called Blessings. Premium card spins, which unlock rarer cards, are earned by donating to a charity of the player's choice, with the card type corresponding to the charity's cause. The game is built with Next.js, Clerk, Neon, and Drizzle, and is currently live on Vercel with Pledge running in sandbox mode for testing.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Structured Shell Nushell Tested Against Bash in 380-Run Agent Workflow Study

A developer conducted a structured experiment in mid-2026 to determine when Nushell, a shell that preserves typed and tabular data through pipelines, genuinely improves AI agent workflows compared to traditional text-based shells like zsh or Bash. The study involved 380 total runs across pipeline comparisons, A/B tests, held-out tasks, and a real-world aggregate case inspired by a master's thesis reconstruction. Rather than replacing zsh entirely, the author used Nushell selectively, guided by a tiered routing policy that assigns tools based on task complexity and data structure. Key pre-measurement failures were identified, including Nushell's ls omitting hidden files without the -a flag, which excluded nearly 64% of files in the test environment. The researcher concludes the findings are bounded exploratory evidence, not universal benchmarks, and the central question remains when structured routing helps, hurts, or makes no material difference to an agent's output.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How Changing a Denominator Can Flip a Property Tax Map Without Altering the Data

A data analysis using Chilean government records demonstrates how property tax maps can produce contradictory conclusions depending on which denominator is chosen — households, residents, or land area — without any change to the underlying figures. The study cross-references Chile's national real-estate cadastre from the tax authority SII with the IGVUST, a socio-territorial vulnerability index published by the Ministry of Social Development. The unit of analysis is the Chilean neighbourhood unit, a civic territory not designed to cover the entire country, meaning rural areas are excluded by default — a gap that can silently distort results. The author argues that while the arithmetic involved is straightforward, the problem lies in presenting one type of territorial indicator as though it answers a different question. The core finding applies beyond Chile: any jurisdiction that maps property assessments against deprivation measures faces the same methodological choices, and mislabelling them produces misleading narratives.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

PantryBridge Uses Google Gemini AI to Simplify Food Pantry Donations

PantryBridge is an open-source, AI-powered web app built for DEV Community's Generosity Edition Weekend Challenge that streamlines the food donation process. Users upload a photo of their pantry shelf, and Google's Gemini AI analyzes the image to generate a structured inventory of items, including quantities, dietary tags, and packaging condition. The app then helps users locate a nearby food bank via Feeding America's real locator, creates a recipe using the donated items, and writes a personalized kindness note to include with the box. A printable donation manifest with an itemized list is also generated for easy intake logging at drop-off. Built with Next.js, Tailwind CSS, and shadcn/ui, the app gracefully falls back to mock data when no Gemini API key is provided, ensuring it remains fully functional for demonstration purposes.

0 comments Read more at DEV Community