SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

72-trial benchmark shows MCP tool output format sharply affects AI agent performance

0
·1 views

A contributor to the CNCF Jaeger MCP server ran a structured 72-trial A/B benchmark to settle a design debate over whether MCP tools should return pre-aggregated summary rows or raw per-bucket time series data. The experiment used two production-grade AI agents — Claude Sonnet and Gemini 2.5 Pro — across six troubleshooting tasks, with three tasks designed to favor each format. Results showed that agents given time series data answered nearly all questions correctly, while those given summary rows declined to answer temporal questions seven times more often. Agents did not produce wrong answers when underfed with data — they correctly identified the limits of the information and refused to commit. The findings suggest that output format, not just token cost, is a critical and measurable factor in MCP tool design.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Developer cuts Go microservice release time 15x by migrating six repos to a monorepo

A software developer spent over an hour manually cutting a release across six Go microservice repositories, juggling dependency ordering across 18 branch operations before a missed dependency broke a build at 11pm. To eliminate the problem, they built an automated release pipeline using Jenkins, Python, and GitLab, running entirely on a local MacBook with ngrok tunneling webhooks to Jenkins. The six Go modules follow a strict dependency chain requiring sequential tagging for some services and parallel processing for others, making manual releases error-prone. After automating the multi-repo workflow, the developer identified the fragmented architecture itself as the root cause and consolidated all six repositories into a single Go monorepo using Go workspaces. The migration reduced release time by 15 times compared to the original manual process, with the full setup reproducible locally and available for others to fork and test.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

EU AI Act Article 50 Now Enforceable: How to Build Compliant Content Provenance

The EU AI Act's Article 50 transparency requirements became enforceable on August 2, 2026, obligating developers shipping generative AI features to EU users to embed machine-readable markings on AI-generated outputs. Regulators and the EU Code of Practice require at least two layers of provenance: signed metadata via the C2PA standard and an imperceptible pixel-level watermark such as Google's SynthID. A single C2PA manifest is insufficient in practice because platforms like X and most CDNs strip metadata on upload or optimization, and screenshots destroy it entirely — a limitation Microsoft acknowledged in its February 2026 Media Integrity report. Invisible watermarks embedded in pixel data survive these operations and complement C2PA signatures, which carry richer structured information including model identity and generation timestamps. Developers using Google's Imagen or Veo receive SynthID watermarking automatically, while those running open-source models such as Stable Diffusion must implement an equivalent watermarking scheme independently.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Developer Builds Team Management Platform Using Vanilla JavaScript and Firebase

Brazilian web developer Pedro Queiroz has shared his self-built project called Connecta, a team management and corporate communication platform with gamification features. The application was built entirely with vanilla JavaScript — deliberately avoiding frameworks like React or Vue — paired with Firebase for backend services including authentication and Firestore. Building without a framework taught Queiroz the importance of manual state management, folder structure, and separation of responsibilities. He also gained deeper insight into Firestore's NoSQL real-time model and implementing role-based access control across both front-end and database security rules. Queiroz plans to add real-time notifications as a next step for the project.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

MailKite launches Symfony Mailer transport package covering Mautic, PrestaShop and Drupal

MailKite has released mailkite/symfony-mailer, an open-source package that registers a native Symfony Mailer transport via a single DSN string (mailkite+api://). The package enables any platform built on Symfony Mailer — including Mautic, PrestaShop 8+, Drupal, and plain Symfony apps — to route email through MailKite's Send API without separate integrations. The transport logic was extracted from the existing mailkite/laravel package, which now depends on this shared library rather than maintaining its own copy. Version 0.1.0 is tagged on GitHub and the package is pending approval on Packagist before composer require becomes available. The release ships with 15 passing unit tests and requires no live network connection to run them.

0 comments Read more at DEV Community