Claude Outperforms Gemini on AI Agent Tests, Despite Higher Per-Token Prices

·6 views

A data-analysis agent used by a company's customers was run with the same tools and tests on two model pairs, Gemini and Claude. Claude passed 95% of customer evaluations versus Gemini's 82%, and produced about a third as many inconsistent results. Its response time at the 90th percentile was roughly 48 seconds per turn, compared with 161 seconds for Gemini, which made far more model calls. Although Gemini charges less per token, it consumed much more input per turn, so its cost came to about $0.139 per turn versus $0.067 for Claude. The author notes that results apply to this specific workload and that the evaluation system is still a proof of concept.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Adding Captions and Chapters to a Product Demo Video in an Afternoon

A product demo landing page autoplayed muted, and a hard-of-hearing teammate and analytics showed that most visitors never unmuted it. The author added accurate captions and chapter markers using only a text editor and HTML, without a video platform or paid captioning service. Captions were generated by uploading the MP4 to an audio-to-text tool and exporting WebVTT, with language set explicitly and speaker recognition disabled. A small Node script was used to flag cues exceeding two lines or 42 characters per line, and chapters were created as a separate VTT file with timestamps found by searching the transcript. The captions track is set to load by default in the HTML track element, and a clickable chapter list was built with JavaScript.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Audit Finds Anthropic's AI Bug-Finding Project Produced Far More Reports Than Fixes

An analysis of Anthropic's Project Glasswing, which aimed to responsibly disclose AI-discovered vulnerabilities, finds that its ledger lists 37,966 total findings but only 217 fixes. VulnCheck's seven-month assessment and a recomputation from a public tracker of credited CVEs (450 on file) show findings growing rapidly while fixes lag far behind. Maintainers such as curl's Daniel Stenberg rejected more than half of the reports they received, and only a small share of the findings have been made public. The analysis acknowledges that the project has produced some real, serious vulnerabilities, including two added to CISA's exploited-in-the-wild catalog. The central concern is that unreviewed machine-generated reports shift a heavy triage burden onto unpaid open-source maintainers.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

AWS Adds IAM Condition Keys to Tell Apart Requests Made Through Its MCP Servers

AWS's MCP Server, part of the Agent Toolkit for AWS, became generally available on May 6, 2026, and the server itself carries no charge, with users paying only for the resources they access. The article notes that developers using Claude Code on an existing subscription can run the workflow without Bedrock inference or an Anthropic API key. Its central claim is that AWS has introduced IAM global condition keys, aws:ViaAWSMCPService and aws:CalledViaAWSMCP, which identify whether a request passed through an AWS managed MCP server and which one. These keys let administrators grant the same role different permissions depending on whether the call originated from an AWS MCP server, so the identical credentials can be read-write in one context and read-only in another. The author presents this as a practical provenance mechanism, though the piece's framing suggests it is not a full containment boundary.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

osFoundry's Prompt Library keeps prompts versioned and turns them into reusable agent skills

osFoundry, a desktop app combining chat, agents, files and tools, introduces a Prompt Library that consolidates prompts in one place with search, favourites and usage counts. Users can save edits as new versions instead of overwriting, so earlier wording remains accessible, or duplicate a prompt to start an independent copy. Prompts containing placeholders such as {{product}} prompt the user for values before being inserted into chat. Proven prompts can also be converted into skills in Studio, which add a trigger, permitted tools, target agents and step-by-step instructions that agents apply automatically when a task matches. Tool permissions and approval policies, configured separately in the Tools tab, determine what an agent can actually do.

0 comments Read more at DEV Community