Evaluating AI APIs by Latency, Timeouts and Usable Output

·5 views

A business-side contributor at an AI API platform argues that demo performance is not enough to judge an AI model. The piece uses a support agent's "Draft reply" feature as an example, where slow responses leave the agent waiting. A timeout would leave the tool with no draft, forcing a retry or a manual reply. Before a team rollout, the author suggests testing the API with about 20 typical tickets. The test would track request duration, timeout count, and how many drafts need only light editing.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Adding Captions and Chapters to a Product Demo Video in an Afternoon

A product demo landing page autoplayed muted, and a hard-of-hearing teammate and analytics showed that most visitors never unmuted it. The author added accurate captions and chapter markers using only a text editor and HTML, without a video platform or paid captioning service. Captions were generated by uploading the MP4 to an audio-to-text tool and exporting WebVTT, with language set explicitly and speaker recognition disabled. A small Node script was used to flag cues exceeding two lines or 42 characters per line, and chapters were created as a separate VTT file with timestamps found by searching the transcript. The captions track is set to load by default in the HTML track element, and a clickable chapter list was built with JavaScript.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Audit Finds Anthropic's AI Bug-Finding Project Produced Far More Reports Than Fixes

An analysis of Anthropic's Project Glasswing, which aimed to responsibly disclose AI-discovered vulnerabilities, finds that its ledger lists 37,966 total findings but only 217 fixes. VulnCheck's seven-month assessment and a recomputation from a public tracker of credited CVEs (450 on file) show findings growing rapidly while fixes lag far behind. Maintainers such as curl's Daniel Stenberg rejected more than half of the reports they received, and only a small share of the findings have been made public. The analysis acknowledges that the project has produced some real, serious vulnerabilities, including two added to CISA's exploited-in-the-wild catalog. The central concern is that unreviewed machine-generated reports shift a heavy triage burden onto unpaid open-source maintainers.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

AWS Adds IAM Condition Keys to Tell Apart Requests Made Through Its MCP Servers

AWS's MCP Server, part of the Agent Toolkit for AWS, became generally available on May 6, 2026, and the server itself carries no charge, with users paying only for the resources they access. The article notes that developers using Claude Code on an existing subscription can run the workflow without Bedrock inference or an Anthropic API key. Its central claim is that AWS has introduced IAM global condition keys, aws:ViaAWSMCPService and aws:CalledViaAWSMCP, which identify whether a request passed through an AWS managed MCP server and which one. These keys let administrators grant the same role different permissions depending on whether the call originated from an AWS MCP server, so the identical credentials can be read-write in one context and read-only in another. The author presents this as a practical provenance mechanism, though the piece's framing suggests it is not a full containment boundary.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

osFoundry's Prompt Library keeps prompts versioned and turns them into reusable agent skills

osFoundry, a desktop app combining chat, agents, files and tools, introduces a Prompt Library that consolidates prompts in one place with search, favourites and usage counts. Users can save edits as new versions instead of overwriting, so earlier wording remains accessible, or duplicate a prompt to start an independent copy. Prompts containing placeholders such as {{product}} prompt the user for values before being inserted into chat. Proven prompts can also be converted into skills in Studio, which add a trigger, permitted tools, target agents and step-by-step instructions that agents apply automatically when a task matches. Tool permissions and approval policies, configured separately in the Tools tab, determine what an agent can actually do.

0 comments Read more at DEV Community