Claude refused 41% of Stripe coding tasks in controlled AI benchmark test

·2 views

A developer building SDKProof, a tool that tests whether AI models generate up-to-date library code, discovered that Claude (claude-opus-5) refused 62 out of 150 Stripe-related coding tasks — a 41.3% refusal rate. The refusals were silent completion-level declines, not readable error messages, which initially caused them to be misclassified as successful outputs due to a bug in the testing harness. By contrast, the same model refused zero out of 100 tasks on Zod, a control library, confirming the pattern was specific to Stripe. Refusal rates varied sharply by task type: payment initiation was refused in all 10 trials, while refund and webhook tasks were rarely or never refused. The findings suggest AI models may apply inconsistent content or safety filtering to financially sensitive API operations, even when the tasks mirror official documentation examples.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Developer launches Toon Tone, a free browser-based cartoon color-memory game

A developer has built Toon Tone, a free, lightweight browser game designed to help users practice color memory by matching cartoon character colors. The game uses HSB (Hue, Saturation, Brightness) sliders instead of traditional RGB inputs, making color reasoning more intuitive. Players receive instant feedback as they attempt to replicate a given color using the sliders. The project requires no account and includes daily prompts along with shareable scores. The creator is seeking community feedback on the interaction design and whether HSB controls feel more natural for this type of casual warm-up game.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Solo Developer Builds AI App That Turns Dish Photos Into Home Recipes

A solo developer has built DishLens, a mobile app that lets users photograph a dish and receive a cookable recipe along with nutritional information. The project, built using React Native (Expo) for the frontend and a custom API backend, relies on Google Vision for dish classification, Anthropic Claude for recipe generation, and Edamam for nutrition data. The developer also engineered a companion service called DriveSync, which keeps a Pinecone vector database synchronized with recipes stored as Google Docs. Key technical challenges included building a robust image preprocessing pipeline — covering blur detection, EXIF stripping, and content moderation — to avoid passing poor-quality inputs to AI models. The entire project lives in a single monorepo and was developed without a team, with the developer noting that AI coding tools accelerated development but could not substitute for domain knowledge and engineering judgment.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Today's AI Cloud-Ops Agents Are Already Obsolete, and Vendor Roadmaps Prove It

AI agents used in cloud operations today are rapidly being outpaced by advances already shipping across major platforms, according to a developer analysis. AWS Bedrock AgentCore, Azure Foundry Agent Service, and Google Vertex have all rolled out managed long-term memory this year, replacing the stateless, session-blind agents most teams currently run. Standardization of tool protocols like MCP, context windows expanding to one million tokens, and platform-native governance features are four key axes along which current setups will soon look outdated. Crucially, these are not speculative roadmap promises — memory, extended context, MCP support, and agent observability tools are already shipping, just not yet universally adopted. When AWS, Azure, and Google converge on the same capabilities within a single year, the author argues, the shift from novelty to default tends to happen fast.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer builds self-documenting repo system to give AI agents persistent project memory

A software developer grew frustrated with AI coding agents losing all project context between sessions, forcing repeated re-explanations of architecture and decisions. To solve this, he restructured his codebase into a living knowledge base centered on two components: an AGENTS.md entry-point file and a modular agent-sitemap directory of Markdown documents. Most modern AI agent tools automatically read AGENTS.md on launch, giving any new agent or model instant access to project overviews, rules, data flows, and conventions. A built-in Documentation Sync Loop requires agents to detect, update, and confirm documentation changes before completing any task. The result is a stateless-AI workaround that lets developers switch models or open fresh sessions without losing accumulated project knowledge.

0 comments Read more at DEV Community