Developer's AI Orchestrator Burned Millions of Tokens Due to Stacked Design Flaws

·1 views

A software developer building an orchestration system on top of Claude Code discovered it was consuming 1–2 million Opus tokens per task, even for minor code changes. The excessive cost stemmed from three compounding factors: all subagents defaulted to the most expensive Opus model tier because no model parameter was explicitly set, each phase spawned fresh subagents that bypassed prompt caching and incurred cold-write costs, and a mandatory five-phase pipeline triggered on virtually every actionable request. The developer noted that the Opus-to-Haiku cost difference is 5x, while Opus-to-Sonnet is only 1.7x, meaning model selection was actually the smallest of the three cost multipliers. Prompt cache misses — caused by slightly different system instructions per phase — proved to be the most significant driver, as the same repository context was re-loaded and billed at full price each time. The postmortem concludes that correctness-focused design rules, when left unexamined through a cost lens, can silently produce extreme token expenditure at scale.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Developer cuts Chromatic visual testing costs tenfold by consolidating story snapshots

A software developer sharing on DEV Community reduced their visual testing bill by up to ten times by rethinking how Storybook stories are structured for snapshot tools like Chromatic and Percy. The core problem was that rendering every component variant, state, and theme as a separate story generated a large number of snapshots, since most per-snapshot tools bill by the number of stories rendered. The fix involved collapsing related variants — such as sizes, states, and themes for a single component — into one consolidated 'AllVariants' or 'AllStates' story that renders a grid of permutations in a single snapshot. The developer cautions that this approach requires balance: merging a single component's internal matrix works well, but independent pages or unrelated flows should remain as separate stories to avoid coupling unrelated visual diffs. The technique is described as tool-agnostic and applicable to any platform that charges or tracks on a per-snapshot basis.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Properly A/B Test AI Avatars Against Text Chat for Conversion Rates

A growing number of AI avatar platforms claim that voice and video avatars convert better than plain text chat, but rigorous independent testing of this claim remains rare. Unlike standard A/B tests, comparing avatars to text chat involves multiple simultaneous changes — interaction modality, response latency, and visual layout — making it harder to isolate the true variable. Experts recommend holding the underlying AI model and knowledge base constant across both variants, while tracking granular metrics such as engagement friction, session depth, and lead quality rather than aggregate conversion rate alone. A notable pattern observed is that avatar variants can drive longer sessions without necessarily improving completed-lead rates, a nuance that overall conversion figures would obscure. Vendor case studies typically benchmark avatars against having no chat widget at all, not against equivalent text chatbots, making self-run controlled tests essential before committing to a premium-priced avatar solution.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

EvalPort Introduces 11-Grader System for Framework-Agnostic LLM Evaluation

EvalPort, a new LLM evaluation platform, has released a grader system featuring 11 distinct types designed to cover the majority of real-world evaluation needs. Each grader type — ranging from exact_match and regex to llm_judge and json_schema — carries its own parameters, model references, and thresholds, making eval suites self-describing. The system was built to be framework-agnostic, meaning any evaluation framework can implement a subset of the grader types without compatibility issues. Test cases reference graders by ID, and multiple graders can assess the same test case, with scores recorded separately in a ResultSet. A custom grader type also serves as an escape hatch for use cases not covered by the 11 built-in options, and SDKs are available for both Python and JavaScript.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

TryHackMe CTF Exposes Azure Cloud Privilege Escalation via Hardcoded SAS Token

A TryHackMe capture-the-flag challenge called CryptoCabana simulates a real-world Azure cloud compromise stemming from poor security practices. Participants begin by discovering a hardcoded SAS token with read and list permissions embedded in client-side JavaScript on a static Azure website, granting unauthorized access to the storage account. Enumerating the storage containers reveals a hidden vault container holding a service principal credential file, which is then used to authenticate directly to Azure. From there, contestants explore an Azure Key Vault containing four secrets, including three key shards and a protected master key, with the flag reconstructed by retrieving older versions of a rotated shard. The challenge highlights critical cloud security risks such as long-lived tokens, publicly exposed credentials, and insufficient secret rotation practices.

0 comments Read more at DEV Community