SShortSingh.
Back to feed

Token Cost of AI Tool Discovery: When Agent Context Overhead Actually Matters

0
·83 views

A developer measured the token cost of injecting tool definitions into an AI model's context on every call, using a tokenizer-based harness without requiring a live LLM. The study found that a small three-tool server costs just 277 tokens per model call, making optimization pointless at that scale. However, costs scale linearly with tool count and description verbosity — 100 verbose tools consume over 20,000 tokens per call, totalling roughly 400,000 tokens across a 20-call agent task. Curating five relevant tools from a set of 100 cuts context cost by approximately 95 percent. The key finding is that the real cost driver is not discovery versus static manifests, but how many tool definitions sit in context and how verbosely they are written.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Developer Shows How to Turn a Web App Into an iPhone App Using Floot and TestFlight

A developer building a holiday gift finder app used the Floot platform to create a web app that uses an LLM and trend data to suggest gifts. The app was then expanded with location, camera, and notification features. The developer explains how the web app was converted into a native iPhone app that carried over those features. The tutorial also covers using Floot's App Store Connect integration to upload the app to TestFlight for beta testing. The guide provides a full prompt for recreating the gift finder app.

0
ProgrammingDEV Community ·

Developer builds NewsNinja RSS reader to show each news story only once

After repeatedly seeing the same story echoed across many outlets on his morning commute, a developer built NewsNinja, an RSS reader designed to group duplicate coverage. Version 1.0 was built in roughly ten days and reached the App Store on June 21, 2024, supporting iPhone, iPad, and Vision Pro. Its core features included grouping articles about the same event, opening pieces in reader mode, and AI-generated summaries as the sole paid feature. The developer first tried AI models to group headlines but found them slow and unreliable, so switched to fast string similarity algorithms that run on-device. The app then shipped frequent small updates, adding languages and features like OPML import in the months after launch.

0
ProgrammingDEV Community ·

Staged retrieval and safe deletes for keeping engineering docs search fresh

A technical guide argues that retrieval over internal engineering documentation should use staged queries, explicit collections, and traceable source context. It recommends treating document deletion as a versioned state transition, using tombstones and revision checks so stale or deleted content is not served. The article describes bounded read paths with timeouts, capped candidates, and exponential backoff on rate limits, plus idempotent consumers for at-least-once event delivery. It also frames freshness as a policy, with re-indexing on publish and periodic reconciliation against the source of truth. The piece mentions a commercial API option for vector search but centers its advice on the retrieval contract and includes a minimal Go example.

0
ProgrammingDEV Community ·

Marketplaces Need Auditable Evidence Behind Email Suppression Decisions

A marketplace must stop welcome emails to invalid seller addresses without assuming every delivery failure proves an address is invalid. The article argues that a simple suppression flag cannot explain why a recipient was excluded, so teams should keep the address revision, delivery evidence, interpretation, and policy version. Source event time and application receipt time should be stored separately to allow accurate reconstruction. Temporary or unresolved failures must not be treated as invalid-address findings, and missing evidence should be recorded as a gap. The author notes that such auditability does not by itself ensure legal compliance, which requires separate review.