How to Build a Crawl Budget to Keep AI Web Agents Efficient and Predictable

·3 views

AI agents that access the web in production environments often face latency spikes, runaway costs, and repeated requests without proper resource controls. A crawl budget acts as a control system that determines which pages deserve attention, how much effort each may consume, and when an agent should stop crawling. Developers are advised to define a task contract upfront, specifying required evidence, freshness needs, source count, latency limits, and cost per task before setting any crawl limits. URLs should be prioritized using signals like query relevance, host authority, content type, and expected cost, rather than treating all discovered links equally. Budgets should operate on three layers — task, host, and page level — to prevent any single domain or stubborn URL from exhausting the agent's overall resource allowance.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

SQL Basics: A Beginner's Guide to DDL, DML, Filtering, and CASE WHEN

A new tutorial on DEV Community walks beginners through the foundational concepts of SQL using a school database as a practical example. The guide covers Data Definition Language (DDL) commands such as CREATE TABLE and ALTER TABLE, which are used to structure a database before any data is entered. It then explains Data Manipulation Language (DML) operations — INSERT, UPDATE, and DELETE — for populating and maintaining records, highlighting the importance of using WHERE clauses to avoid unintended bulk changes. The article also details filtering techniques using operators like BETWEEN, IN, LIKE, and logical connectors AND and OR to retrieve precise data with SELECT queries. The tutorial aims to help learners understand not just SQL syntax but the reasoning behind each concept, making it a practical starting point for those new to database management.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Angular Adapter Lets Developers Build Terminal UIs Using Familiar Angular Patterns

A new open-source package called @cyia/opentui-angular has been released, enabling Angular developers to build terminal user interfaces (TUIs) without dealing with low-level rendering details like ANSI escape sequences or character buffers. The adapter connects Angular's component system to OpenTUI, a terminal UI framework previously tied to React, by replacing React's rendering engine with Angular's own. It registers OpenTUI's built-in components — such as boxes, text, and inputs — as Angular custom elements, allowing developers to use them directly in Angular templates. The package also rewrites OpenTUI's hook APIs using Angular's inject(), effect(), and computed() functions, and includes a dedicated keymap subpackage for declarative keyboard shortcut bindings. Custom renderables and plugin slots are also supported, giving developers full flexibility to extend terminal interfaces using standard Angular patterns.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Build a Scalable A/B Testing Experiment Service from Scratch

A software developer documented the process of designing an experiment service capable of running A/B tests at scale, outlining five core components: client signup, experiment configuration, variant assignment, event tracking, and reporting. The system allows businesses to split website traffic across multiple variants with configurable weights, then measure conversions against exposures to determine a winner. A key architectural constraint identified is that the variant assignment API must respond in milliseconds since it is called on every page load, making it the sole latency-critical path in the entire system. Each component is tied to a distinct API endpoint, and multi-tenancy is supported so multiple clients can manage their own experiments independently. The article serves as a practical system design walkthrough, covering both the conceptual vocabulary of experimentation and the technical trade-offs involved in building reliable testing infrastructure.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer finds LLM is fastest link in real-time AI pipeline, not the bottleneck

A developer building LiveSuggest, a browser-based real-time meeting assistant, instrumented his entire AI pipeline in production to find where latency was actually coming from. After a week of metrics, he discovered that the language model — long assumed to be the slow part — delivered a first token in roughly 1.2 seconds at the median, using both GPT-4o-mini and Claude Haiku 4.5. Speech-to-text transcription was faster at the median (around 75ms) but spiked to nearly nine seconds at the 95th percentile due to how silence-based sentence-ending detection handles non-stop speakers. A gating layer that decides whether enough new content exists to warrant a suggestion turned out to be the most impactful hidden cost in the pipeline. The findings challenged the developer's assumption that fighting LLM token latency would dominate optimization efforts, pointing instead to tail-end STT behavior and prompt-construction logic as the real culprits.

0 comments Read more at DEV Community