SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

Developer maps 50 web crawling capabilities to help govern AI agent access

0
·1 views

Developer Ajnas N B has published a framework mapping 50 capabilities of the open-source Cockroach Crawler tool across seven functional categories for governed web crawling. The guide addresses how AI agents should handle web access not as a single feature but as a stack of distinct decisions covering URL discovery, allowed destinations, data extraction, and resource limits. A core principle of the framework is that the agent's creator should set origin boundaries and resource ceilings, with model-facing inputs able to narrow but never expand those constraints. The article doubles as a practical checklist for developers using any crawler, asking them to define input contracts, output contracts, failure behavior, and authority boundaries before an agent relies on a capability. Cockroach Crawler is available under the MIT license, with the reviewed 0.7.0-rc.1 prerelease accessible via the next channel on npm.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

AR Finds Quiet Relevance in Education and Science After Metaverse Hype Fades

After the metaverse boom of 2021–2022 collapsed under failed investments and shifting attention toward generative AI, augmented reality has quietly persisted without fanfare. Unlike virtual reality, AR does not aim to replace physical space but layers digital information onto it, making it more practical for real-world settings like classrooms, labs, and hospitals. Developers found that accurate spatial anchoring on mid-range phones — rather than expensive headsets — is sufficient for meaningful AR experiences. A university project called AstroFlow demonstrated this using WebXR-based 3D visualization for astronomy and anatomy, requiring minimal specialized infrastructure. The key insight is that AR's value lies in solving a focused tracking-and-rendering problem, particularly for subjects where spatial relationships — such as molecular structures or orbital mechanics — are lost when flattened onto 2D media.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

AI Price Tiers Are Reshaping Software Architecture, But Most Devs Are Missing It

The ongoing AI price war has created distinct model tiers — cheap fast models for routine tasks and powerful frontier models for complex reasoning — which is fundamentally changing how developers should design applications. Routing requests to a single model regardless of task complexity now means either wasting money on simple jobs or underperforming on hard ones. Separately, million-token context windows are making retrieval-augmented generation optional for many use cases, requiring engineers to reassess whether they still need vector databases. The EU AI Act's high-risk provisions also became enforceable this month, mandating that chatbots identify themselves as AI and that synthetic media carry labels for anyone serving EU users. Together, these shifts mean modern AI architecture now demands decisions about model selection, task routing, cost, and regulatory compliance — a more complex problem than most current tutorials address.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Study finds median MCP server consumes 3,150 tokens per turn before any task runs

A developer who maintains an MCP server directory sampled 60 remote servers to measure how many tokens their tool listings consume before an AI agent begins any actual work. The median server sent back enough JSON to cost roughly 3,150 tokens per turn, with the top quarter exceeding 14,000 tokens each. Notably, token cost correlated with schema verbosity rather than tool count — a 54-tool server proved two and a half times more expensive than an 84-tool one. Because tool lists are resent to the model on every conversation turn, a 20-turn chat with four average servers could burn around 252,000 tokens on capability descriptions alone. Nearly half of the sampled servers failed to respond to a basic discovery call, raising additional concerns about the reliability of MCP directory listings.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Developer releases OTI, an open-source one-time secret sharing app built with AdonisJS 6

A developer named Oguzhan Karacabay has released OTI, a free open-source app that lets users share messages or small text files via a one-time-use link. The message is encrypted in the browser, with only encrypted data stored on the server and the private key embedded in the link itself. Key features include password protection, QR sharing, encrypted .txt file support up to 100 KB, an expiry countdown, and a confirmation step to prevent chat previews from accidentally consuming the link. The project was built using AdonisJS 6, TypeScript, MySQL, Redis, and Vite, with an AI coding assistant used for parts of the implementation. The code is MIT licensed and available on GitHub, though the developer notes it has not undergone a professional security audit.

0 comments Read more at DEV Community
Developer maps 50 web crawling capabilities to help govern AI agent access · ShortSingh