Allowed in robots.txt but blocked by the server? A Python script checks AI crawler access

·4 views

A robots.txt file that permits GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot shows only what a site allows, not what its server actually returns. A WAF or CDN rule can answer a crawler with a 403 even when the file welcomes it. The author shares a Python script that checks both layers, comparing each bot's robots.txt permission against the HTTP status returned when the request carries that bot's user agent. In a test against a local server, OAI-SearchBot received a 403 despite being allowed, which the script flags. The author cautions that the check sends requests from the tester's own IP, so real crawlers verified by IP or reverse DNS may see different results, and recommends confirming with server logs.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Krea 2 cleanup in ComfyUI: one shared recipe across three input and output setups

Three Krea 2 cleanup workflows in ComfyUI follow the same skeleton: reverse-prompting the input image, rebuilding it through three depth-structure passes with one shared LoRA, then upscaling with 4x-UltraSharp. The variants differ only in I/O, covering a single image, a local folder loop, and a RunningHub ZIP batch. All three use the same Krea 2 Turbo model, text encoder and VAE, with 8-step KSampler passes and a console that selects render stages, the final long edge, and the reverse-prompt route. The single-image graph is presented as the clearest example, and the author suggests a 2048-pixel long edge as the best value for cost, though this is guidance rather than a measured benchmark. The source also notes that the in-graph label on the batch variant is outdated, so the node graph should be trusted over the note text.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Per-customer budget caps for Claude API calls, with prompt caching priced correctly

A developer guide explains why Anthropic's console limits don't suit products serving many customers, since they cannot isolate one heavy account. It proposes a per-customer monthly spending cap for the Messages API, implemented in TypeScript with Redis. The key cost pitfall is that the usage field's input_tokens excludes cached reads and cache writes, which must be priced separately or costs are undercounted. The approach includes a price table per model, a cost function that splits 5-minute and 1-hour cache writes, and throws on unknown models so new ones are not silently priced at zero. The article notes that batch discounts, long-context tiers and server-tool fees are not modeled, so the resulting cap is a floor rather than a ceiling.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Tracking LLM spend per customer in Node.js using AsyncLocalStorage

A common problem is that an OpenAI invoice shows total spending but not which customer generated it, which matters for flat-priced plans. The article proposes attaching a customer ID to the request's async context once, through middleware, instead of threading it through every function. Developers can wrap the OpenAI client so each call reads that context and records its cost, with feature tags added via nested context scopes. Background jobs can start their own context tied to the relevant customer. Costs are converted to dollars at write time using a dated price table, with attention to cached input tokens billed at a lower rate.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

fold() vs reduce() in Kotlin: How the Starting Accumulator Shapes Android Code

The article contrasts Kotlin's fold() and reduce(), which both collapse a collection into a single value but differ in how they begin. fold() takes an explicit seed, so its result type can differ from the element type, while reduce() starts from the first element and requires a compatible type. The author illustrates this with a download-sync example that sums byte counts both ways and builds a transfer report from a list of parts. The piece also notes that reduce() suits merging same-shaped values, such as combining time windows into one covering window. It highlights failure modes like empty collections causing exceptions and overflow in byte totals, all tied to the choice of seed.

0 comments Read more at DEV Community