Developer builds tool to detect live robots.txt changes injected by CDNs like Cloudflare

·1 views

A developer discovered that the robots.txt file stored in a site's repository may differ from what crawlers actually receive, particularly when CDNs like Cloudflare inject their own rules at the edge. Cloudflare's AI Crawl Control feature can rewrite robots.txt in transit without modifying the origin file, meaning site owners may be unaware of what Googlebot or AI crawlers like GPTBot are actually seeing. Finding no existing tool that checks the live, served version of robots.txt — only generators for building static files — the developer built an open-source utility called robots-check. The tool can be run via npx with zero dependencies and detects whether the live file contains CDN-injected rules. It also includes a CI flag to fail builds if a site is accidentally fully disallowed, and a GitHub Action for automated workflow integration.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Why CF7 to Trello Integrations Fail and How to Fix Each Cause

Integrations between Contact Form 7 and Trello frequently break due to a small set of recurring, poorly documented issues. Trello's authentication requires both an API Key and a separately generated API Token, and omitting or swapping either value will cause all requests to fail. A common configuration mistake involves entering a Board ID where a List ID is required, since both are visually identical 24-character strings but serve different API purposes. Additionally, many free CF7-Trello plugins only map a single form field to the card description, meaning data from other fields is silently discarded. Developers can avoid these issues by verifying credentials and IDs with direct API calls before connecting any form, and by manually building multi-field description strings using Markdown concatenation.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Static calculator site wallmath.com loads just 16 requests with zero ads

Developer-built calculator site wallmath.com was designed to function like a static document, with no user accounts, server round trips, or analytics. A headless Chromium audit on September 10, 2026 recorded only 16 network requests on the home page, with just two requests going to a single third-party host, Google's ad syndication server. All fonts are self-hosted as subsetted variable woff2 files, and all CSS and JavaScript are inlined directly into each page rather than loaded from external sources. Google Ads is present but configured with a flag that blocks ad requests from firing, ensuring ads only appear in two pre-selected locations and cannot be toggled remotely without code changes. A build-time verifier renders all 17 pages and checks 98 invariants on every deployment to ensure these constraints are consistently enforced.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

VMs, Containers, Serverless: How to Choose the Right Compute Model

Virtual machines, containers, and serverless are three distinct ways to run code, each defined by how much of the underlying environment an application carries with it. VMs bundle a full operating system, offering strong isolation and control but at the cost of size and slow startup times. Containers share the host OS kernel, making them lightweight, fast, and portable — the default choice for most modern services. Serverless platforms like AWS Lambda run code entirely on demand, requiring zero infrastructure management but offering less control and struggling with long-running workloads. In practice, most real-world systems combine all three, selecting each based on the specific needs of individual workloads rather than applying one model across the board.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

AI Gateways Are Becoming Essential Infrastructure for Managing Multi-Model Apps

As development teams increasingly rely on multiple AI providers simultaneously, managing API keys, costs, and safety checks across services has grown chaotic. An AI gateway is a centralised control layer that sits between applications and model providers, handling routing, cost attribution, rate limiting, caching, and security in one place. By routing simpler requests to cheaper models and reserving powerful ones for complex tasks, teams can significantly reduce token spend without rewriting logic across every service. The gateway also solves a persistent FinOps problem by attributing AI spend to specific features or teams, making costs visible and governable rather than buried in scattered invoices. While adding a gateway introduces a small latency overhead and an extra component to maintain, its consolidation benefits are increasingly seen as standard practice in production AI deployments.

0 comments Read more at DEV Community