A single robots.txt rule cleared 957,000 ghost URLs left by a previous site owner

·19 views

A developer discovered that six weeks after launching a 66-page static Next.js site, Google had indexed only one page while tracking nearly 957,000 URLs on the domain. The domain had previously hosted an e-commerce store, and Google was still crawling hundreds of thousands of old product and query-string URLs that no longer existed. Because the static site returned a 200 status for every query-string variant of the homepage, Google flagged over 335,000 of them as soft 404s and spent its crawl budget on the defunct store instead of the new content. Adding a single robots.txt directive — 'Disallow: /*?' — blocked all query-string URLs from being crawled, freeing the crawl budget for the actual site pages. The developer also corrected an over-eager sitemap that stamped every page with the latest build time, which had caused search engines to treat all pages as perpetually updated.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

AI agents can use your browser session instead of your password, experts say

APX, a free open-source desktop app, lets users run their own AI agents locally, including for tasks on websites. The article argues that the key question is who signs in, which browser session is used, and what the agent may do afterward, rather than whether the agent can open a browser. Users are advised to start with narrow tasks, verify the site address before entering credentials, and complete one-time codes or CAPTCHAs themselves. APX offers a browser handoff feature so a person can take over the tab for sensitive steps, though this requires a desktop browser rather than a headless one. The piece also cautions that a successful login does not confirm the task was completed, and recommends signing out and reviewing permissions once access is no longer needed.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How BreathTime's petition system works: proposals, admin review and signing

BreathTime is a small civic platform where registered users can propose petitions, sign them and leave supporting comments. Any logged-in user submits a proposal with a title, description and addressee, which is saved as pending and reviewed by a human administrator who can approve or reject it, with rejection comments sent back to the author. Only administrators publish approved petitions, setting the goal (1,000 signatures by default), and each petition is signed once per user, enforced both in code and in the database. Signatures can include an optional public comment, and the platform openly lists current limitations such as no real-time updates, no pagination in the admin list, no audit log and no automated moderation.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Queue consumers need age-based alerts and idempotent sends to handle rate limits

A weekly digest pipeline can rely on a local token bucket placed before an idempotent send, backed by durable retries and a metric tracking the oldest queued work. Relying on queue depth alone is insufficient, since a consumer can appear healthy while missing the delivery window. The recommended approach is to record enqueue timestamps and alert when the oldest ready message exceeds the customer-facing delivery promise. Rate limiting decides when a request may start, while idempotency keys and durable state transitions prevent duplicate customer emails after timeouts. Teams should document the delivery window, normal drain rate, and maximum retry horizon in a runbook to set sensible thresholds.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Delivering Large CSV and PDF Exports to Browsers via Object Storage Presigned URLs

The article recommends generating large CSV or PDF exports outside the web request, storing the finished file as an immutable object, and letting browsers download it through short-lived presigned URLs. It argues that the application should manage authorization and export state, while object storage handles the byte transfer and the browser manages resumption. Separating these concerns prevents slow downloads from tying up application workers, rather than simply raising request timeouts. The author stresses that export readiness, URL expiry, and retry behavior must be tracked as distinct states, with retries using stable export identifiers and byte ranges to avoid duplicated work or mixing bytes from different object versions. The piece also notes that the exact validation method depends on the specific storage service and should be tested.

0 comments Read more at DEV Community