A single robots.txt rule cleared 957,000 ghost URLs left by a previous site owner

·1 views

A developer discovered that six weeks after launching a 66-page static Next.js site, Google had indexed only one page while tracking nearly 957,000 URLs on the domain. The domain had previously hosted an e-commerce store, and Google was still crawling hundreds of thousands of old product and query-string URLs that no longer existed. Because the static site returned a 200 status for every query-string variant of the homepage, Google flagged over 335,000 of them as soft 404s and spent its crawl budget on the defunct store instead of the new content. Adding a single robots.txt directive — 'Disallow: /*?' — blocked all query-string URLs from being crawled, freeing the crawl budget for the actual site pages. The developer also corrected an over-eager sitemap that stamped every page with the latest build time, which had caused search engines to treat all pages as perpetually updated.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Guide outlines how to build security-audited AI chat interfaces in TypeScript

A technical guide published on DEV Community details how frontend engineers can build production-grade AI chat applications in TypeScript that meet enterprise security standards. The article addresses growing threats such as prompt injection attacks, PII leakage, and supply chain vulnerabilities, which have raised the security bar for AI-facing interfaces in 2024. It walks through a full development lifecycle using Vercel's assistant-ui library alongside a Next.js backend-for-frontend proxy, ensuring LLM API keys are never exposed directly to the browser. Key steps covered include strict TypeScript configuration, dependency hygiene with pnpm, context sanitization, PII redaction, and automated security scanning. The guide targets compliance with frameworks such as OWASP Top 10 for AI, SOC 2, and ISO 27001.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Obsidian Emerges as a Go-To Local Knowledge Base for AI Agents in 2026

Obsidian, a local markdown-based note-taking app, is gaining traction in 2026 as a personal knowledge management tool that integrates directly with AI agents like Claude Code. The app stores notes as plain Markdown files on the user's device, requires no cloud dependency, and has been free for commercial use since early 2025, making it attractive for privacy-sensitive workflows. AI plugins such as Smart Connections use Retrieval-Augmented Generation to let users query their entire note vault, while tools like Ollama enable fully offline AI interaction without data leaving the device. In January 2026, Obsidian CEO Steph Ango published the MIT-licensed 'obsidian-skills' repository on GitHub, which teaches AI agents to understand Obsidian's formatting conventions and quickly accumulated over 13,000 stars. A related open-source plugin, claude-obsidian, builds on this foundation by automatically generating linked wiki pages per source and maintaining context across sessions, reflecting a broader shift toward AI-augmented personal knowledge bases.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

WebAssembly Explained: Real-World Uses, Limits, and When to Skip It

WebAssembly (WASM) is a binary instruction format that serves as a compilation target for languages like C, Rust, and Go, allowing code to run in browsers at near-native speeds. It outperforms JavaScript for compute-heavy tasks such as numerical processing, cryptography, and media encoding, but offers no advantage for DOM manipulation or tasks already optimized by JavaScript's JIT compiler. Practical applications include running FFmpeg for in-browser video processing, executing SQLite databases locally, and encoding modern image formats like AVIF using codec libraries not yet natively supported by browsers. However, WASM comes with trade-offs: frequent calls across the JS/WASM boundary add overhead, large file sizes (FFmpeg's core weighs around 30MB) require careful lazy loading and caching strategies, and enabling multithreading demands specific HTTP security headers that can break third-party scripts. Developers are advised to treat WASM as a targeted tool for specific performance bottlenecks rather than a universal replacement for JavaScript.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Backstage Evolves Into AI Hub for Developers, Used by 3,400+ Organisations

At KubeCon + CloudNativeCon Europe in Amsterdam in March 2026, the Backstage developer portal community held its fifth BackstageCon, highlighting the platform's significant evolution since Spotify open-sourced it in 2020. Originally built at Spotify in 2016 to manage hundreds of microservices, Backstage is now a CNCF incubation project used by over 3,400 organisations serving more than two million developers, commanding an estimated 89 percent share of the developer portal market. The platform's biggest shift in 2026 is its deep integration with AI, including Model Context Protocol (MCP) server support that allows AI coding agents like Claude Code and Cursor to directly access the Software Catalog, Templates, and documentation. A new catalog entity type called AiResource, introduced in v1.54, makes AI models and agents first-class citizens within the portal. Spotify reported that its internal AI Knowledge Assistant, connected via MCP, cut developers' time spent on answerable routine requests by 47 percent, underscoring the practical productivity gains of the AI-integrated approach.

0 comments Read more at DEV Community