SShortSingh.
Back to feed

Making JavaScript-rendered pages crawlable with build-time pre-rendering and auto-noindex for thin pages

0
·8 views

A Korean information site generated as static HTML had sections drawn by JavaScript from a JSON file, leaving crawlers with almost no readable text. The author pre-renders that same JSON into the HTML at build time, so crawlers and users see equivalent content. Two further fixes followed: a lastmod date that changes only when page content actually changes, and a pass that marks thin generated pages noindex while keeping their links followable. The write-up includes the relevant build-script code and notes known weaknesses, such as the date-hashing approach ignoring fraction changes. Some interactive sections remain thin because their output depends on user input.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Banks and Publishers Restrict AI Agents as Identity Gaps Emerge

After customers began handing banking credentials to AI assistants, Citigroup and Bank of America moved to bar agents from client accounts, and Schwab began blocking outside AI tools that attempt to move money. Content publishers showed a similar shift, with Cloudflare's one-click AI crawler block adopted by over a million customers and the share of reputable sites blocking AI scrapers rising sharply from 2023 to 2025. Analysis of crawler traffic suggests much of it is training-related rather than live user activity, while robots.txt enforcement remains inconsistent across top domains. The article argues that current agents cannot prove who they are or who authorized them, leaving operators to rely on easily spoofed signals. Proposed remedies include cryptographically signed agent identities under RFC 9421 and Visa's Trusted Agent Protocol for commerce.

0
ProgrammingDEV Community ·

AIClaw Adds Opt-In Skill Access Tracking That Excludes Prompt and Tool Content

AIClaw's v3.7.3 release introduced an opt-in OpenTelemetry Logs event, aiclaw.skill_invocation, that records when an agent accesses a skill and how that access was triggered. The event classifies access as explicit, when a user mentions a skill and it loads, or implicit, when a skill loads or its SKILL.md is read without such a mention. Prompt text, skill bodies, and tool payloads are kept out of the exported events, and file paths are resolved locally. The tracker deduplicates repeated loads within a turn, so one turn can produce both an explicit and an implicit event for the same skill. The feature was released October 8, and the latest public version at the time of writing is v3.7.15.

0
ProgrammingDEV Community ·

ElevenLabs AI Voice Helps Real Estate Sites Add Narrated Virtual Property Tours

A developer guide explains how AI text-to-speech can add narration to property listings, making virtual tours more immersive and engaging. It describes how to get started with ElevenLabs, a TTS service with a free tier, by generating an API key and listing available voices. The article includes a Python script that converts a property description into an MP3 narration file via the API. It also covers voice cloning as a way to keep a consistent brand voice across listings. The approach is presented as a low-cost alternative to studio recording and hired narrators.

0
ProgrammingDEV Community ·

Fixing Stripe webhook 401 errors on Supabase Edge Functions by disabling JWT verification

A common cause of Stripe webhooks failing with 401 on a Supabase Edge Function is the platform's built-in JWT check, which rejects requests before the function's code runs. The article explains how to tell this apart from a 401 returned by the developer's own code, by inspecting the response body in Stripe's delivery log. The fix is to disable JWT verification for the webhook function only, either via config.toml or the --no-verify-jwt deploy flag, so that redeploys don't re-enable it. Because the endpoint becomes publicly reachable once this change is made, the author stresses that the Stripe signature must then be verified inside the function, since otherwise anyone who knows the URL could send forged events.