SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

Googlebot Saw Empty Tables on a JavaScript Site Due to Two Technical Bugs

0
·2 views

A team running market-data tools built on JavaScript found that most of their pages showed "Crawled - currently not indexed" in Search Console and initially blamed content quality. Crawl stats revealed that 96% of Googlebot requests were 4XX errors, caused by embeddable widgets on third-party sites repeatedly posting to a script-only endpoint about 80,000 times a day. A robots.txt rule blocking that endpoint resolved the waste. A separate bug was found when a bot-filtering proxy rejected Googlebot's renderer, which lacks typical browser headers, leaving tables empty. The team verified crawlers using published IP ranges with DNS as a fallback, and recommended caching, distinguishing failed lookups from negative results, and testing live URLs repeatedly.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Browser-based screen recorder Vidsay offers free Loom-style capture with no install or account

A school teacher from Pakistan built Vidsay, a free screen recorder that runs entirely in the browser and captures screen, webcam, and microphone at 720p. Recordings are saved as .webm downloads and kept in a local browser library, with no uploads, servers, or account required. The free tier allows 25 videos of up to five minutes each, and a mobile mode records from the device camera. Unlike Loom, it supports download-only sharing, since it has no servers for hosting shareable links, and its developer warns that .webm playback can vary across phones. The project is unofficial, unaffiliated with Loom, and its creator is seeking user feedback on performance and on the unclear Pro tier.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

SnapByFace Uses AI to Locate a Specific Person Across Large Photo and Video Collections

A developer has created SnapByFace, an AI-powered face search tool for finding a particular person in large libraries of photos and videos. Users import their local media, choose a face to look for, and the system analyzes faces across the files to surface matches. The tool can also pinpoint relevant moments within videos. The developer says it is aimed at photographers, parents, teachers, and event organizers who otherwise must review thousands of images by hand. The creator built it after finding that searching for a specific individual remains largely manual in standard photo management software.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Claude Code 2.1.316 adds subagent effort setting, ultrathink kill switch, and Opus fast mode deprecation

Anthropic's Claude Code release 2.1.316 reached npm on the morning of its publication, according to a developer's write-up on DEV Community. The update introduces CLAUDE_CODE_SUBAGENT_EFFORT, which lets users set the effort level that subagents run at instead of inheriting the parent session's level. A CLAUDE_CODE_DISABLE_ULTRATHINK switch turns off the ultrathink keyword, which now runs a turn at the highest effort level. Fast mode for Opus 4.8 is deprecated as of October 30 and will be removed on November 30, after which requests will return errors rather than falling back to standard Opus 4.8. The changelog names Opus 5.5 as the migration path for affected users.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

AI Tools Won't Replace Web Developers Anytime Soon, Analysis Argues

Fears that AI will make web developers obsolete are growing as AI tools advance and are marketed as all-encompassing solutions. The article points to a reverse-engineered replica of Adobe's interface, which looked flawless on the surface but proved unmaintainable because of poor architectural choices made without expert oversight. It also describes non-technical managers deploying AI-generated code that becomes tangled and hard to own, leaving developers to fix bugs. Drawing on the shift caused by WordPress, the author argues AI will likely augment developers by handling repetitive work while humans manage complex problems. The piece advises developers to learn AI toolchains and understand their limits rather than ignore them.

0 comments Read more at DEV Community