How to check whether robots.txt is blocking Google, and fix it safely

·7 views

A single line in a robots.txt file can stop Google from crawling an entire site, and since pages still load normally for visitors, the problem often goes unnoticed until search traffic drops. The file must be checked at the root of each host, since subdomains and protocol variants can serve different versions. HTTP status codes matter: a 200 response is followed, a 404 or 410 means no restrictions, and server errors can cause Google to pause crawling. Rules are grouped by user agent, and Googlebot follows its own most specific group, so an old bot-specific rule can silently block Google. Google's Search Console robots.txt report and URL Inspection tool help confirm what is blocked, while blocking in robots.txt prevents crawling but not necessarily indexing.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Developer open-sources four MIT-licensed tools for AI-driven e-commerce work

Naser Sadrmanesh, an AI product engineer and store founder, has released four small open-source projects, all under the MIT license. The tools were built from utilities he repeatedly needed for client online stores. One is llm-search-eval, which compares search system changes using a shared label pool, LLM judges, and statistical tests. Another is woocommerce-mcp, which gives AI agents access to WooCommerce stores with read-only defaults, preview-and-confirm writes, and audit logging. A third, commerce-agent-skills, is a library of 29 Markdown skills that guide coding agents toward correct e-commerce implementation details.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Debugging a failing Docker container by committing it and overriding the entrypoint with a shell

A developer using Docker Compose found that a Django container kept exiting shortly after launch, showing exit code 127 in the docker ps output. To investigate, they committed the failed container to a new image so its state could be preserved. They then started that image with the entrypoint overridden to launch an interactive shell, using /bin/bash or /bin/sh if bash is unavailable. This gives access to run diagnostic commands inside the container's environment, with the specific checks depending on the problem.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Planning Active Directory Protected Users membership and managing its exceptions

Microsoft's Protected Users group applies fixed protections against credential theft, such as blocking credential caching, NTLM and Digest delegation, and limiting Kerberos to newer encryption. Because these restrictions cannot be toggled per account, adding a user is a compatibility decision as well as a security one. The article recommends first inventorying authentication dependencies through logs, then running a small pilot that tests interactive sign-in, administration, Kerberos tickets and delegation. Organizations should keep a tested break-glass account, record approvals and rollback steps, and review membership regularly so the group stays small and current. Protection only applies on supported hosts and domain controllers, so the platform should be verified before reporting an account as protected.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Treat AI Agent Tools Like Public APIs to Prevent Duplicate Actions

A developer's first production AI agent issued the same refund three times after a payment API returned a 502 error and the model retried the call. The root cause was a missing idempotency key, a bug familiar from standard HTTP client code rather than a language model problem. The article argues that the model should be treated as an unpredictable API caller, so tools need the same safeguards as public APIs, including versioning, timeouts, idempotency, scoped credentials, and rate limits. It recommends classifying every tool as read, write, or destructive, with that classification enforced in code rather than documentation. The author also advises generating idempotency keys in the application, since models will retry calls for many reasons.

0 comments Read more at DEV Community