SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

GPTBot, ClaudeBot and More: How AI Crawlers Work and Why They Matter

0
·1 views

AI crawlers are automated bots that fetch web pages to serve two distinct purposes: training large language models like GPT and Claude, or retrieving live content to cite in real-time AI answers. Major crawlers active in 2026 include GPTBot and OAI-SearchBot from OpenAI, ClaudeBot from Anthropic, PerplexityBot from Perplexity, Google-Extended, Bytespider from ByteDance, and CCBot from Common Crawl. Training crawlers collect text in bulk to improve AI models but rarely send traffic back to the source, while retrieval crawlers fetch pages on demand and typically cite and link them in AI-generated answers. Website owners can identify these bots via user-agent strings in server logs and selectively allow or block them through robots.txt and llms.txt files. Because blocking a training crawler and blocking a retrieval crawler have very different consequences for visibility, experts treat the two decisions separately under the emerging practice of generative engine optimization.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Developer's AI Agent Completes 1,087 Tasks Autonomously in a Month, Sparking Job Displacement Debate

A developer building personal AI agent systems reports that his setup completed 1,087 tasks autonomously in a single month, with 88.1% of tasks requiring no human intervention and an average response time under four minutes. The system uses cron jobs, guards, and specialized skill workflows running around the clock, handling routine work such as nightly LinkedIn performance analysis. Observing that one person with such tools can match the output of multiple full-time employees, the developer raises concerns about the fate of workers in administrative, clerical, and coordination roles whose core tasks AI agents can already perform. The IMF estimates that 40% of global jobs could be affected by AI, and the developer argues the shift is already happening incrementally across organizations adopting automation. In an upcoming book chapter, he examines four policy responses under debate — robot taxes, value-added levies on automation, universal basic income, and reskilling programs — framing the discussion as a practitioner's honest reckoning rather than activism.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Open-Weight AI Models Narrow Gap With Proprietary Rivals, Reshaping Build Decisions

Open-weight AI models have significantly closed the performance gap with closed, proprietary models on general reasoning and coding benchmarks, making them viable for a wider range of practical applications. While closed frontier models still hold an edge on the most complex reasoning tasks, the difference is no longer the dominant factor in most build decisions. Teams with strict data-residency needs or high-volume workloads are increasingly considering self-hosting open-weight models, despite the added operational overhead compared to using a standard API. The competitive pressure from open-weight alternatives is also pushing closed model providers to reconsider their pricing. As a result, model selection is evolving into a continuous, workload-specific evaluation rather than a one-time architectural commitment.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Developer Shares Hands-On Amazon S3 Labs Covering Security, Encryption and Lifecycle

A developer published a practical walkthrough of Amazon S3 features commonly used in AWS storage workflows, focusing on implementation and verification rather than theory. The labs covered bucket configuration, lifecycle management, IAM and EC2-based bucket policies, SSE-KMS encryption with CloudTrail verification, pre-signed URLs, CLI sync, versioning, static website hosting, CORS, and Object Lock. Lifecycle rules were configured to automatically transition objects from Standard storage to Standard-IA after 30 days and Glacier Flexible Retrieval after 90 days. Bucket policies were tested to allow specific IAM user read access, EC2 role upload access, and to explicitly deny all non-HTTPS requests using the aws:SecureTransport condition. The post serves as a companion to two earlier theoretical S3 sessions and emphasizes real-world configuration, troubleshooting, and behavioral verification.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

How Toxic Workplace Practices Undermine Productivity and Team Collaboration

Large organizations increasingly suffer from toxic work cultures driven by competitive performance systems, punitive processes, and poor leadership practices. Mechanisms such as stack ranking, misused performance improvement plans, and excessive time tracking shift employee focus from collective goals to personal survival. The integration of AI tools with unrealistic benchmarks further fuels burnout and disengagement by making workers feel replaceable. These overlapping pressures erode trust, stifle innovation, and accelerate staff turnover, raising recruitment costs and depleting institutional knowledge. Experts argue that shifting from competition-driven to collaboration-focused management models is key to restoring long-term organizational health and productivity.

0 comments Read more at DEV Community
GPTBot, ClaudeBot and More: How AI Crawlers Work and Why They Matter · ShortSingh