SShortSingh.
Back to feed

GPTBot and OAI-SearchBot: How to Set robots.txt Rules for AI Crawlers

0
·2 views

OpenAI operates two distinct crawlers: GPTBot gathers content that may be used to train models, while OAI-SearchBot indexes pages for ChatGPT search. Blocking one does not affect the other, so confusing them is a common robots.txt error. Site owners should also check CDN and firewall settings, since those can block bots by IP even when robots.txt allows them. A simple user-agent test with curl can serve as a first check, though it cannot detect IP-based rules. The llms.txt file, a proposed site summary for AI tools, does not replace these steps, and Google says it is unnecessary for its AI search features.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

AI-assisted work finds smaller sticky Kakeya set over F3 than classical construction

Researchers at Munim, Inc. report an explicit 30-point sticky Kakeya set in F_3^4, one point smaller than the classical 31-point construction. A colour-lifting step extends this to sticky sets of size 31 * 2^(n-4) - 1 for every n >= 4, so the classical set is minimal only up to dimension 3. Whether the best growth rate for sticky sets stays at 2 remains unresolved, and the authors have narrowed the search to tree structures of height at least 5. The best known lower bound for sticky sets is about 1.905^n. The results have not yet been checked by independent mathematicians, and the authors invite corrections.

0
ProgrammingDEV Community ·

Test-Time Compute: Spending More Inference Effort May Beat Training a Larger Model

Deep learning has traditionally improved models by training larger ones with more parameters, data, and compute. An alternative approach, test-time compute, allocates extra computation when a model answers a query, letting it reason longer, generate multiple candidates, or check its work. It takes two forms: sequential scaling, which lengthens a single chain of reasoning with revisions and backtracking, and parallel scaling, which samples many answers and selects the best via majority vote or a verifier model. Because a correct answer among many samples is increasingly likely, the verifier is often the real bottleneck. A 2024 study by Snell and colleagues found that on easy and medium problems, scaling inference compute outperformed scaling model size, though larger pretrained models retained an edge on the hardest tasks.

0
ProgrammingDEV Community ·

ClearTok Automates Bulk Removal of TikTok Reposts for Users

A developer article examines how browser automation can ease the burden of managing large TikTok repost histories. Users who have accumulated hundreds of reposts often face a tedious, repetitive process when removing them one at a time through the app's standard menus. The piece introduces ClearTok, a desktop browser extension that automates this removal workflow while keeping the user in control of which items are processed. The tool works within the user's existing signed-in session and follows TikTok's own Remove repost option rather than introducing a new interaction model. The article frames the approach as a usability question about making everyday social media management more efficient without sacrificing user oversight.

0
ProgrammingDEV Community ·

How to Use AI for Cover Letters Without Adding Facts You Did Not Earn

A guide argues that AI can help write a cover letter if it only organizes facts the applicant already supplies, such as through a question-and-answer interview process. Generic AI drafts tend to produce vague, interchangeable phrasing, and models may invent metrics when details are missing, creating false claims in a signed application. The author recommends strong, specific openers and removing identifying or confidential information, using placeholders for personal details. A nine-step prompt workflow is described, including identifying key skills, answering clarifying questions in brief notes, outlining, drafting under 300 words, and checking every number against the applicant's notes. The piece also advises reviewing employer AI-use policies before submitting.