Is GPTBot Silently Blocked by Your Cloudflare WAF? How to Test and Fix AI Crawler Access

·2 views

You launch a site, set up your `robots.txt` to welcome AI search bots, and move on: txt User-agent: GPTBot Allow: / A few weeks later, you check your server access logs or wonder why your content never gets cited in ChatGPT Search or Perplexity. You find zero crawler hits. The culprit is almost never your robots.txt. It is usually an edge security rule or web application firewall (WAF) blocking the crawler before the request ever touches your origin server. Here is what is happening under the hood and how to test it.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

OneInbox: a fine-tuned 4B model that reads my inbox so I don't have to

What I Built OneInbox is a daily email digest that sorts my inbox into four buckets: Action required: needs a reply, or has a deadline within 7 days Worth a look: useful, but nothing to do FYI: receipts, confirmations, security notices Noise: promos, newsletters, social The idea came from a complaint in a hackathon Discord about how tiring it is to keep checking inboxes. I have the same problem, so I built it for my own inbox. The core is a LoRA fine-tune of Qwen3.5-4B (Apache 2.0). Each morning it reads the last 24 hours of mail and produces a page I can read on my phone in under a minute. Ev

0 comments Read more at DEV Community

ProgrammingDEV Community ·

StudySprint

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend I built StudySprint, a collaborative study planner web application designed specifically for a friend who is currently tackling a heavy load of university coursework. They often struggled with managing deadlines across different subjects and lacked a unified system to practice active recall while managing their time. StudySprint solves this by integrating: A dynamic Subjects & Tasks Manager to keep track of upcoming exams and assignments. A built-in Pomodoro Timer with "Focus", "Short Break", and "Long Break" mode

0 comments Read more at DEV Community

ProgrammingDEV Community ·

BreakMyQuery: find the row that breaks your SQL

BreakMyQuery: find the row that breaks your SQL This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend. I built BreakMyQuery for a colleague(also a friend🫠) who is a new graduate hire and is struggling with SQL joins or let's say he's scared of SQL joins. A query can pass the sample data while quietly dropping a customer who has no orders. For someone learning joins, seeing the missing customer gives them a concrete place to start investigating. BreakMyQuery offers eight SQLite exercises covering joins, NULLs, aggregation, ties, arithmetic, dates, and anti-joins.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Study Tracker

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend I built StudyBuddy — a focused, aesthetic, all-in-one study companion designed for my friend Maya, who is preparing for intense software engineering certification exams while juggling a full-time job. Maya was constantly struggling with cognitive fragmentation: She used one app for a Pomodoro timer (filled with intrusive ads and subscription banners). Another app for flashcards (which charged monthly fees just to generate quizzes from decks). A messy notebook to track her daily study streaks. Constant switching be

0 comments Read more at DEV Community