Why ChatGPT Couldn't Read a Blog: A Server-Level 403 Block Hidden Behind robots.txt

·6 views

A blogger tested whether ChatGPT could access their website after noticing it never cited their posts. When asked to open a page, ChatGPT reported a 403 Forbidden error, indicating the site was blocking it. Checking robots.txt revealed no restrictions, so the author used curl with different user agents to test the server directly. Results showed that Google and Bingbot were allowed while OpenAI's crawlers, Claude, and Perplexity, along with some SEO tools, received 403 responses. The author concluded that a server-level rule, rather than robots.txt, was responsible for turning these bots away.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Sole Trader Builders Face Quarterly HMRC Reporting as Digital Receipts Become Necessary

Sole traders and landlords with qualifying income over fifty thousand pounds began Making Tax Digital for Income Tax on 6 April 2026. They must keep digital records of income and expenses and send quarterly updates to HMRC through compatible software instead of a single annual return. The threshold drops to thirty thousand pounds in April 2027 and twenty thousand pounds in April 2028, and it applies to turnover rather than profit. The rules require each transaction to be recorded digitally, which often means photographing receipts or forwarding invoices to bookkeeping apps. Suppliers serving trade customers are responding by offering emailed itemised receipts, automatic invoice delivery and importable transaction exports.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

AI Safety Agent Shows Gains From Memory in Drug Trial Review Demo

TrialSleuth AI is an agent that reviews a new drug before its first human trial and draws on past results. In a demo, a made-up cancer drug called HLV-772 was reviewed first with memory switched off, producing a generic answer without specific evidence. With memory enabled, the same review flagged concrete risks tied to recalled past studies, including liver impairment, drug interactions, and dosing timing. A new heart-safety finding was then added to memory, and the next review included a heart-monitoring recommendation without any code or prompt changes. The app presents flagged risks for human experts rather than declaring a drug safe.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Ridurre il peso di un PDF scansionato con il downsampling delle immagini in Python

Un PDF creato da foto di telefono è pesante soprattutto perché ogni pagina contiene un JPEG ad alta risoluzione disegnato su una superficie molto più piccola. Per alleggerirlo occorre individuare le immagini, calcolare la loro densità effettiva in pixel per pollice, ricampionarle a un valore adeguato e riscriverle nel file. L'autore propone una soluzione in circa trenta righe di Python con PyMuPDF e Pillow, testata su un contratto di sei pagine di oltre 30 MiB. Il controllo iniziale mostra che le immagini costituiscono quasi l'intero peso del file, mentre testo e struttura ne occupano una quota trascurabile. L'articolo confronta inoltre i risultati con quelli ottenuti dall'editor PDF di un'app desktop sullo stesso file.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Why AI agents need separate working and episodic memory layers

AI agents are stateless, so anything they remember between sessions must be stored outside the model or placed in its context window. Simply enlarging context windows does not solve the problem, since research from Anthropic and Chroma shows recall and performance degrade as input length grows. Working memory is short-lived and must be managed through trimming or summarization, each with its own trade-offs. Episodic memory lives in external stores and is retrieved on demand, with the choice of store depending on how the agent needs to recall information. Approaches like the Berkeley paging model and consolidation of stored episodes into summaries help manage what moves between layers.

0 comments Read more at DEV Community