SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

How to verify AI crawler traffic in server logs using published IP ranges

0
·2 views

Major AI crawlers such as GPTBot, OAI-SearchBot, ChatGPT-User, and PerplexityBot appear frequently in access logs, but user agent strings can be easily faked. The companies behind these crawlers publish the IP ranges they use as JSON files, which makes verification possible. The article describes a DNS-based check that Google documents for Googlebot, though the author opts for the simpler JSON list for batch processing. It then presents a standard-library Python script that parses combined-format logs, identifies requests claiming to be known crawlers, and compares client IPs against each crawler's published ranges. The author tested the script on a small handwritten sample log.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Study finds AI-assisted developers slower on some tasks, prompting a book on agentic software development

The author argues that a single measurement showing a productivity gain or loss should not be taken as a verdict on AI coding tools, since the full body of results forms a trend that slopes in a particular direction. The piece distinguishes between evidence grades, noting that weakly supported claims can motivate practices but cannot establish them, and that small studies generalize only to similar groups. It draws on the Lindy effect to argue that Parnas's 1972 information-hiding principles remain relevant because they limit the errors an AI agent can make. The author also notes that AGENTS.md, a configuration convention roughly eighteen months old, is promising but not yet proven to be lasting infrastructure. The post introduces a book on the Agentic Development Lifecycle and invites readers to consider how they would ship AI-generated code safely.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Apple Concedes iPhone Duo Crease Cannot Be Eliminated, Says Matte Coating Only Softens It

Three days after the iPhone Duo went on sale, debate over a visible crease on its foldable inner display has intensified. Apple's hardware engineering chief acknowledged on September 19, 2026, that the physical crease cannot be removed and said the company reduces its visibility with a nanotexture surface and matte coating. Commentators note the device's 15,999-yuan price implies a level of display perfection the crease contradicts, even though Apple's spec sheet never promises a crease-free screen. A reviewer's test of 11,500 folds showed a deeper crease, which critics cite, though the piece argues this is only about 5.75% of Apple's rated 200,000-cycle hinge life and does not show hinge failure. The article also argues that crease visibility lacks any shared testing standard, so no vendor can easily quantify it.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Building a Python Workflow That AI Agents Can Discover Using APFlow

A tutorial walks through building a small Python workflow that combines two regional request counts into one report using local sample data and SQLite. It explains how APFlow, an open-source orchestration engine, separates workflow structure from execution order, with independent regional tasks able to run concurrently. Python functions are registered as capabilities with schemas that can be exposed to agents through MCP, A2A, REST, and CLI adapters. The example runs without a model provider or API key and requires Python 3.11 or newer. The article also covers how dependency data flows between tasks, how capability contracts can be inspected, and how to move from a local example toward a production-style setup. APFlow is noted as still classified as Alpha, so the author recommends starting with verifiable outcomes before adding external integrations.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

'Rejection File' Take-Home Grades Candidates on Patches They Refused to Ship

A proposed take-home exercise argues that hiring should weigh the AI-generated proposals a candidate rejects, not just the final code they submit. The Rejection File asks candidates to deliver a working patch alongside a rejections.json record of at least one rejected model proposal, backed by a repro command that fails if the rejection was correct. The format is designed to run in two to three hours on free-tier infrastructure and to separate engineers who verify AI output from those who simply relay it. Scoring weights rejection evidence most heavily at 30 percent, followed by patch correctness at 20 percent, with reproduction quality, scope discipline, budget discipline, and handoff notes making up the rest. The author frames the approach as a response to a recent debate over engineers acting as unverified intermediaries between requesters and AI tools.

0 comments Read more at DEV Community