SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

Server logs reveal AI crawlers outnumber search bots, and a third of 'Googlebot' traffic is fake

0
·2 views

A website owner reviewed two weeks of nginx access logs, keeping only bot requests whose IP addresses genuinely belonged to the companies behind them. Verified AI crawlers made 1,590 requests, more than all search engines combined at 563, with Meta's crawler and ClaudeBot visiting more often than Googlebot. Common analytics tools missed this activity because these bots do not run JavaScript, so only server logs captured it. Of 331 requests claiming to be Googlebot, 119 came from rented Google Cloud servers rather than Google, and some of these impostor bots attempted to access secret files such as .env files and cloud keys, all of which the server blocked. The author recommends reverse DNS checks to identify fake Googlebots and returning 403 errors for sensitive files, noting that robots.txt cannot stop bots that misrepresent themselves.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Developer Automates Daily Google Calendar Screenshots Using Playwright, rclone and systemd

A developer built a workflow on Ubuntu to automatically capture a rolling seven-day view of Google Calendar as an image, rather than exporting event data through an API. The pipeline uses Python with Playwright to drive a headless Google Chrome browser, saves dated PNG files locally, and uploads them to Google Drive via rclone, all scheduled by a systemd user timer. The author found authentication the most challenging part, as Google rejected sign-in from Playwright's bundled Chromium, so a dedicated persistent Chrome profile was used instead. Sessions were reused successfully by launching both the manual sign-in and headless runs with the same Linux password-storage mode. The original workflow was tested on Ubuntu 22.04 with Playwright 1.48.0, while the public repository's generalized version has not been independently tested with a live account.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Developers Share Lessons From Building Fail-Closed Verification Tools for AI Agents

The author reports building two small open-source verification tools for the emerging AI agent economy, which share a rule: an unverifiable result should never be reported as clean. Al-Muqarin checks claims against signed, dated snapshots of public sanctions lists and returns a not-measured result when a list is missing, stale, or below a minimum version. agent-receipt-verify checks x402/EIP-712 payment receipts and reports incomplete ones as failures rather than passing them. The team tests its tools by planting deliberate bugs and confirming that tests catch each one, with Al-Muqarin catching all 548 planted faults. The authors acknowledge that other services already sign sanctions-screening results and that their tools have no paying users yet. They also disclose an unresolved weakness in a separate prototype, where the verifier identity field is free text and can be exploited.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Teks melingkar dibuat ulang dengan Python dan fontTools, membandingkan jarak antarhuruf

Tata letak teks melingkar yang terlihat rumit sebenarnya memposisikan setiap huruf di sepanjang busur dan memutarnya menghadap ke luar. Penulis menunjukkan cara meniru fitur Type on a Path di aplikasi desktop osFoundry dengan Python, menggunakan fontTools, numpy, dan Pillow. Dengan skrip tersebut, stempel KUMO COFFEE ROASTERS dibuat di atas lingkaran berdiameter 450,6 px. Hasil pengukuran memperlihatkan bahwa memutar huruf dengan sudut sama menghasilkan jarak tidak rata, dengan celah negatif di sekitar huruf lebar dan spasi antarkata yang menggelembung. Penulis juga membahas posisi teks rata tengah pada path tertutup bergantung pada titik awal.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Why multi-file C++ programs fail in online compilers and how to fix them

Many online C++ compilers accept only a single editor file, so a program split into a header and source file fails with a missing-header error. Pasting only the header does not help, because the member function definitions live in the .cpp file, which then triggers linker errors. The article explains that the program must compile and link all source files together. It shows an example Student class split across header and source files, and notes rules such as exact filename matching, folder support, and file limits on one platform. It also describes how to submit multiple files through an API using Python.

0 comments Read more at DEV Community