30-Day AI Crawler Study Reveals Bytespider Dominates, Bing Beats Google for GEO

·1 views

A developer running a Generative Engine Optimization (GEO)-focused website spent 30 days logging AI crawler activity using Cloudflare D1 to understand how AI systems index content. Bytespider, operated by ByteDance, recorded over 500 hits per day on the small site, far outpacing all other bots and suggesting aggressive training-data collection. Bingbot averaged 98 daily hits, making Bing Webmaster Tools the most practical free analytics tool for GEO practitioners, while ClaudeBot showed a steady 18–22 daily visits with a consistent 24–48 hour indexing lag after new content is published. GPTBot and PerplexityBot were comparatively low-volume, though the latter was noted for higher-quality engagement. Based on these findings, the developer has adopted measures including an llms.txt file, structured FAQ and Article schema markup, answer-first paragraph formatting, and cited data points to improve AI discoverability.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

AI Agent Phones Let LLMs Operate Smartphones Autonomously Like Human Users

An AI agent phone is a real or cloud-hosted smartphone controlled entirely by a large language model, allowing it to tap, swipe, type, and navigate apps just as a human would. Unlike browser-based agents, it can handle mobile-only workflows such as ride-hailing, food delivery, mobile banking, and two-factor authentication that have no web equivalent. The agent perceives the screen through a combination of the OS accessibility tree and vision-based screenshot analysis, then executes actions in a continuous loop until a task is completed. Teams can run these agents on local hardware for privacy-sensitive work or on cloud-hosted devices for scalable, parallel operations. Common applications include automated QA testing, data collection from native apps, and automating workflows that lack a public API.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

ClickHouse 26.8 LTS Brings 57 Breaking Changes Across Five Releases Since 26.3

ClickHouse has announced its 26.8 LTS release, succeeding the widely used 26.3 LTS, with the release branch already versioned but no final tag or Docker image published as of 27 August 2026. Upgrading from 26.3 to 26.8 is not a single-step process — users must account for breaking changes introduced across versions 26.4, 26.5, 26.6, and 26.7 as well. In total, 57 unique backward-incompatible changes have been identified across the five releases, after deduplicating one entry that appeared twice in the upstream changelog. Beyond explicit breaking changes, approximately 70 settings also had their default values altered, none of which are flagged in the official backward-incompatibility lists. Experts recommend waiting for around version 26.8.3 before upgrading production systems, consistent with ClickHouse's historical pattern of issuing several rapid patch releases after each LTS launch.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

8,000+ WordPress plugins have known CVEs since 2023; millions of installs remain at risk

A developer analyzed over 15,500 publicly documented vulnerability records across 8,010 WordPress plugins dating back to 2023, using the GitHub Advisory Database and the WordPress.org plugin API. The research found that 3,780 vulnerable plugins have been removed from the WordPress.org directory, yet affected websites receive no dashboard warning or notification of the removal. Around 2,115 plugins with known vulnerabilities and no updates in over 12 months remain installable today, collectively accounting for roughly 6.7 million active installs. The analyst also noted that ranking plugins by raw CVE count is misleading, since well-maintained plugins with bug-bounty programs tend to accumulate more reported flaws than neglected, unaudited code. A public index of findings and the full scoring methodology has been published online for independent review.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Builds Low-RAM Football Data Pipeline to Extract xG and Referee Stats

A software developer has published a technical tutorial detailing the construction of a professional-grade football data scraper targeting Flashscore, built on the Apify platform. The tool addresses two core engineering challenges: excessive memory consumption from browser-based scraping and Flashscore's proprietary pipe-delimited data format served over CDN endpoints instead of standard APIs. The solution uses a two-phase hybrid pipeline where a Puppeteer browser instance briefly handles authentication token capture before shutting down, cutting RAM usage from roughly 1.5GB to 70MB. Parallel HTTP/2 workers then handle the bulk of data extraction, decoding granular match statistics including Expected Goals, referee assignments, and half-time breakdowns. The extracted datasets can be piped directly into Python, Pandas, or exported as JSON or Excel files.

0 comments Read more at DEV Community