SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

Most AI crawlers appear in server logs, but Google-Extended and Applebot-Extended never do

0
·6 views

Despite being widely referenced in robots.txt files, Google-Extended and Applebot-Extended never actually send HTTP requests to websites, making them invisible in server logs. In contrast, nine real AI crawlers from OpenAI, Anthropic, and Perplexity do send identifiable user-agent strings and can be tracked in WordPress access logs. Vendors confirm that index-building bots generally honour robots.txt directives, but user-triggered fetchers — such as ChatGPT-User and Perplexity-User — may ignore those rules because a human initiated the request. This means a page blocked from AI training can still be fetched moments later to answer a live user query. WordPress site owners can monitor all nine documented crawlers without a plugin by adding a small script to the mu-plugins directory that logs matches to a capped database option.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Why RGB Math Fails at Color Perception and How Scientists Fixed It Over 50 Years

Simple RGB distance formulas treat all color differences equally, but human eyes are far more sensitive to changes in green than in blue, making raw math an unreliable measure of perceived color similarity. To address this, researchers developed Delta E (ΔE), a perceptual color-difference standard where a score of 1.0 represents the smallest gap a human eye can detect. All modern color spaces trace back to 1920s experiments in which participants mixed red, green, and blue light to match every visible color, eventually producing the XYZ system as a universal foundation. Over the following five decades, four successive formulas — ΔE76, CMC, ΔE94, and CIEDE2000 — were developed, each correcting the previous one's weaknesses, particularly around blue and blue-violet tones. Today, industries such as automotive paint and commercial printing rely on strict ΔE thresholds — as tight as 0.5 for car paint — to ensure color consistency.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Study tests LLMs' ability to spot UI rule violations from screenshots vs DOM data

A developer ran a controlled experiment to measure how accurately large language models can detect UI threshold violations — such as font sizes and tap target dimensions — using only screenshot crops, with ground truth labels derived from DOM measurements. Ten custom style rules were tested across 12 pages rendered at two viewport widths, producing 96 scorable judgment pairs after excluding pages where elements were absent. GPT (via ChatGPT UI) achieved 90% accuracy on the contrastive rules containing both pass and fail cases, while Gemini 2.5 Flash scored 68% on the same set. Both models performed well on visually apparent rules like wrapping headings, but struggled significantly on purely numeric rules such as font size and line length. The author notes the tap-target rules could not test true threshold discrimination, as each rule contained only violations or only passes across all tested pages.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Vercel AI SDK Alternatives Gain Ground as Hosting Costs Bite in 2026

As of September 2026, developers are increasingly seeking alternatives to the Vercel AI SDK due to cost and architectural constraints, according to a guide updated this month. Vercel's pricing model, which charges by function invocation, bandwidth, and GB-hours, can cause AI workload bills to spike dramatically — a $20 bill can balloon to $200 when agent loops run intensively. The SDK also assumes Next.js and Vercel hosting as defaults, creating friction for teams wanting to deploy backends elsewhere. Emerging alternatives like Mastra, built by the Gatsby founders, and LangGraph offer more flexible deployment options including Cloudflare Workers, AWS Lambda, and self-hosted VPS servers. Affordable VPS options from providers like netcup are highlighted as cost-effective hosting solutions for long-running AI agents.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

LSP-ember brings autocompletion and LSP support for Ember's .gjs/.gts files in Sublime Text

A developer has released LSP-ember, a new Sublime Text package that enables Language Server Protocol features — such as autocompletion, go-to-definition, and hover — for Ember's template tag file formats (.gjs and .gts). The tool acts as a bridge between Sublime Text and Glint, the TypeScript tooling layer for Ember, bringing the editor closer to feature parity with VS Code. The release follows an earlier May 2025 package called Template Tag, which had added syntax highlighting for these file types but left writing-side tooling unsupported. As part of this update, the original single syntax definition has been split into two separate ones — Glimmer JS and Glimmer TS — each correctly extending JavaScript and TypeScript respectively. The developer is seeking community help with testing, bug reports, and contributing modern Ember code snippets to improve the package further.

0 comments Read more at DEV Community