SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

Developer releases open dataset tracking 28 AI crawler user agents and their behavior

0
·1 views

A developer has published an open dataset cataloguing 28 AI crawler user agents, including GPTBot, ClaudeBot, and PerplexityBot, available on GitHub and via a live JSON endpoint. The dataset distinguishes between crawlers used for model training and those used for AI-powered search indexing, helping site owners make more targeted blocking decisions. Each entry includes a purpose field and a compliance rating indicating whether the bot actually honors robots.txt directives in practice, not just in vendor documentation. The developer notes that robots.txt alone is insufficient enforcement for bots rated non-compliant, recommending pairing it with WAF or CDN-level rules. The dataset is reviewed monthly, and contributions flagging missing or outdated crawler entries are welcomed.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Antigravity CLI and Neovim Offer a Lightweight Setup for Remote Cloud Development

A developer working across an iPad, Google Cloud Shell, and ephemeral GPU VMs on Google Compute Engine needed a lightweight code editor that could run over SSH without complex configuration. The solution pairs Neovim — configured using only its built-in options and a single init.lua file — with Antigravity CLI, an AI assistant that handles most of the writing, refactoring, and file saving. Because the AI agent manages heavy editing tasks, users do not need deep Vim expertise to benefit from the setup. The configuration relies on Neovim's native features such as autocmds and built-in settings, adding only two external packages for theme and syntax support. The result is a portable, resource-efficient development workspace that works consistently across any terminal environment.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Dev Team Rebuilt a Cultural Feed Ranker Twice — Without ML or a Data Science Team

A collaborative cultural curation platform running on PHP and MySQL spent roughly a year iterating through two generations of its content feed ranking algorithm. The first version moved beyond chronological ordering by combining popularity, recency, and basic genre affinity into a single SQL-based score, but testing revealed that personalization was largely invisible due to a content-surfacing rule dominating the first page. The team rebuilt the system by shifting scoring logic from SQL into application code and rethinking how user taste is defined, ultimately creating what they call a narrative affinity model. Without a data science team or machine learning pipeline, calibration relied on synthetic user profiles, first-page composition audits, and repeated observation of real feed output. The engineers also solved a foundational normalization problem first, mapping movies, books, games, music, and editorial posts into a unified data shape before any ranking logic could be applied.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Key LLM Concepts You Need to Run AI Models Locally on Your Own Machine

Running AI models locally requires understanding the distinct roles of the model file, the server (such as Ollama or LM Studio), the local API, and client applications that connect to it. Large language models work by predicting the next chunk of text using learned numerical weights, and running them on personal hardware is called inference. Hardware resources — particularly GPU VRAM, system RAM, and CPU — determine whether a model runs efficiently, with model weights needing to fit into VRAM for optimal speed. Model parameter counts, measured in billions, indicate capability and hardware demand, ranging from lightweight 1–3B models suitable for any laptop to 70B-plus models requiring workstation-grade hardware. For conversational use and automated workflows, instruct-tuned model variants are recommended over base models, as they are specifically trained to follow instructions.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Developer shares lessons from failed Django deployment on Render despite successful build

A developer deployed a Django application to Render, only to encounter persistent 500 and 400 Internal Server Errors despite the platform reporting the service as live. The root cause was that the frontend's production build directory, expected at frontend/dist/, was never generated during deployment, as the build step had not been properly configured. This highlighted a key distinction: a repository containing source files does not automatically produce compiled build output unless the deployment pipeline explicitly runs the build process. The developer traced the issue through Render's logs, which showed Gunicorn running correctly while individual requests failed, underscoring that a live deployment status does not guarantee all application components are functioning. The experience revealed that multi-step deployments — covering dependency installation, frontend builds, and backend setup — must each be accounted for, especially since local environments can mask missing production steps.

0 comments Read more at DEV Community
Developer releases open dataset tracking 28 AI crawler user agents and their behavior · ShortSingh