AIFeed Protocol Lets Publishers Sign Content Permissions for AI Web Crawlers

·2 views

A developer has released AIFeed, an open-source protocol designed to give website publishers control over how AI web crawlers access their content. The tool allows publishers to sign a JSON manifest using Ed25519 cryptography and anchor it via DNS, establishing verifiable content permissions. Setup for static sites is straightforward, requiring a single command, and the protocol claims significant reductions in data transfer compared to standard HTML. The project is still in early stages, with no external cryptographic review completed and specifications remaining in draft form. Code and documentation are available on GitHub, with an MCP server also listed on Smithery.ai.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Three-Layer Framework Proposed to Fix GA4's Blind Spots in AI Search Measurement

Google Analytics 4 captures only a fraction of traffic driven by AI assistants, making it an incomplete tool for measuring Answer Engine Optimization (AEO), according to a framework published by Search Engine Land. The model argues that AI search can shape buyer behavior before any trackable click occurs, through branded searches, direct visits, or faster sales cycles that standard last-click reports fail to attribute. To address this, the framework splits AEO measurement into three layers: direct attribution from identifiable AI referrals, influenced attribution from indirect demand signals, and a longer-term brand visibility layer. Marketers are advised to use GA4's AI Assistant channel data as a baseline floor estimate rather than a complete picture of AI-driven demand. The framework acknowledges that other factors such as seasonality, PR, and brand campaigns can affect the same metrics, so AEO signals must be evaluated in context.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Vinkius Connects AI Agents to Bright Data Web Scraping via MCP Framework

A platform called Vinkius has launched a Bright Data connector designed to give AI agents direct, governed access to large-scale web scraping and proxy infrastructure. Built on MCPFusion, an open-source TypeScript framework, the integration allows agents to perform real-time data gathering tasks such as market research and competitive analysis without developers manually managing scrapers or API credentials. The connector uses a single gateway token model, eliminating credential sprawl and reducing security risks associated with giving large language models unrestricted API access. All agent operations run inside isolated V8 sandboxes enforcing eight governance policies, including SSRF prevention and audit logging. The system supports both synchronous web requests for lighter tasks and an asynchronous pipeline for high-volume jobs such as scraping thousands of LinkedIn or Amazon pages.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

NFS Storage Triggers I/O Failures in Air-Gapped Kubernetes Cluster

A self-managed Kubernetes cluster operating in an air-gapped private datacenter is facing critical storage failures due to its sole reliance on NFS as the primary StorageClass. Applications like Elasticsearch and Camunda's Zeebe engine require POSIX semantics — including atomic rename, fsync, and file locking — which NFS cannot reliably provide because of its stateless protocol and network latency. These limitations cause race conditions, data inconsistencies, and write amplification that degrade cluster reliability. Local storage alternatives such as Longhorn are ruled out because Elasticsearch data already exceeds the 100GB SSD capacity of each server, while cloud-based block storage options are unavailable in the isolated environment. The DevOps team must transition to a block storage solution that satisfies POSIX requirements to prevent continued data integrity risks and potential downtime.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How a Common Pagination Bug Silently Wiped Client Data at Scale

A software engineer recounts a recurring pagination bug encountered at multiple ad-tech companies, including Beeswax (now The Buyer Cloud), where missing adgroup data was discovered on a Friday afternoon after quietly degrading for days. The issue affected only the platform's largest clients, whose high volumes of line items exposed a flaw in the default page-based pagination strategy used by most ORM frameworks. Page-based pagination relies on count queries that scale linearly with table size, making it increasingly unreliable as data grows. The engineer argues that cursor-based pagination should be the default at scale, a point made more urgent now that AI agents also consume APIs and require stable, deterministic data retrieval. Early detection through growth-based observability, rather than reactive incident response, is proposed as the key safeguard against such outages.

0 comments Read more at DEV Community