Cloudflare's AI Crawler Policy: How to Block Scrapers Without Hurting SEO

·1 views

Cloudflare introduced its 'Accountable Mixed-Use AI Crawlers' policy in March 2024, automatically blocking crawlers that do not declare their purpose via HTTP headers. The policy has gained traction amid rising data-privacy regulations such as GDPR and CCPA, which carry fines of up to 4% of annual revenue for misuse of data. Platforms including Medium, Substack, and YouTube have reported a 73% surge in content-removal requests linked to AI training activity. Website owners can protect their content by configuring robots.txt rules, adding custom firewall rules in the Cloudflare dashboard, and enabling JavaScript challenges for sensitive pages. The key challenge is distinguishing legitimate search-engine bots like Googlebot and Bingbot from unauthorized AI scrapers to avoid unintended drops in organic search traffic.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Why Python Raises UnboundLocalError and How to Fix It Correctly

Python's UnboundLocalError confuses developers because the variable in question visibly exists elsewhere in the code, yet the error still occurs. The root cause is that Python scans an entire function body before executing it, and any variable assigned anywhere inside the function is pre-marked as local throughout that function. This means a line like 'count += 1' causes Python to treat 'count' as a local variable, making the outer module-level 'count' inaccessible within that function. The cleanest fix is usually to pass the value as a parameter and return the updated result, avoiding shared mutable state altogether. When a function must modify a module-level variable, the 'global' keyword explicitly instructs Python to reference the outer scope instead of creating a local shadow.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

AWS Workshop Day 1: Hands-On Training Covers S3, CloudFront, and EC2 Basics

A two-day cloud workshop focused on AWS fundamentals kicked off with its first session covering core infrastructure concepts. Participants logged into the AWS Management Console and learned how AWS Regions and Availability Zones support global reliability and low latency. The session included practical exercises with Amazon S3, where attendees created storage buckets, uploaded files, and served content via pre-signed URLs and a CloudFront CDN distribution. Learners were also introduced to Amazon EC2, launching virtual instances, connecting via SSH, and deploying the Nginx web server. The session provided a foundational overview of cloud storage, content delivery, and compute services on AWS.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Gas Audit Flags $10–30M Annual Inefficiency in Bybit's Smart Contracts

A gas-optimization audit of Bybit's on-chain smart contracts, covering Ethereum Mainnet and L2 networks Optimism, Arbitrum, and zkSync, was completed on 16 September 2026. The audit, conducted by a senior DeFi security researcher, reviewed all contract components tied to Bybit's approximately $15.1 billion TVL. A total of 29 findings were identified across categories including unbounded loops, excessive storage operations, and inefficient calldata handling, with unbounded loops posing the highest risk of transaction failures. The protocol received a gas-risk score of 3 out of 10, indicating no critical security vulnerabilities but notable inefficiencies. Left unaddressed, these issues could result in an estimated $10–30 million in avoidable gas fees annually given current transaction volumes.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

IBM i Developers Share RPGLE Pattern to Automate User Access Provisioning

A developer has published a production-tested RPGLE pattern on DEV Community that automates user access provisioning on IBM i without middleware or external servers. The system works by reading JSON request files dropped into an IFS inbox directory, which a scheduled RPGLE job periodically scans and processes using SQL and IBM's QSYS2 system views. Each request specifies a target user and a model user, and the program clones the model's profile and entitlements to the new account using a CL wrapper around the CPYUSRPRF command. The solution uses RPG's DATA-INTO opcode with a YAJL-based parser to deserialize JSON directly into data structures, eliminating manual token parsing. An audit trail is maintained by writing history records and emitting JSON audit events, addressing a common compliance gap in manual access provisioning workflows.

0 comments Read more at DEV Community