How to Verify Whether AI Crawlers in Your Server Logs Are Genuine

·4 views

User-agent strings like GPTBot or ClaudeBot can be spoofed by anyone, and scrapers often do so to get past sites that whitelist well-known bots. Misidentifying fake bots can lead to inaccurate traffic stats, wrongful blocking, or overly permissive firewall rules. Many AI companies publish CIDR IP ranges for their crawlers, and checking an address against those lists is a practical verification step. Some crawlers, such as Googlebot and Bingbot, are instead verified through a two-step DNS lookup, reverse then forward, which is harder to fake. Since vendor documentation and verification methods change over time, developers should confirm current details from each bot's official page before relying on them.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Live airband feed from KORS airport in Victoria, Canada, with ground stop reported

A live airband radio feed covering all frequencies from the KORS airport has been made available online. The stream was shared on Hacker News, where it gained attention from readers. The associated headline also notes that Victoria, Canada is under a ground stop. The source material provides few further details about the reason for the ground stop or its expected duration. The information is drawn from a brief Hacker News post linking to the orcasairband.com site.

0 comments Read more at Hacker News

ProgrammingDEV Community ·

Former contractor's lingering cloud account leads to leak of unreleased product plans

A small crowdfunding company in Japan discovered that its unreleased product plans had surfaced at a competitor just weeks before launch. Investigation of access logs revealed that a former outsourced staffer, whose contract had ended months earlier, still had active access to the company's cloud storage and had downloaded the new product materials. The contractor said they had accessed the files out of curiosity and shared them with an acquaintance, without intending harm, and the full extent of the spread could not be traced. The case highlights guidance from Japan's IPA that access rights should be revoked promptly when a contract ends, as internal-misuse leaks remain a persistent security threat. The company is now reviewing all active accounts, including those of former staff and external collaborators.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Browser-based JPEG metadata stripping removes EXIF and GPS without re-encoding

Photos from phones often embed EXIF metadata, including camera details, timestamps, and sometimes precise GPS coordinates, which can persist when files are shared through messengers, email, or cloud links. Earlier canvas-based approaches remove this data but re-encode the image, degrading quality each time. The article describes a lossless alternative that parses a JPEG's marker segments and copies all but the metadata segments (APP1, APP13, COM) byte-for-byte, leaving the compressed pixel data untouched. The method runs entirely in the browser, so nothing is uploaded, and it typically yields smaller files. The author notes that the ICC colour profile in APP2 should be kept to avoid dulling wide-gamut images, and that a production implementation needs extra checks for padding bytes and segment lengths.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Markup vs margin: the pricing error that quietly erodes marketplace profit

A common mistake among small sellers is confusing markup, which measures profit against cost, with margin, which measures profit against price. The article notes that marketplace and payment fees are charged on the sale price, so a seemingly healthy markup can leave very little profit. Using an illustrative fee stack of 13% plus a $0.30 fixed fee, a $26 price for a $20 item yields only about a 9% margin instead of the intended 30%. Solving for the price that delivers a 30% margin gives roughly $35.62. The author also shows that a 20% discount can cut profit by more than half, and that discounts beyond about 34% lead to losses on each sale.

0 comments Read more at DEV Community