Web scraping: When switching from Python's requests to httpx doesn't improve speed

·1 views

A technical analysis compares Python HTTP clients requests and httpx for web scraping tasks. The article explains that httpx offers modern features like async support and HTTP/2 capability while maintaining compatibility with requests' API. It argues that for single-target scraping with built-in politeness delays, client speed is rarely the bottleneck, making async benefits negligible. However, httpx provides significant time savings when scraping hundreds of independent targets simultaneously. The piece concludes that developers should analyze their specific scraping bottlenecks before migrating, as code complexity costs may outweigh performance gains.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Prevent lost test results by documenting exit codes before ephemeral hosts expire.

A development article advises teams to preserve crucial test data before ephemeral hosts are recycled. It warns that logs, commands, and exit codes can become permanently lost if not captured promptly. The article proposes a clear workflow involving designated roles of runner, reviewer, and reclaimer. It stresses that documentation should be stored in a wiki or repository, not in chat logs or on the temporary host itself.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Syed Jawad's Kaggle challenge entry tests AI models on data recovery claims.

Syed Jawad submitted an entry to a Kaggle benchmarking challenge on October 10. The project involved testing ten different AI models. The core question examined was their capability to restore deleted data. The work was shared as a community post on DEV Community.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

AI models tested on ability to predict data loss from destructive commands.

A recent Kaggle benchmark tested 10 AI models on their ability to predict data loss from destructive filesystem, git, and SQLite commands. The models were given matched pairs of commands and a restore script, then asked to identify which data resources would be lost and recoverable. Performance varied widely, with Gemini 3.7 Flash scoring perfectly and some models scoring as low as 1 out of 18. The benchmark revealed that SQLite scenarios were the most difficult for the models, with a median accuracy of 58%. A significant finding was that 25 model answers incorrectly claimed no permanent loss would occur when the provided restore script would have left data unrestored.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Beginner developer uses AI agents to build and submit mobile game to four stores

A software development beginner in Korea directed multiple AI coding agents to create a mobile game over a 17-day period. The resulting game, 'Afterclose: Night Delivery', was submitted to four digital storefronts, including Google Play and the App Store. The developer's role involved high-level direction, testing on physical devices, and setting guardrails for the AI agents, rather than writing code directly. Despite over 1,450 automated tests passing, a critical bug affecting controls and sound was only discovered during manual testing on a real phone. The process involved splitting development tasks by file area across separate worktrees to manage parallel AI work.

0 comments Read more at DEV Community