Truck Driver Turned Coder Discovers RSS After 20 Web Scraping Blocks

·1 views

A Japanese truck driver teaching himself Python repeatedly hit 403 Forbidden errors while attempting to scrape news data from a major website using BeautifulSoup and Requests. Despite trying over 20 workarounds — including rotating User-Agent headers, adding randomized delays, and adjusting request intervals — the site's bot detection and web application firewall continued blocking access. After his twentieth failed attempt, he reconsidered his approach and discovered an RSS feed linked in the website's footer. Using the Python library feedparser, he retrieved the same structured data in just a few lines of code, with no HTML parsing or browser disguise needed. This experience became the founding moment for his AI summarization application, puoppo.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Dev Series 'Left of the Loop' Closes With a Meditation on Intentional Endings

The final installment of the 'Left of the Loop' series on DEV Community draws on Herodotus's account of the phoenix to reflect on how teams and organizations should approach inevitable endings. The piece argues that teams facing dissolution, restructuring, or key departures should deliberately transfer shared knowledge and framing rather than letting it dissipate by chance. It references earlier entries in the series, including concepts called 'the Hestia,' 'the Boule,' and 'the Mimesis,' to illustrate how intentional handoffs differ from passive continuity. The author contends that starting over is not the same as starting from nothing, provided the groundwork is laid before the ending arrives. The series concludes with the assertion that the only thing within anyone's control is choosing what gets preserved before the fire.

0 comments Read more at DEV Community

ProgrammingHacker News ·

HART OS: Open-Source AI Operating System Designed to Run Without Datacenters

A project called HART OS has been shared on Hacker News as an open-source operating system built specifically for artificial intelligence workloads. The system is designed to allow frontier AI models to operate without relying on traditional datacenter infrastructure. The project is hosted publicly on GitHub under the organization hertz-ai. At the time of posting, the submission had received minimal community engagement, with only 3 points and no comments. The initiative reflects a broader interest in decentralizing AI compute away from large-scale cloud facilities.

0 comments Read more at Hacker News

ProgrammingDEV Community ·

Stale Cucumber Tests Are Silently Bloating Your CI Pipeline

Many software teams carry large collections of Cucumber feature files in their CI pipelines that pass consistently but rarely catch real bugs, adding significant time and compute cost to every code push. Engineering best practice treats a test that never fails as a liability rather than a safety net, since it consumes resources while delivering no meaningful signal. Bloated test suites slow developer feedback loops, increase flakiness, erode confidence in genuine failures, and create ongoing maintenance overhead with every codebase refactor. Experts recommend auditing scenarios with the same rigor applied to performance issues, using CI execution reports to identify scenarios that either never run or have not failed in several sprints. A smaller, targeted test suite provides clearer coverage visibility and faster, more trustworthy pipeline results than a large ceremonial one.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Three Portfolio Mistakes That Cost QA Engineers Job Opportunities

A hiring guide aimed at QA and automation engineers identifies three common portfolio errors that immediately reduce their chances with hiring managers. The first mistake is listing multiple automation tools without showing actual code, which signals superficial familiarity rather than real expertise. Second, portfolios that display only passing tests fail to demonstrate how an engineer handles flaky builds, slow CI environments, or transient API failures. The recommended fix is to include retry logic, explicit logging, and failure screenshots that show deliberate decision-making under realistic conditions. Ultimately, hiring managers want portfolios that answer what the engineer decided and why, not ones that read like a keyword checklist.

0 comments Read more at DEV Community