Developer Builds Low-RAM Football Data Pipeline to Extract xG and Referee Stats

·1 views

A software developer has published a technical tutorial detailing the construction of a professional-grade football data scraper targeting Flashscore, built on the Apify platform. The tool addresses two core engineering challenges: excessive memory consumption from browser-based scraping and Flashscore's proprietary pipe-delimited data format served over CDN endpoints instead of standard APIs. The solution uses a two-phase hybrid pipeline where a Puppeteer browser instance briefly handles authentication token capture before shutting down, cutting RAM usage from roughly 1.5GB to 70MB. Parallel HTTP/2 workers then handle the bulk of data extraction, decoding granular match statistics including Expected Goals, referee assignments, and half-time breakdowns. The extracted datasets can be piped directly into Python, Pandas, or exported as JSON or Excel files.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

AI Agent Phones Let LLMs Operate Smartphones Autonomously Like Human Users

An AI agent phone is a real or cloud-hosted smartphone controlled entirely by a large language model, allowing it to tap, swipe, type, and navigate apps just as a human would. Unlike browser-based agents, it can handle mobile-only workflows such as ride-hailing, food delivery, mobile banking, and two-factor authentication that have no web equivalent. The agent perceives the screen through a combination of the OS accessibility tree and vision-based screenshot analysis, then executes actions in a continuous loop until a task is completed. Teams can run these agents on local hardware for privacy-sensitive work or on cloud-hosted devices for scalable, parallel operations. Common applications include automated QA testing, data collection from native apps, and automating workflows that lack a public API.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

ClickHouse 26.8 LTS Brings 57 Breaking Changes Across Five Releases Since 26.3

ClickHouse has announced its 26.8 LTS release, succeeding the widely used 26.3 LTS, with the release branch already versioned but no final tag or Docker image published as of 27 August 2026. Upgrading from 26.3 to 26.8 is not a single-step process — users must account for breaking changes introduced across versions 26.4, 26.5, 26.6, and 26.7 as well. In total, 57 unique backward-incompatible changes have been identified across the five releases, after deduplicating one entry that appeared twice in the upstream changelog. Beyond explicit breaking changes, approximately 70 settings also had their default values altered, none of which are flagged in the official backward-incompatibility lists. Experts recommend waiting for around version 26.8.3 before upgrading production systems, consistent with ClickHouse's historical pattern of issuing several rapid patch releases after each LTS launch.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

8,000+ WordPress plugins have known CVEs since 2023; millions of installs remain at risk

A developer analyzed over 15,500 publicly documented vulnerability records across 8,010 WordPress plugins dating back to 2023, using the GitHub Advisory Database and the WordPress.org plugin API. The research found that 3,780 vulnerable plugins have been removed from the WordPress.org directory, yet affected websites receive no dashboard warning or notification of the removal. Around 2,115 plugins with known vulnerabilities and no updates in over 12 months remain installable today, collectively accounting for roughly 6.7 million active installs. The analyst also noted that ranking plugins by raw CVE count is misleading, since well-maintained plugins with bug-bounty programs tend to accumulate more reported flaws than neglected, unaudited code. A public index of findings and the full scoring methodology has been published online for independent review.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Windows + WSL2 + Ollama: Choosing the Right Architecture for Local AI Dev

A developer setting up a local AI environment on Windows with WSL2 is weighing two architectural approaches for running Ollama and large language models. Option 1 runs Ollama natively on Windows while the application lives inside WSL2, communicating via HTTP on localhost port 11434. Option 2 keeps both the application and Ollama entirely within WSL2, with GPU access passed through to the underlying Windows drivers. The developer's current setup follows Option 1 and is functional, but they are questioning whether Option 2 offers better long-term performance and workflow benefits. The post invites community input from developers who have hands-on experience with either configuration.

0 comments Read more at DEV Community