How to Build a Scheduled Job Scraper That Tracks New Postings Week by Week

·1 views

A developer tutorial on DEV Community outlines how to build a scheduled web scraper that detects newly posted and closed job listings by comparing consecutive weekly scrapes. The core technique relies on diffing stable job IDs rather than titles, since titles can be edited and cause false positives in change detection. The guide warns against persisting state from failed or incomplete fetches, as partial data can falsely report mass job closures. It also covers handling the first run separately to avoid flooding alerts with hundreds of spurious "new" listings. Beyond individual postings, the method enables tracking hiring momentum by counting role changes per company over time, offering a signal for job seekers and sales teams alike.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

LibreFang 2026.7.21 Patches Multi-User Data Leaks and Adds MCP Resources

LibreFang has released version 2026.7.21, incorporating 61 pull requests from four contributors since the previous release ten days ago. The update addresses critical security issues, including a knowledge graph data leak that could expose one user's memory to another on shared agents, and adds cross-account channel guards to block unauthorized agent-to-channel messages. A four-pass repository-wide security audit identified and resolved more than 15 bugs spanning authorization gaps and token-quota race conditions. New features include per-user LLM provider credentials with spend tracking, operator-controlled provider allowlists, real-time Slack progress updates for long-running tasks, and online editing of HAND.toml manifests from the dashboard. The release also completes MCP integration by implementing the resources primitive, extending agent access beyond tools alone.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Uses AI Agents to Build 50-Game Educational Platform GamesMom

A software developer built GamesMom, a browser-based educational gaming platform featuring over 50 games, with significant assistance from AI agents throughout the development process. The developer found that AI was most valuable in reducing repetitive tasks such as coding boilerplate, debugging, documentation, SEO workflows, and testing, rather than replacing core engineering judgment. Key decisions around architecture, user experience, accessibility, and product direction still required human expertise and were not delegated to AI tools. The experience led the developer to conclude that AI functions best as a development partner integrated across the full lifecycle, not merely as a code generator.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How bilingual Arabic-English web scrapers silently return empty data fields

Developers building scrapers for bilingual Arabic-English platforms can encounter a subtle bug where data fields return null without any errors or failed requests. This happens because regional subdomains often serve localized Arabic pages, causing English-language CSS selectors to match nothing while still returning a valid HTTP 200 response. Arabic pages also use Eastern Arabic numerals, which standard ASCII regex patterns like [0-9]+ silently miss, causing numeric fields to appear absent rather than unreadable. Defensive checks that raise loud errors when critical fields are entirely empty across all records help catch locale mismatches early. Additional pitfalls include substring keyword matching across languages, where terms like 'coo' or 'partner' incorrectly match unrelated job titles, making word-boundary enforcement essential.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to build a reliable hiring intent score using volume, seniority, and momentum

A DEV Community article outlines a method for scoring how actively a company is hiring by combining three inputs: open role volume, seniority mix, and momentum based on role count changes over time. The author warns that raw job counts are misleading without normalisation for company size, recommending a capped logarithmic volume calculation so large firms do not dominate the score. Seniority weighting is highlighted as particularly error-prone, since job title classifiers can misread words like 'coo' in 'Coordinator' or treat 'Assistant Vice President' as an executive role. The article provides specific regex fixes using word boundaries to prevent such misclassifications, which in the author's own data reduced the apparent share of executive-level roles from 18% to 16% and shifted company rankings. Momentum on a first run is also addressed, with the author recommending a neutral placeholder value rather than zero to avoid making all companies appear stagnant on day one.

0 comments Read more at DEV Community