Craigslist JSON-LD lacks ID fields, forcing title-based joins with 89% success rate

·1 views

Developers building a Craigslist multi-city listings scraper discovered that the site's JSON-LD blocks contain no shared identifiers such as product IDs, URLs, or SKUs to match against static HTML listings. The only common field available for joining the two data sources is the listing title, despite titles not being unique across posts. Using a per-title FIFO queue approach, engineers were able to successfully match 290 of 325 listings — an 89% recovery rate — on a captured 298-item search page. Listings that could not be fully matched still returned partial data including title, price, and location, with fields like coordinates, currency, and images left null rather than dropping the row entirely. Separately, Craigslist was found to have migrated from numeric post IDs to alphanumeric strings in a new URL scheme, meaning any scraper relying on legacy numeric-only regex patterns will silently fail on current pages.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

How to Enable HTTP/2 and HTTP/3 in C# for Faster Network Performance

Developers using C# can significantly improve application network performance by upgrading from HTTP/1.1 to HTTP/2 or HTTP/3 via .NET's HttpClient. HTTP/2, supported since .NET Core 3.0, introduces multiplexing and HPACK header compression, reducing header size by up to 95% and allowing multiple requests over a single TCP connection. HTTP/3, available in stable form since .NET 7, builds on the QUIC protocol over UDP, eliminating head-of-line blocking and enabling faster connection establishment through combined transport and security handshakes. Developers can configure the desired protocol version using HttpVersionPolicy settings on HttpClient or individual request messages. A fallback strategy can also be implemented to attempt HTTP/3 first, then HTTP/2, before defaulting to HTTP/1.1.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How Four Conflicting Timestamp Formats Silently Corrupted Months of Data

A data reconciliation system had been producing small daily discrepancies for months, with errors manually corrected each time without deeper investigation. The root cause was discovered only after a daylight saving clock change caused a larger spike: four separate systems were all writing to identically named timestamp columns but using different time zone conventions. The web app used UTC, a legacy batch loader used server local time that silently shifted meaning after a data centre migration, a third-party feed's time zone offset was being discarded by the parser, and an internal tool stored dates with no time, defaulting to midnight in whatever zone the database connection used. Errors went undetected because most activity fell mid-day, where an hour's ambiguity rarely crosses a date boundary — failures clustered around late-evening transactions, daylight saving weekends, and cross-system duration calculations. The team resolved the issue by standardising all new storage to UTC, separately recording local time where relevant, and flagging the ambiguous historical data range as lower confidence rather than attempting an unreliable backfill.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Builds AI Posting Assistant, Discovers It Fabricated a Promise From Test Data

A developer building an AI-powered content assistant for LinkedIn found the agent misread internal Zapier validation posts as public content, incorrectly inferring an unfulfilled promise to followers. The tool was designed to analyze recent post history and suggest three topic angles rather than generating a draft immediately. During early testing, the agent treated its own interpretations of historical data as confirmed facts, a flaw the developer addressed by adding a rule requiring the agent to flag inferences and seek confirmation. A secondary gap emerged when the agent suggested covering a topic the developer had already written about elsewhere that same morning, since the skill had no visibility into external platforms. The experience highlighted two practical pitfalls in agent design: polluted training data can produce confident but false conclusions, and an agent's awareness is strictly limited to the data sources explicitly provided to it.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

AWS Bedrock AgentCore Simplifies Deploying AI Agents to Production at Scale

AWS Bedrock AgentCore is a managed service that allows developers to deploy local AI agents built with frameworks like Strands, LangGraph, or CrewAI directly to production without custom backend infrastructure. In the second part of a tutorial series, a dependency-auditing agent called Release Radar — previously running only on a local machine — is deployed to AWS using AgentCore via a simple CLI workflow. The service automatically handles hosting, scaling, IAM permissions, and observability, eliminating the need to manually configure Lambda functions or API Gateway. Deployment speed depends on package size: bundles under 250MB are pushed to S3 and deploy in seconds, while larger packages with native compiled extensions follow a slower container-build path. The tutorial also highlights input validation as a critical security step, since the agent reads attacker-writable content like repository descriptions and release notes that could contain prompt-injection attempts.

0 comments Read more at DEV Community