Developer shares architecture lessons from building an AI news crawler pipeline

·1 views

A developer building yocho, an AI industry news service, has published an architecture note detailing the challenges between content acquisition and publication as of September 9, 2026. The service covers AI models, chips, infrastructure, and business news, using a FastAPI and Python backend with PostgreSQL and a Next.js frontend. The post explains that a successful server response does not guarantee usable article content, identifying several independent failure points such as URL discovery, content extraction, relevance classification, and publication conditions. One concrete example highlighted is a classifier fix that separates energy storage from computer memory, illustrating how shared keywords can cause miscategorization at category boundaries. The developer also notes that article generation and database transaction commits are handled carefully to ensure retry and accounting state survive external interruptions like provider rate limiting.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Trace-Native CI/CD Catches AI Agent Failures That Traditional Tests Miss

Software teams are discovering that even 100% CI coverage and comprehensive unit tests fail to catch real-world AI agent failures caused by edge-case inputs, API drift, and prompt changes. A practical example shows how an e-commerce chatbot incorrectly reported items as out of stock due to a malformed third-party API response that only appeared for a specific product ID outside business hours. A travel assistant case further illustrates the gap, where an invalid date input like 'July 32' triggered a silent fallback response instead of a helpful correction, with no existing test covering the scenario. The proposed solution, demonstrated using a tool called Tracely-ai, involves capturing full production traces and replaying them as hermetic regression tests that include every tool call, step, and token exchange. Integrating these trace-based tests directly into CI/CD pipelines ensures that real production failures become enforceable acceptance criteria before future deployments.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Nixamp Bug Silently Played a Web Redirect Into Phone Calls Instead of Audio

A streaming platform called Nixamp, which lets users dial a phone number to listen to live streams via a six-digit code, shipped a bug that caused callers to hear nothing at all. The issue stemmed from the stream's listen link being a browser redirect that set a cookie, which the telephony provider Telnyx could not handle since it fetches audio anonymously without cookie support, returning a 401 error instead of audio. A pre-existing automated test failed to catch the problem because it compared two identical strings rather than verifying actual audio playback. The fix introduces two separate URLs per listing — one for browsers and one for direct audio playback — along with a new endpoint that streams content as a continuous MP3 across track boundaries. Version 0.4.1 has been released with the patch, and also resolves additional issues including missing Matroska container support and a firewall port that was not opened by the installer.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to know when persistent CUDA memory errors mean you need a new GPU

Developers working with large AI models often encounter repeated CUDA out-of-memory errors that survive standard fixes like batch size adjustments, quantization, and cache clearing. Before considering a hardware upgrade, experts recommend ruling out software-side causes such as oversized context windows, large KV cache reservations, or memory fragmentation. The RTX PRO 6000, with 96GB of VRAM, is positioned as a solution for users who have exhausted config-level fixes, capable of running 70B-parameter models at FP8 precision on a single card. Two units linked via NVLink can pool 192GB of memory, enabling full unquantized inference and fine-tuning that previously required multi-GPU server setups. However, the article advises that renting GPU instances remains a more cost-effective option for those with occasional rather than consistent high-memory workloads.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How Engineers Build Sub-15ms RFID Access Systems for 30,000-Person Conferences

Large-scale conferences with tens of thousands of attendees face critical infrastructure challenges when validating badges and managing crowd flow at entry gates. A cloud-dependent system risks cascading delays, as even a 2-second API lag can create physical bottlenecks that disrupt event schedules. The proposed architecture uses local edge nodes running on embedded Linux devices to authenticate RFID badges offline in under 15 milliseconds, independent of internet connectivity. Each edge node maintains an encrypted local store of attendee credentials and uses HMAC cryptographic verification combined with bitwise zone-permission checks to prevent tag cloning and unauthorized access. Raw UHF telemetry data is reduced at the edge through sliding-window deduplication and signal-strength filtering before being forwarded to analytics databases for real-time operational dashboards and sponsor reports.

0 comments Read more at DEV Community