TrendVidStream Replaced SQLite Job Queues With NATS JetStream for Regional Video Events

·1 views

TrendVidStream, which runs video discovery crawlers across 8 regions, replaced an inline cron-based processing system with NATS JetStream to handle events like indexing, cache invalidation, and sitemap updates. The original approach fused data fetching and fan-out into a single process, causing partial failures when any step crashed or stalled. A SQLite jobs-table queue was also evaluated but rejected due to polling inefficiency, locking contention, lack of native fan-out, and inability to work across machines without added replication. NATS JetStream was chosen for its push-based delivery, multi-consumer support, independent storage, and cluster capability — all in a single Go binary requiring no container infrastructure. The team configured a stream with subject-based regional filtering, a 72-hour message retention window, and a 2-minute deduplication window to ensure resilience during deployments.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

How a mileage app cut GPS errors from 50% to 95% accuracy by doubting bad data

A developer building a mileage-tracking app discovered that GPS signals are frequently unreliable in real-world conditions such as tunnels, urban canyons, and parking garages, causing absurd readings like a stationary car clocking 400 km/h. The team shifted their approach from trusting GPS as a source of truth to actively filtering out physically impossible data points before they could affect trip logs. When GPS drops out entirely, the app now uses dead reckoning — combining the last known speed and heading with accelerometer data — to estimate position during short blackouts like tunnel crossings. A weighted filtering system was also introduced so that jittery or inconsistent GPS signals have less influence on the final position estimate than clean, stable ones. These layered techniques collectively pushed trip-tracking accuracy from around 50 percent to 95 percent.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

10 AI Trends Reshaping Software Development in 2026

Artificial intelligence is rapidly transforming how developers work, with autonomous AI agents now capable of planning tasks, writing code, browsing the web, and coordinating with other agents to complete complex workflows. The Model Context Protocol (MCP) has emerged as a standardized way to connect AI models with tools like GitHub, Slack, and databases, reducing manual integration overhead. Multi-agent systems, where specialized agents handle research, coding, testing, and deployment independently, are enabling teams to tackle sophisticated software projects more efficiently. AI can now generate full application stacks from natural language descriptions, covering frontend, backend, authentication, and CI/CD configuration, shifting developers toward reviewing and refining rather than writing every line. Smaller, lightweight AI models are also gaining traction for production use, offering faster responses, lower costs, and offline capability, as AI adoption becomes a standard part of the development workflow rather than an experimental tool.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

npm API retirement breaks pnpm audit in CI pipelines, blocking all frontend deploys

Frontend deployments began failing after npm retired the legacy audit endpoint that pnpm audit relies on, returning an HTTP 410 Gone error. No code or dependencies had changed — a third-party infrastructure decision silently broke the security gate and blocked all queued deploys, including unrelated bug fixes. The root issue is that pnpm audit is not a local scan; it serializes the dependency tree and sends it to a specific npm API that no longer exists. The recommended fix is to replace pnpm audit with a tool like osv-scanner, which scans the lockfile directly against the public OSV database without depending on any vendor endpoint. Quick workarounds such as adding '|| true' or upgrading pnpm were found to be ineffective or counterproductive, as the retired endpoint affects all versions.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

PySBD vs yasbd-lib: How Two Python Libraries Approach Sentence Boundary Detection

Two Python libraries — PySBD and yasbd-lib — tackle sentence boundary detection with fundamentally different architectures. PySBD, available since 2020 and supporting 22 languages, works by temporarily replacing punctuation with placeholder tokens before splitting text, then restoring the original content. This mutable approach means character-offset spans must be reconstructed after processing, which can become computationally expensive on long or repetitive documents. In contrast, the newer yasbd-lib supports 39 languages and treats the input string as immutable, tracking index pointers throughout detection rather than modifying the text at all. As a result, yasbd-lib produces character spans natively during parsing, avoiding the post-processing overhead that PySBD requires.

0 comments Read more at DEV Community