Apache SeaTunnel detects Kafka partition expansion in real time without job restart

·2 views

Apache SeaTunnel streaming jobs, by default, do not consume data from newly added Kafka topic partitions unless the job is restarted, risking data loss or duplicate records. To address this, SeaTunnel provides the partition-discovery.interval-millis parameter, which periodically scans Kafka for new partitions and adds them to the consumer automatically. In a demonstration using SeaTunnel 2.3.12, a Kafka topic named 'ksource' was expanded from 2 to 4 partitions while a streaming job was running. With partition-discovery.interval-millis set to 5000 milliseconds, the job detected the new partitions within seconds and began consuming messages from them without interruption. This approach eliminates the need for a job restart and avoids the associated risks of downtime, missed data, or replayed historical records.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Why Offline-First Architecture Should Be a Design Decision, Not an Afterthought

A developer building Aafiyat, a clinic management system, argues that offline-first architecture should be planned from the start rather than bolted on later. Unlike online-first apps that cache data and wait for reconnection, offline-first apps write data locally first and sync with the cloud only when connectivity is restored. Aafiyat uses Electron and SQLite to store patient records, prescriptions, and billing data locally, queuing changes for synchronization with Supabase once the network returns. The developer is applying the same principle to Nankana Home Care, a PWA for medical assistants using IndexedDB for local storage. While acknowledging challenges like conflict resolution and partial sync failures, the author contends that apps serving clinics, field workers, or areas with unreliable connectivity cannot afford to treat offline use as an edge case.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Understanding LINQ Deferred Execution: Why Your Query Runs Later Than You Think

LINQ queries in C# do not execute at the point of declaration; instead, they store a recipe for retrieval that only runs when iterated via methods like foreach, ToList(), or Count(). Because queries evaluate against the live data source each time they are enumerated, modifying the underlying collection between enumerations will affect the results. This behavior can cause serious performance issues, such as triggering multiple database calls from a single query variable used in different places. Developers can avoid redundant executions by materializing results early using ToList(), ToArray(), or ToDictionary(), which store a fixed snapshot in memory. Deferred execution is a deliberate design rooted in functional programming's lazy evaluation concept, introduced to C# when LINQ launched in 2007, and it offers real benefits when composing dynamic queries before a single database call is made.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

CISA gives federal agencies 3 days to patch two critical Cisco zero-days under active exploit

CISA added two critical Cisco vulnerabilities to its Known Exploited Vulnerabilities catalog within the same week in September, setting patch deadlines of just three days for U.S. federal agencies. The first flaw, CVE-2026-76461, is a root-level remote code execution bug in Cisco Secure Email Gateway triggered by a malicious email requiring no authentication or user interaction. The second, CVE-2026-76460, is an authentication bypass in Cisco Identity Services Engine with a perfect CVSS score of 10.0, affecting the system that controls network access permissions. Both vulnerabilities were already being actively exploited in the wild at the time CISA issued its advisories. Security experts warn that patching alone is insufficient, as attackers may have already planted persistent backdoors on compromised systems before patches are applied.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Why AI Agent Retry Logic Is Really an Event-Driven Architecture Problem

Synchronous call chains in multi-service backends appear reliable during demos but break down under real production load, as tail latency compounds across each service hop. When one downstream service slows or fails, the entire chain stalls — even services that completed their work correctly are blocked. Introducing a message queue decouples services so each can process events independently, preventing one slow component from backpressuring the rest. However, queues introduce their own challenges: at-least-once delivery semantics, duplicate messages, ordering guarantees, and poison message handling. The author argues these same distributed-systems problems resurface in AI agent pipelines that execute sequential tool calls, making retry logic a queueing problem in disguise.

0 comments Read more at DEV Community