How to safely replay dead-lettered messages in RabbitMQ without causing new incidents

·1 views

When a bug that caused message failures is fixed, engineers often need to replay hundreds of dead-lettered messages back to their original queues in RabbitMQ. RabbitMQ dead-letters messages under conditions such as consumer rejection, TTL expiry, queue length limits, or delivery limit breaches, attaching an x-death header that records the message's origin and failure reason. Before replaying, teams should confirm the root cause is resolved, identify the correct destination, and filter out any poison messages still in the dead-letter queue. Options for replaying include the management UI's Move Messages panel, a dynamic shovel via rabbitmqctl, or a custom script using a client library like Python's pika. Each approach has trade-offs around selectivity, x-death header cleanup, delivery confirmations, and auditability that determine whether the replay succeeds or triggers a second incident.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Why System Thinking and Verification Matter More Than Coding Speed in the AI Era

AI coding tools can now generate standard boilerplate and routine functions in seconds, compressing the time engineers spend on initial implementation. However, this shift moves the primary bottleneck from writing code to verifying it, as machine-generated output still requires deep scrutiny for logic errors, security flaws, and edge cases. AI assistants also lack visibility into an organisation's legacy systems, deployment pipelines, and undocumented constraints, making human judgment essential for integration work. As raw code output becomes easier to produce, the ability to translate ambiguous requirements into sound technical decisions grows more strategically valuable. Experts argue that engineers who invest in system architecture, rigorous testing, observability, and debugging skills will be best positioned in an AI-assisted development landscape.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Builds Incident-Response AI Agent Trained on 104 Real Tech Postmortems

A software developer built an AI incident-response agent using the OpenSRE dataset of 114 real postmortems from companies including Slack, GitHub, AWS, and Cloudflare. The developer retained 104 incidents in a memory bank, which were processed into 946 discrete memories connected by over 7,000 links. The core motivation was teaching the agent to recognize 'trap actions' — remediation steps like rollbacks or restarts that historically worsened outages rather than resolved them. In a test scenario involving a checkout service returning errors after a deployment, the memory-equipped agent correctly identified a likely dependency issue and warned against rollback, while the same model without memory fabricated details and recommended the potentially harmful rollback. The developer acknowledges no formal comparison was run against synthetic data, framing the project as a practical experiment rather than a rigorous benchmark.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Claude Task Master CLI Automates Full Pull Request Lifecycle Without Manual Git Commands

Claude Task Master, a command-line tool invoked via the claudetm CLI, automates the entire pull request workflow by running an autonomous loop through four stages: planning, working, PR lifecycle, and verification. The tool reads a codebase, generates tasks, writes and commits code, opens pull requests, handles CI failures, responds to review comments, and merges approved PRs automatically. A state persistence feature saves progress to a local file, allowing interrupted runs to resume exactly where they left off, making it suitable for long-running development tasks. The tool also exposes a REST API and an MCP server, enabling integration with CI pipelines, dashboards, and custom orchestration systems via a simple JSON contract and configurable webhooks. Released under the MIT license, Claude Task Master supports multiple parallel profiles with isolated credentials, allowing teams to run different API configurations simultaneously.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How Industrial AIoT Pipelines Turn Raw Device Signals Into Operational Insights

Industrial IoT environments in factories generate heterogeneous data from RFID readers, BLE beacons, UWB systems, forklifts, and enterprise platforms like ERP and MES, making data normalization a core engineering challenge. A well-designed event pipeline converts raw technical signals — such as a tag being detected — into enriched operational events that include asset identity, production context, and location. Enriching location data with inventory status, production orders, and timestamps makes downstream analytics significantly more actionable for operations teams. Data quality fundamentals, including timestamp consistency, duplicate detection, and device identity validation, must be established before machine learning can reliably extract value. An edge computing layer distributed across the facility enables near-real-time event processing closer to the source, supporting both operational responsiveness and advanced analytics.

0 comments Read more at DEV Community