How a Single Mismatched Health Check Path Silently Broke an ECS Fargate Service

·7 views

An engineering team discovered their AWS ECS Fargate service had been silently failing for days after a routine deployment check revealed zero running tasks against a desired count of one. The root cause was a health check still pointing to /api/health while a recent code refactor had moved the endpoint to /health, causing ECS to repeatedly kill and restart the task every 60 seconds. Because no alerts were configured for the RunningTaskCount metric, the failure went completely unnoticed despite burning compute resources throughout. The team identified three available but unwatched signals: ECS console service events, the CloudWatch RunningTaskCount metric, and the absence of application logs. To prevent recurrence, they introduced a pre-deploy health check path verification step, a CloudWatch alarm that triggers a Slack notification when running tasks fall below the desired count for five or more minutes, and EventBridge-based streaming of ECS service events to CloudWatch Logs.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

A single AWS Security Group change silently broke a healthy application

A developer discovered that modifying one egress rule in an AWS Security Group caused their EC2 application to lose connectivity to its database, despite all instances appearing healthy. No deployment had occurred, and CPU metrics were normal — the only symptom was a connection timeout. The root cause was an accidental restriction on outbound traffic from the application's Security Group, which severed the return path for database connections. Although AWS Security Groups are stateful, an overly restrictive egress rule can still block traffic at the network policy level without triggering any health alerts. The incident highlighted that cloud debugging must go beyond server status checks to examine port-level, directional, and path-specific network permissions.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Properly Integrate OpenAI API Into a Production App

Integrating OpenAI's API into a prototype is straightforward, but deploying it in a production environment requires significant additional engineering work. Teams must place the API behind a server-side backend to protect credentials, as exposing API keys in client-side code can lead to misuse and unexpected costs. Input and output handling, cost controls, and observability layers are essential components that separate a reliable feature from a fragile demo. Unlike traditional web services, language models can fail silently by returning confident but incorrect responses, making monitoring especially critical. Engineers who skip these safeguards typically face costly retrofitting later, making deliberate, structured integration the more efficient long-term approach.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How Node.js Handles Thousands of Requests on a Single Thread: The Event Loop

Node.js is single-threaded yet handles massive concurrency through a mechanism called the event loop, which offloads slow operations like file reads and network calls to the system and retrieves results asynchronously. Each iteration of the event loop cycles through ordered phases — timers, I/O polling, setImmediate callbacks, and close callbacks — before moving on. Between every phase, microtasks such as resolved Promises and process.nextTick() callbacks are drained first, meaning they execute before the loop advances to its next stage. This explains why a resolved Promise logs before a zero-delay setTimeout in execution order. Developers are advised to avoid blocking the event loop with heavy CPU tasks, and instead use worker threads, the cluster module, or external queues for compute-intensive work.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Install and Configure OpenLiteSpeed Web Server on Ubuntu 22.04

OpenLiteSpeed is a free, open-source web server supporting HTTP/2, HTTP/3, and native PHP, installable on Ubuntu 22.04 via an official setup script since it is not available in default repositories. After installation, the server runs as a systemd service and provides a graphical admin console accessible on port 7080 for managing configuration. Administrators can create virtual hosts with custom web roots, configure access and error logging, and enable URL rewriting through the console interface. TLS certificates can be obtained using Certbot's webroot method and then manually linked to an HTTPS listener on port 443, since no native OpenLiteSpeed plugin exists for Certbot. The full setup results in a secured, production-ready web server serving content over HTTPS with automatic certificate renewal support.

0 comments Read more at DEV Community