SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

How a 60-second scrape interval hid 30-second outages for weeks

0
·2 views

An engineering team spent two weeks dismissing customer reports of 30-second site hangs because their monitoring graphs showed no anomalies. The root cause was a compounding blind spot: a 60-second scrape interval, a 5-minute rate window, and a 5-minute alert hold-off meant any incident shorter than roughly four minutes was mathematically invisible. Both the alerting rules and the investigation dashboards relied on the same averaged expressions, so the team had no independent signal to fall back on. The fix involved switching from smoothed rate thresholds to raw counts of slow or failed requests over short windows, and using max_over_time to capture spikes rather than averaging them away. On the first day the revised rules went live, three alerts fired before midday, each lasting around 40 seconds and each matching an existing support ticket.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Fresh Cloud Servers Hit With Hundreds of Intrusion Attempts Within Minutes of Going Online

A developer rented three servers in Frankfurt, New York, and Singapore without registering them in DNS or sharing their addresses, then monitored unsolicited connection attempts for roughly 45 minutes per region. The first uninvited connection arrived just 48 seconds after the Singapore server booted, with Frankfurt and New York following shortly after. In total, the three servers logged 3,480 connection attempts from 860 unique IP addresses, including over 1,000 attempts using real login credentials. The most targeted port was telnet (port 23), reflecting persistent botnet activity aimed at IoT devices such as routers and cameras. The experiment, which cost under three cents, showed that any public IP address faces roughly 700 to 800 unsolicited connection attempts per hour simply by being reachable on the internet.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

DDL vs DML in SQL: Understanding the Core Difference Between Structure and Data

DDL (Data Definition Language) and DML (Data Manipulation Language) are two fundamental categories of SQL commands that serve distinct purposes. DDL commands such as CREATE, ALTER, DROP, TRUNCATE, and RENAME deal with the structure of database tables — defining, modifying, or removing them entirely. DML commands, including SELECT, INSERT, UPDATE, and DELETE, operate on the actual data stored within those tables without altering their structure. A key practical distinction is that DROP deletes an entire table along with its data, while TRUNCATE removes only the rows but preserves the table structure. In short, DDL shapes the container, while DML manages what goes inside it.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

SQL Window Functions vs GROUP BY: Key Differences Every Beginner Should Know

Both GROUP BY and window functions help summarize data in SQL, but they behave differently in how they return results. GROUP BY collapses multiple rows into a single summary row per group, losing individual row details in the process. Window functions, by contrast, retain every original row while adding a new calculated column alongside each one. Functions like AVG() with OVER/PARTITION BY, RANK(), and LAG() allow comparisons and rankings without reducing the dataset. Knowing which tool to use depends on whether you need a condensed summary or per-row calculations with context.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

How to Write Status Updates That Actually Help Teams Make Decisions

A post by Asael Shinder on DEV Community argues that most workplace status updates fail because they list completed tasks rather than answering what stakeholders truly need to know. Effective updates should convey three things: where the work stands against its deadline, what has changed since the last update, and what is needed from whom and by when. The author distinguishes between vague terms like 'blocked' and actionable ones, such as 'blocked on a decision from Marco by Wednesday.' Slippages should be communicated as soon as they are anticipated, not after they become unavoidable, so others can adjust their plans. A well-structured update, the author contends, can take three minutes to write yet eliminate the need for follow-up meetings entirely.

0 comments Read more at DEV Community