Empty Grafana Panels Can Hide Broken Queries Without Any Alert

·2 views

A DEV Community article warns that dashboard panels can sit blank for months because their underlying metric names no longer exist, and Grafana shows no error for this. Prometheus treats a query for a missing metric as valid and returns an empty result, so the panel appears calm even during incidents. Causes include renamed metrics, altered scrape jobs, or deleted recording rules. The author proposes a manual audit to find dead panels and a CI check to catch broken dashboards before code merges. The piece also flags upcoming risk from Kubernetes v1.37, which enables native histograms by default and could silently remove classic histogram series.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Understanding RPO, RTO and DynamoDB Point-in-Time Recovery for AWS Disaster Recovery

This second part of an AWS high-availability series explains the two core recovery metrics: recovery point objective, the maximum tolerable data loss measured in time, and recovery time objective, the maximum tolerable downtime. The author stresses that these targets are business decisions rather than engineering ones, with engineers responsible for quantifying the cost of each option. It highlights the consequences of missing them, including SLA penalties, regulatory fines and reputational harm. The piece then introduces DynamoDB Point-in-Time Recovery, which continuously captures table changes so a table can be restored to nearly any second within a rolling window of up to 35 days.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Passing Checks Can Hide How Little Room Systems Have Left

A nightly job that once finished in 41 minutes now takes 88 minutes against the same 90-minute window, yet both results show as a pass on the dashboard. The article argues that pass/fail reporting discards the margin, meaning the distance between current performance and the point of failure. It recommends expressing margins in the units of what varies, such as time until a limit is reached, rather than as percentages. The piece distinguishes three types: headroom for capacity, tolerance for correctness, and freshness for how long evidence remains valid. It notes that tolerance and freshness erosion often goes unnoticed because no alert fires while the figures are still inside their thresholds.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

SessionStart Hook Feeds Claude Code Failing Tests Before Its First Turn

A developer describes a way to make Claude Code aware of failing tests from the outset. Claude Code's SessionStart hook can add plain-text output as context before the first prompt. A short script can run fast pytest tests, keep only the failures, and pass that list to Claude. The author notes that output is capped at 10,000 characters and that the wording should be factual rather than phrased as commands, to avoid triggering prompt-injection defenses. The approach uses a standard-library-only Python script saved in the project's .claude/hooks directory.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Linux nosymfollow mount flag blocks symlink following on one mount, not all path risks

Linux 5.10 introduced the MS_NOSYMFOLLOW mount flag, which stops path resolution from following symbolic links on a specific mounted filesystem. It suits content roots and upload directories where a symlink should never serve as an escape route to other host paths. The protection is limited to that mount, so paths crossing into other mounts or links created before the mount may behave differently than operators expect. Administrators are therefore advised to treat it as one layer alongside application-level path validation, sandboxing, and least-privilege service accounts. Testing on the actual production storage path, along with logging denials and enforcing the option through configuration management, is recommended.

0 comments Read more at DEV Community