How to Debug a Real Incident Using Metrics, Logs, and Traces Together

·2 views

A DEV Community tutorial series concludes by deliberately introducing a performance bug into a PHP mini-app to simulate a real production incident. The injected code causes 15% of requests to slow down by 2–4 seconds and 5% to return HTTP 500 errors, mimicking a failing payment service dependency. Using a pre-built observability stack — Prometheus, Loki, and OpenTelemetry — the author walks through how rising p95/p99 latency on dashboards first signals that something is wrong. Log queries then reveal a specific business-level error message pointing to a payment service timeout, narrowing the investigation further. Trace data from the OpenTelemetry collector confirms the two distinct request-duration clusters, linking slow requests and failures to the same root cause without ever viewing the source code.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Developer builds private AI home assistant using MQTT to avoid open ports or VPNs

A developer wanted to query a home GPU-powered Ollama instance from their phone without exposing their network to the internet. Instead of the usual Docker, reverse proxy, and VPN setup, they used MQTT — a publish/subscribe messaging protocol — as a neutral meeting point between their phone and home PC. Neither the home machine nor the browser client accepts incoming internet connections; all communication is routed through a rented Linux server running a Mosquitto MQTT broker. The entire system spans roughly 1,000 lines of code across a static browser page, a lightweight PC service, and a Python plugin handling tokens and Ollama calls. The developer noted alternatives like Tailscale and Cloudflare Tunnel but preferred MQTT for its simplicity, device-agnostic access via URL, and reliance on broker-level authentication rather than a custom auth layer.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Automate Email Verification Testing in Playwright CI Pipelines

End-to-end testing of email verification flows presents a challenge when Playwright triggers a signup and must then retrieve and act on a verification email automatically. Using real inboxes like Gmail introduces complications around CI access, parallel test runs, and stale messages interfering with results. A cleaner approach involves generating unique test email addresses programmatically and routing them to a dedicated test inbox accessible via API. Once the verification email is received, tests can assert its content, extract the verification link or OTP, and complete the full user journey without any manual intervention. This method enables fully automated, unattended email verification testing in CI/CD pipelines.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Convolvger 0.5.0 lets you archive AI chats from ChatGPT, Claude, Gemini, and Grok locally

Convolvger, an open-source command-line tool, allows users to save AI conversations from major platforms as local Markdown and JSON files without requiring an account or sending telemetry. Version 0.5.0, the latest release, adds support for Grok alongside existing support for ChatGPT, Claude, and Gemini. The tool works around each platform's technical limitations — such as app-shell pages and browser-only restrictions — by querying the same API endpoints the web interfaces use or employing a bookmarklet approach. Archived files can be independently verified, and the tool accurately handles edge cases like deleted shares, withheld files, and inconsistent sender labels. Convolvger requires Python 3.12 or higher, is MIT licensed, and is available via pip, uv, or pipx.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer's Blockchain Reconciliation Tool Caught Five Bugs in Its Own Code First

A developer building an open-source tool called onchain-tieout — designed to verify whether a wallet's reconstructed transaction history matches the balance reported on-chain — discovered the first errors it flagged were in the tool itself. A wrong assumption about Etherscan's pagination limit caused silent data truncation, while a single block containing over 1,000 token transfers exposed a flaw in the cursor logic. Separately, HTTP 429 rate-limit responses were being misclassified as unreadable token contracts, masking network errors as data facts. Two additional bugs were found through manual code review rather than testing, both involving ambiguous inputs being silently converted into plausible but incorrect outputs. When fully corrected and run against vitalik.eth, the tool processed 10,476 balance rows, with 8,281 tying out exactly — most discrepancies traced to airdropped spam contracts.

0 comments Read more at DEV Community