SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

Engineer Debugs vLLM KV Cache Corruption That Triggered 14,720 RPS Incident

0
·1 views

A software engineer received an on-call alert at 8am after their vLLM-based model serving system experienced a KV cache corruption issue during peak traffic of 14,720 requests per second. The system, which uses PagedAttention for model serving, began throwing cache corruption errors that brought operations down. Three initial fixes — adjusting NCCL timeouts, restarting Kubernetes pods, and checking GPU health — each failed, consuming roughly 45 minutes of debugging time. The root cause was ultimately resolved with a single-line change in the Dockerfile. To prevent recurrence, the team added Grafana and PagerDuty alerts monitoring NCCL communication errors and all-reduce latency thresholds.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Why 'Unexpected Token' JSON Errors Are Usually Not a Parser Problem

When developers encounter a SyntaxError from JSON.parse() or res.json(), the root cause is often not a malformed JSON file but a flawed HTTP request returning HTML instead of JSON. Common triggers include wrong URLs, missing authentication, or expired tokens, which cause the response to begin with HTML markup rather than valid JSON. In cases where the JSON itself is genuinely malformed, typical culprits are trailing commas, single quotes, unquoted keys, comments, or Python-style values like True and None. Double-parsing a fetch response is another frequent mistake, since res.json() already handles parsing and calling JSON.parse() on top of it breaks the flow. Developers can diagnose and fix these issues using online validators to locate the exact error line and repair tools that convert lenient syntax to strict JSON.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Student Builds On-Device AI Security App Aigris to Detect Phishing Without Cloud

A second-year Computer Engineering student has developed Aigris, an experimental Android application designed to protect users from phishing, malicious URLs, and social engineering attacks using AI that runs entirely on-device. Unlike most security tools that rely on remote cloud servers, Aigris processes data locally on the smartphone to preserve user privacy and reduce latency. The app is built using Kotlin, ONNX Runtime, and on-device machine learning models optimized to work within mobile performance and battery constraints. Current features focus on phishing detection, suspicious URL analysis, and real-time threat alerts, with future plans including social engineering detection and browser protection. The developer has open-sourced the project to invite community feedback and collaboration as development continues.

0 comments Read more at DEV Community
0
ProgrammingHacker News ·

European Organizations Increasingly Ban Personal Messaging Apps at Work

A growing number of European organizations have implemented bans on personal messaging apps in professional settings. The trend reflects mounting concerns over data security, privacy compliance, and regulatory requirements such as GDPR. Both public institutions and private companies are restricting tools like WhatsApp and Telegram in favor of approved enterprise communication platforms. The compiled list, published by Birdy Chat, highlights how widespread this policy shift has become across the continent.

0 comments Read more at Hacker News