FP8 KV Cache Bug Crashes vLLM on Workstation Blackwell GPUs at Startup

·14 views

Running vLLM with FP8 KV cache on DeepSeek MLA models causes an out-of-shared-memory crash on workstation and consumer Blackwell GPUs, including the GB10 and RTX PRO 6000. The root cause is a 1,024-byte gap between the per-SM shared memory budget of 102,400 bytes and the per-block opt-in limit of 101,376 bytes on sm_120 hardware. With FP8 KV, the MLA decode kernel's second pipeline stage requires exactly 102,400 bytes, hitting the per-SM ceiling but exceeding the per-block cap. A proposed upstream fix pins num_stages to 1 for MLA kernels, which allows the server to start and generate output correctly on sm_120. Developers have flagged that a blanket fix sacrifices throughput for bf16 KV paths, and recommend computing actual tile shared-memory usage against device limits instead.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

LeeX CEO Oscar Awowari on Building a Geospatial Search Layer for City Discovery

Oscar Awowari, Founder and CEO of LeeX, is developing a city discovery platform that goes beyond conventional text search to enable location-based discovery of businesses, events, and infrastructure. The core engineering challenge lies in building a dedicated geospatial search layer that can answer spatial queries such as finding places within a set radius or within a geographic boundary. To handle this efficiently, the system relies on spatial indexing, which narrows search candidates to a relevant geographic region rather than scanning every location in the database. Techniques like radius search and bounding box filtering are used in combination, applying cheaper broad filters first before running more precise distance calculations. Awowari emphasizes that scalability must be considered from the outset, as the platform is designed to grow from thousands to potentially millions of indexed locations across a city.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer releases free Node.js CLI tool to detect hreflang SEO errors

A developer has published an open-source, zero-dependency Node.js command-line tool called hreflang-audit, designed to identify common international SEO errors in hreflang implementations. The tool crawls websites or parses XML sitemaps and runs 11 checks covering issues such as invalid ISO language codes, missing return links, broken target URLs, and canonical conflicts. Each detected issue is accompanied by a fix hint, and output can be generated as a human-readable report or in JSON format for use in automated pipelines. A GitHub Action is also included, allowing teams to schedule weekly audits and trigger build failures if hreflang regressions are detected. The project is MIT licensed and available on GitHub, with sitemap-versus-HTML mismatch detection planned as a future addition.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Zero-Trust Authorization Framework Proposed for Debt Collection Systems

A technical framework applying NIST SP 800-207 Zero Trust Architecture to debt collection operations has been outlined, separating authorization decision logic from execution logic. The design introduces a Universal Action Request Contract that uses cryptographic pseudonymous references instead of raw personal data, ensuring privacy across communication channels. Incoming evidence is classified into four states — missing, stale, conflicting, or invalid — to enforce strict, verifiable authorization checks before any collection action is taken. The architecture mandates deterministic decision engines over AI language models for final authorization rulings, with LLMs limited to fact extraction and record reconciliation. Short-lived, sender-constrained permits based on RFC 9449 are proposed to replace conventional bearer tokens, reducing risk in high-consequence collection operations.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Step-by-Step Guide to Installing Graphify for Use with Claude CLI

Graphify is a command-line tool that converts a codebase, configuration files, or documentation into a queryable knowledge graph, outputting files such as graph.json and GRAPH_REPORT.md. The tool is installed globally on Linux or WSL using uv, a Python package manager, and only project-specific configuration needs to be repeated per repository. Users must install the appropriate backend client extra — such as the Anthropic, Gemini, or OpenAI package — depending on which large language model they intend to use for semantic extraction. API keys for the chosen backend must be set as environment variables, with a noted caveat that quoted values in .env files can cause authentication errors if not stripped correctly. The guide also recommends limiting graph extraction to relevant project paths and using the --max-concurrency 1 flag when hitting rate limits on free-tier LLM APIs.

0 comments Read more at DEV Community