How to Deploy the 600GB Inkling-NVFP4 Model on GKE Spot A3 Without Crashing

·45 views

Developers attempting to run the 600GB Inkling-NVFP4 AI model on a cost-discounted Google Kubernetes Engine Spot A3 instance — equipped with 8 NVIDIA H100 GPUs and 640GB of VRAM — frequently encounter crashes before a single prompt is processed. A key software issue is an ABI mismatch caused by installing bleeding-edge packages like vLLM and Transformers on the default Ray cluster image, which upgrades NumPy to version 2.0 while the base system expects version 1.0, breaking inter-node communication. The recommended fix is to use the official vllm/vllm-openai:v0.26.0 image for both Head and Worker nodes, as it bundles compatible versions of all required libraries and eliminates download timeouts during startup. On the hardware side, the model's weights alone consume roughly 600GB of the available 640GB VRAM, leaving almost no room for the KV Cache — the GPU memory used for short-term context — which defaults to reserving space for up to one million tokens and triggers an Out-Of-Memory crash. Tuning the KV Cache allocation is therefore a necessary step alongside the software environment fix to successfully serve the model in production.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Node.js 26.9 enables FFI by default, benchmarks show 37ns per call

Node.js 26.9.0, released on 16 September, makes the node:ffi module available by default, removing the need for the --experimental-ffi flag that was previously required in earlier 26.x versions. The change inverts the flag logic: --no-experimental-ffi is now needed to disable the feature, marking a genuine behavioral shift rather than a documentation update. Benchmarks comparing node:ffi against N-API addons show FFI is roughly 7-8% slower per call at around 37 nanoseconds, meaning it does not offer a performance advantage over existing native addons. However, FFI outperforms N-API by 15-20% in bulk buffer operations, where a single call processes large datasets, making it better suited to high-volume data transfers than frequent small calls. The key practical benefit of node:ffi remains convenience — it eliminates the need for a compiler toolchain, node-gyp, or per-ABI rebuilds in deployment environments.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer fixes hidden UX flaw that drove 80% of users away before first run

A developer noticed a sharp drop-off in usage for one of their Apify data-fetching tools, where only 1 in 5 users who opened the input page actually ran the actor. The culprit turned out to be a raw JSON textarea field requiring users to manually enter a structured array of objects — an unintuitive barrier for casual users. The fix involved adding simple, flat individual input fields that internally map to the existing array format, making the tool accessible without breaking advanced multi-item functionality. After spotting the pattern in one tool, the developer audited their entire portfolio and found four more actors with the same issue, applying the same fix across all of them. The incident highlighted that UX friction invisible in code reviews can only be detected by analyzing where real users abandon a product.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Proxmox VE Auth Bypass Flaw Exposes Tens of Thousands of Instances Online

A critical authentication bypass vulnerability, tracked as PSA-2026-00043-1, affects Proxmox Virtual Environment versions 7.x through 8.0.3, allowing attackers to gain passwordless root access by exploiting a flaw in the tfa-challenge parameter handling. Internet scans using application fingerprints identified roughly 34,000 to 400,000 Proxmox instances publicly reachable online, depending on the query method used. Security researchers cautioned that a raw port 8006 count of over four million is misleading, as it reflects all services on that port rather than confirmed Proxmox deployments. One affected version branch lacks a vendor patch, meaning some exposed systems cannot be fully remediated through upgrades alone. Administrators are advised to enable two-factor authentication on all accounts, restrict public access to port 8006, and audit logs for unauthorized root logins.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer shares key lessons from 22-day hands-on AI systems building sprint

A software developer documented a 22-day self-directed learning sprint focused on building practical AI systems, covering topics from LLM fundamentals to retrieval-augmented generation (RAG) and automated evaluation. The experiment corrected several common misconceptions, including the true purpose of RAG — retrieving relevant facts at request time rather than serving as a privacy tool — and clarified how embeddings differ from next-token prediction mechanisms. Using tools like Ollama and local models, the developer demonstrated that LLMs are stateless by design, with apparent memory in chat apps being the result of deliberate state management by the application layer. Tokenization tests revealed that prompt costs and latency vary significantly by language, with Hindi requiring roughly 2.5 times more tokens per word than English. The overall takeaway was that a useful AI feature requires the same engineering discipline as any production software: defined inputs, constrained behavior, observability, and measurable evaluation criteria.

0 comments Read more at DEV Community