SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

Open-Source Tool 'Waste' Streams Giant AI Models from NVMe to Cut RAM Use

0
·1 views

A new open-source GitHub project called 'Waste' offers a way to run extremely large AI models, such as the 2.78-trillion-parameter Kimi K3, on hardware with limited RAM. Instead of loading an entire model into memory, Waste streams only the required weights directly from NVMe storage during inference. Written in C and designed to be dependency-free, the tool is intended to integrate easily into existing development workflows. The approach could benefit engineers working in resource-constrained environments like edge devices, IoT systems, or low-memory cloud instances. However, developers are cautioned that streaming weights from storage may introduce latency, making performance benchmarking essential before production deployment.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Self-Hosting Open AI Models on AWS: Real Costs and Trade-offs Explained

Companies are increasingly exploring self-hosted open AI models such as Meta's Llama 4, DeepSeek, and Alibaba's Qwen as alternatives to costly commercial APIs, with some reporting savings of up to 70%. Open models use publicly downloadable weights that run on a company's own hardware, eliminating per-request fees paid to third-party providers. GPU memory is critical for practical deployment, as CPU-based inference is too slow for team use, generating only 2–5 tokens per second versus 30–80 on a GPU. The recommended software stack includes vLLM as the inference engine and Open WebUI for a browser interface, both open-source and compatible with existing OpenAI-based tools. Chinese open models have grown rapidly, now accounting for over 30% of enterprise traffic on OpenRouter, up from just 4.5% in early 2025.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Wide Events gem bundles Rails request context into one OpenTelemetry span for AI agents

Wide Events is a new Ruby gem designed to consolidate all relevant production context from a single Rails request or background job into one flat OpenTelemetry root span. The tool addresses a key blind spot for coding agents, which can read source code quickly but lack visibility into runtime details such as active feature flags, query counts, or which tenant experienced an issue. In a real production example, the gem helped pinpoint a 28.9-second outbound HTTP call by surfacing timing data on a single queryable row, compared to parsing 82 spans manually. Developers define their own application-specific attributes — such as account IDs or search outcomes — while the gem handles attaching them to the root span at request completion. Wide Events requires Ruby 3.2 and Rails 7.1 or later, and is compatible with any OTLP-compatible backend or can fall back to emitting a single JSON log line.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Anthropic Adds OS-Level Sandbox to Claude Code, but Key Gaps Remain

Anthropic has introduced OS-level sandboxing for Claude Code, using macOS Seatbelt and Linux bubblewrap to constrain what Bash commands and their child processes can access. Write access is restricted by default to the current working directory and session temp, but read access spans nearly the entire filesystem, leaving sensitive files like ~/.aws/credentials and ~/.ssh exposed unless manually configured. The sandbox fails open by default, meaning it silently disables itself if unavailable, and does not run at all on native Windows. Anthropic's own documentation acknowledges that the feature reduces risk but does not constitute a complete isolation boundary, with explicit warnings about TLS inspection gaps and Docker socket exposure. Security analysts note that sandboxing addresses containment but does not resolve consequence or recovery — scenarios where a syntactically correct command causes unintended large-scale damage remain outside its scope.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Six CSS and SVG techniques to recreate Apple's Liquid Glass effect on the web

Apple introduced its Liquid Glass visual design across iOS 26 and macOS, prompting web developers to replicate the effect using CSS and SVG. A developer spent several weeks researching the technique and found that most online guides rely solely on backdrop-filter blur, which produces a dull, washed-out result rather than genuine glass. The key missing ingredients include boosting color saturation alongside the blur, adding a subtle background tint, and using mask-image to create soft, feathered edges. More advanced approaches involve stacking multiple blur layers to simulate gradient blur, since CSS does not natively support it. The developer documented six techniques in total, ordered by browser support, along with common pitfalls such as ancestor overflow clipping and Safari prefix requirements.

0 comments Read more at DEV Community
Open-Source Tool 'Waste' Streams Giant AI Models from NVMe to Cut RAM Use · ShortSingh