27B LLM Squeezed Under 6GB via Ternary Quantization, But CPU Speed Disappoints

·1 views

A developer tested Prism's Ternary-Bonsai-2-27B, a ternary-quantized large language model that compresses a 27-billion-parameter model into under 6GB, on a budget Hetzner VPS with no GPU. The model uses an aggressive PTQ1_0 format where each weight is reduced to one of three values, requiring Prism's custom llama.cpp fork since the format is not yet supported upstream. While Simon Willison had reported 20–44 tokens per second on Apple Silicon with Metal acceleration, the CPU-only test on the VPS yielded just 2–3 tokens per second. At that speed, the model is viable for background batch processing but impractical for real-time interactive use. The tester also noted an unexplained throughput variance of up to 40 percent between server restarts with no configuration changes.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Why AI Systems Need New Observability Metrics Beyond Latency and Uptime

Traditional software monitoring tools are inadequate for AI-native systems because they measure technical success rather than output quality, missing failures like hallucinations, biased responses, or unsafe content. A technically successful HTTP 200 response can mask serious problems such as fabricated information, ignored retrieval context, or policy violations. Large language models introduce non-deterministic behavior, meaning identical inputs can produce varied outputs and quality can degrade silently after model or prompt updates. Experts advocate for a dual observability strategy that combines conventional infrastructure metrics with AI-specific Service Level Indicators (SLIs) measuring trustworthiness, relevance, and safety. Without these specialized SLIs, organizations risk deploying AI systems that appear operationally healthy but are fundamentally unreliable or harmful in real-world use.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

URL Encoding Pitfalls: Why %20, +, and Special Characters Break APIs

URLs are composed of distinct parts — scheme, host, path, query string, and fragment — each with its own encoding rules, and treating them as uniform text is a common source of bugs. Characters like &, #, =, and ? carry structural meaning in URLs, so user-supplied data containing these must be properly encoded before being inserted into a URL component. A frequent source of confusion is that spaces can be represented as either %20 or + depending on context: standard percent-encoding uses %20, while HTML form encoding uses +, and a literal plus sign must be encoded as %2B to avoid being misread as a space. JavaScript developers often misuse encodeURI() and encodeURIComponent(), where the former preserves URL-structural characters and is unsafe for encoding individual values, while the latter correctly encodes characters like & and = within a single component. Using dedicated APIs such as URLSearchParams and the URL constructor is generally the safer and clearer approach for building URLs with dynamic query parameters.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer audits 2 months of Claude Code sessions, finds $1,675 in API costs and 117 risky deletes

A developer building an AI safety tool called Paveo replayed 59 days of their own Claude Code sessions to stress-test the product before releasing it publicly. Across 106 sessions and over 8,000 model calls, the usage would have cost $1,674.75 at standard API rates, with one single session reaching $452.92. A starter safety policy, when applied retroactively to all tool calls, would have blocked 670 actions, including 117 recursive force-delete commands and 3 forced git cleans. The developer notes that not every blocked command would have been a genuine mistake, but argues that a false refusal is preferable to an unrecoverable data loss. Paveo, which runs locally without any network access and supports multiple AI coding tools, is set to become publicly available this week.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Strong Coding Fundamentals Matter More Than AI Prompting Skills, Developer Argues

A developer named Cesar Aguirre published a career-focused article on DEV Community on September 28, addressing anxiety around artificial intelligence in the tech industry. The piece targets programmers experiencing 'FOMO' — fear of missing out — over the rapid rise of AI tools. Aguirre argues that solid programming fundamentals provide more lasting value than simply learning to craft AI prompts. His central message is that chasing AI hype is less effective than building timeless core skills as a developer.

0 comments Read more at DEV Community