Custom local LLM engine decodes 3.6x faster under concurrent load but trails on cold prompts

·1 views

A small team running AI agents on one mini PC built its own inference engine, unum-rt, to speed up decoding while another request is loading a long prompt. In one benchmark row, it decoded at 45.1 tokens per second against 12.4 for the open-source engine it started from, a 3.6x gap. Across a round-two tally, it won three rows, tied one, and lost six, mostly on long cold prompts where the open-source engine read tokens faster and reached first token sooner. The team also reports higher graphics-memory use after its benchmark. The results come from one hardware setup, mostly single runs, and two differently packed versions of the same 4-bit model, and the engine is not public.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Developer finds his own cron job returns success while every fetch fails silently

A developer who wrote about monitoring small products admitted his own repos lack proper error reporting, relying only on console.error logs read in Vercel. In Upwork Scout, a cron job that scans job listings wraps each category fetch in Promise.allSettled, so when every fetch fails, such as from an expired token, the run still returns HTTP 200 and passes health checks. The AI scorer has a similar flaw, where a failed Claude call silently degrades the pipeline to filters-only instead of raising an alert. In VoiceDash, webhook forwarding uses unawaited fetch calls with empty catch blocks, so failed deliveries are dropped while Retell still receives a success response. The author argues these are wrong-but-working failures that only silence-based alerting would catch.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

React Server Components Flight Protocol Flaw Enables Unauthenticated RCE (CVE-2026-126187)

A critical deserialization flaw, tracked as CVE-2026-126187 with a CVSS 4.0 score of 9.5, has been disclosed in React Server Components' Flight wire protocol. Attacker-controlled bytes can break the protocol's framing and smuggle in a module reference disguised as data, which the server then executes. When combined with Next.js 16's Cache Components feature, this can lead to unauthenticated remote code execution on the server. The affected packages include react-server-dom-webpack, react-server-dom-parcel, and react-server-dom-turbopack across several 19.x version ranges. Patched versions 19.0.9, 19.1.10, 19.2.9, and 19.3.1 were released, with the fix merged on October 14, and a public proof-of-concept now exists.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Otomoto Scraper Tutorial: Pulling Polish Car Listings into Python

This tutorial explains how to use the Otomoto Scraper Apify Actor to extract passenger-car listings from otomoto.pl, a Polish car marketplace, into structured data. The Actor returns one row per listing with price, make, model, year, mileage, fuel, gearbox, engine power, location and seller details. Readers need Python 3, a free Apify account and an API token, plus the apify-client package installed. The search is controlled through input fields such as search terms, price and year ranges, fuel and seller type, sorting and result caps. The guide also shows how to verify that the returned rows match the search term and notes that some fields may be empty.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Why VAT Results Need Provenance to Be Explainable

A VAT calculation can be arithmetically correct yet hard to trust later, since a bare result does not show where its rate came from or which data and versions produced it. Calculation provenance links a financial result to its rate data, mapping versions, effective dates, and calculation contract. Two calculations using the same 19% rate can still differ in their evidence paths, so matching numbers do not establish matching provenance. The VAT Engine keeps the numeric rate separate from structured evidence describing its origin, including states such as verified, legacy_unverifiable, and unavailable. Such graded evidence states convey uncertainty more accurately than a simple true or false verification flag.

0 comments Read more at DEV Community