Hosted LLM Providers Show Up to 40% Throughput Gap on Identical Models

·11 views

A DEV Community deep dive examines why the same open-weights model can deliver very different speeds depending on the provider hosting it. Citing NeuBench speed tests, the article reports token throughput differences of up to 40% across major hosted endpoints. It explains that inference runs in two phases, a compute-bound prefill step that sets time to first token and a memory-bandwidth-bound decoding step that sets throughput. The variance is attributed to choices such as the inference engine, hardware tier and interconnect, and KV cache quantization or offloading. The piece closes with a Python example that streams completions to measure TTFT and end-to-end throughput so engineers can monitor providers in production.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

model-radar: a tool to find free LLMs that actually make tool calls

The author built two small commands, model-radar and model-canary, to run alongside AI agents. model-radar lists free OpenRouter models and tests whether each one really makes a tool call, since many claim support but respond in prose only. model-canary monitors the paid model and credit balance, failing over to a cheaper or local model when needed. Setup takes about ten minutes with Python 3.11 or newer and requires no pip installs, and the guide is tested weekly on a fresh Ubuntu machine. The tools can run without credentials to verify installation, though probing and failover require an OpenRouter account. The author suggests running them periodically via cron, with failover triggered after two consecutive hard failures.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Study finds AI-generated code often compiles but fails security checks

Veracode's 2026 tests found AI models produced code that compiled almost every time but failed security checks on about 44% of tasks, with no meaningful improvement since 2025. Earlier research on GitHub Copilot found roughly 40% of generated programs were vulnerable, and a 2023 study showed developers using AI assistants wrote less secure code while feeling more confident in it. Models also frequently suggested nonexistent software packages, which attackers could exploit by publishing malicious ones under those names. Experts recommend treating AI output like code from an unfamiliar contractor, with careful review, security scanning, and testing before production use rather than banning the tools.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Google Dismantles XCPE Botnet Preinstalled on 1.3 Million Android Set-Top Boxes

Google and its partners disrupted XCPE, a backdoor that had enrolled more than 1.3 million Android set-top boxes into a botnet. Google Threat Intelligence Group, which published its findings on 2 November 2026, attributes the malware with high confidence to Videostrong Technology, an Android device manufacturer. The backdoor was embedded in device firmware, ran with system-level privileges, and reinstalled itself when users tried to remove it. Affiliates reportedly used the compromised devices for residential proxy services, ad fraud, and additional malware. As a result, Google cut ties with Videostrong, barred it from producing certified Android devices, and took down the command infrastructure, cutting off an estimated 10 million more compromised devices.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Using If-Match with ETags to stop stale edits overwriting newer changes

When two editors change the same settings page, an unconditional save can let the later request silently restore older data. A strong ETag sent with an If-Match header lets the server apply an update only if the client's copy is still current. A mismatch returns 412 Precondition Failed, prompting the editor to reconcile rather than overwrite. The article includes a sample HTTP exchange, a small Node.js model demonstrating the rule, and a note that weak tags cannot satisfy If-Match. It also warns that checking the version and writing in separate steps can reintroduce races, so the check and update should be a single conditional database operation.

0 comments Read more at DEV Community