How to Run Honest Edge LLM Benchmarks on Devices Like Jetson Nano

·1 views

A guide published on DEV Community on July 12, 2026 outlines a rigorous methodology for benchmarking large language models on edge devices such as the Jetson Nano running Ollama. The author warns that a single tokens-per-second figure is insufficient, and that key variables like model quantization, context length, power mode, and cooling must all be documented for results to be reproducible. Recommended test cases include cold starts, repeated requests, long-context runs, and offline behavior, with metrics covering median and p95 time-to-first-token, peak memory, temperature, and throttling. The article also distinguishes between on-device inference and server-side approaches like MonkeyCode, noting the same measurement discipline applies to both. The author concludes that a useful benchmark is one that enables another developer to reproduce the original product decision, not simply one that reports the highest number.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Colibri Runs 744B GLM-5.2 AI Model on 25GB RAM Using Single-File Pure C Engine

A developer named Nokka published Colibri, a single-file inference engine written in roughly 2,400 lines of pure C with zero external dependencies, capable of running GLM-5.2, a 744-billion-parameter Mixture-of-Experts model, on a consumer machine with just 25GB of RAM. The engine works by keeping only the dense portion of the model (about 9.9GB) permanently in RAM while streaming the remaining 21,504 routed experts from NVMe storage on demand as each token is processed. GLM-5.2, released on June 13, 2026 by Z.ai under an MIT license, activates only around 40 billion parameters per token despite its massive total size, which makes the disk-streaming approach viable. The system loads and becomes ready to respond in approximately 32 seconds, generating text at 0.05–0.1 tokens per second on a 12-core WSL2 development machine. Colibri also implements compressed KV-cache persistence to disk, allowing conversation history to survive restarts without reloading the full model context.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Go Middleware Ordering Rules That Prevent Silent Production Failures

A production API ran smoothly for six months before a misplaced panic-recovery middleware caused 2% of requests to silently return 500 errors with no logs or traces. The root cause was a recover middleware placed inside the logging layer, causing it to swallow panics after the logger had written its line but before a request ID was set. Based on 14 years of running Go APIs, the author outlines three non-negotiable middleware ordering rules: Recover must be the outermost layer, Request ID must precede the logger, and Auth or rate-limiting should sit closest to the handler but after observability layers. A small chaining helper is recommended to avoid deeply nested, hard-to-read middleware declarations. Getting the order wrong inverts the entire execution flow, turning trivial ten-line functions into hard-to-debug production hazards.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Adds SigNoz Observability to MERN Stack App Using OpenTelemetry

A developer integrated SigNoz, an open-source observability platform, into RoaVista, a full-stack room booking application built on the MERN stack. The setup involved self-hosting SigNoz via Docker and instrumenting the Node.js/Express backend using OpenTelemetry to push trace data to the SigNoz OTLP endpoint. The standout feature was distributed tracing, which provided visual flame graphs breaking down request latency across MongoDB queries, Express middleware, and authentication steps. Previously, a slow API route offered no clear indication of where the bottleneck occurred, but tracing made it possible to pinpoint delays and errors at each stage. The experience shifted the developer's approach from reactive debugging with console logs to proactive, metrics-driven monitoring.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Python Class Variable Mutation Trap Explained for Experienced Developers

A common Python pitfall involves mutable class variables being shared across all instances, producing unexpected behavior. When a list is defined as a class attribute, all instances reference the same object, so mutations like append affect every instance simultaneously. This differs from immutable types such as integers, where augmented assignment creates a new instance variable rather than modifying the class-level object. The correct fix is to initialize mutable attributes inside __init__, ensuring each instance gets its own copy. Class variables remain appropriate for constants, shared configuration, or carefully managed class-wide counters when modified explicitly through the class name.

0 comments Read more at DEV Community