7.5B Gemma model outscores 24B Devstral in independent 56-task coding benchmark

·1 views

A developer ran a custom 56-task coding benchmark across 16 local AI model configurations on a single 16 GB GPU card, completing 36 full runs under identical conditions. Google's Gemma-4-e4b, a 7.5B-parameter model in a 4.97 GiB file, scored 42 out of 56, outperforming Devstral-small-2-24b, a 24B model, which scored 40. All tasks were graded using hidden deterministic tests run by compilers and test runners, with no model judging another model's output. The results did not align with any public leaderboard and in at least one case directly inverted a public ranking. The benchmark also revealed significant score variance between identical consecutive runs for some models, suggesting output consistency is itself a meaningful performance dimension.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

create-next-mui CLI gets smarter dependency sorting, Oxlint support, and auto-install

The open-source CLI tool create-next-mui, which scaffolds Next.js projects with Material UI pre-configured, has received a series of incremental but meaningful updates. The developer behind the tool overhauled the dependency merging logic so that generated package.json files now maintain consistently sorted dependencies for easier readability. A dual-linter setup was introduced, pairing the fast Oxlint for real-time diagnostics with ESLint, which continues to handle import cleanup and code organization rules. The CLI also now prompts users after project scaffolding to optionally run npm install automatically, removing a manual step from the setup process. The updates reflect a philosophy of reducing developer friction through small, practical refinements rather than large feature additions.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How One Developer Built a Lean Homelab Server Using Alpine Linux

A developer documented their process of building a minimal homelab server on a Zimablade machine using Alpine Linux, deliberately avoiding bloated NAS distributions and unsolicited web interfaces. The setup uses MergerFS to pool multiple drives into a single mount point and SnapRAID for parity-based data protection without traditional RAID lock-in. The installation involves configuring Alpine with LVM-based disk partitioning, OpenSSH, and XFS-formatted drives identified via UUID entries in fstab. Additional tools like bash, util-linux, and FUSE are added to replace Alpine defaults and support the storage stack. The author notes this is a personal configuration walkthrough rather than a universal tutorial, and individual setups may vary.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Release Notes Tell Users What Changed, Not Whether a Deploy Is Safe

A widely read developer essay argues that release notes and deploy decisions serve fundamentally different purposes, yet teams routinely conflate the two. Release notes are outward-facing summaries of what already happened, written for users and support staff, while deploy decisions require inward-facing risk assessments covering what could go wrong. The author notes that AI tools have made generating release notes dramatically faster, but that speed has not improved the quality of deployment decision-making. Critical context — such as which code paths were touched, how carefully a PR was reviewed, and what CI checks showed — lives in pull requests and review states, not in the diff summary. The essay concludes that teams should continue sending release notes to users, but invest in separate tooling for engineers who need to evaluate the safety of a release before pushing it to production.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

New Benchmark Tests AI Coding Agents on Real 2026 Pull Requests Across 5 Languages

A team built 'octobench', a coding benchmark using 25 real tasks drawn from pull requests merged in 2026 across Python, PHP, Rust, C++, and JavaScript open-source projects, to avoid the contamination and arbitrary grading problems found in existing benchmarks. Each task reconstructs the repository state just before a fix was merged, with agents working like contractors and being evaluated by the project's own held-out tests. Octomind's agent using an open model scored 24 out of 25, outperforming Claude Code with Opus, while the same underlying model in a different setup solved only 19 tasks at twice the cost. The benchmark was designed to stay fresh by targeting recently merged fixes, several of which were harvested within days of being committed. The results suggest that agent architecture and tooling matter as much as the underlying model when tackling real-world software engineering tasks.

0 comments Read more at DEV Community