Cheap vs. premium AI code review: why aggregate benchmark scores mislead teams

·1 views

A comparison published by Entelligence on September 14, 2026, pitted GPT-5.6 Luna ($0.0041 per review) against GPT-6 Astra ($0.113 per review) across the same pull requests on several open-source codebases. While Astra outperformed Luna overall — finding 92 verified bugs at 96% precision versus Luna's 69 at 74% — the gap varied sharply by codebase, with Luna performing comparably on everyday application code but falling significantly short on security-sensitive code like Keycloak. On authentication-related bugs specifically, Luna caught only 9 of 24 versus Astra's 19, and missed multi-step logic flaws involving permission overrides and reusable recovery codes. A repeatability test further revealed that neither model consistently reproduced its own findings across multiple runs, exposing instability that single-run benchmark scores conceal. The authors caution that because surrounding code in the benchmark is publicly available and predates both models' training cutoffs, scores likely overstate real-world performance on unseen codebases.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Ollama's 5-Minute Default Caused 214 Model Reloads in a Single Day

A developer running a local AI chat app discovered that Ollama's default 5-minute idle timeout was silently evicting models from VRAM, forcing a costly cold reload from disk on every subsequent request. Over 24 hours and 1,180 requests, the system logged 214 model load events, with cold starts taking 11.4 seconds to first token compared to just 0.9 seconds when the model was already warm. A cron job set to run every 10 minutes was hitting the 5-minute timeout every single time, meaning it had never once used a warm model. The root cause was partly obscured because passing keep_alive in the request body has no effect on Ollama's OpenAI-compatible endpoint — only the native API honors it. Setting the server-side environment variable OLLAMA_KEEP_ALIVE=24h and moving embeddings to a separate CPU-only instance reduced daily model reloads from 214 down to just 9.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Separate Identity, Consent, and Session Management in Patient Portal OAuth

A technical guide published on DEV Community outlines best practices for implementing OAuth-based login in healthcare patient portals using Node.js. The author argues that OAuth handles identity but does not address whether an application should access specific health data categories at a given moment. The piece recommends treating consent and session lifetime as decisions distinct from authentication, with explicit user-facing disclosures showing data category, purpose, and trigger before any access occurs. It also covers refresh token rotation, audit logging of consent state changes, and the risk of revoke actions that only update a UI checkbox without blocking subsequent API calls. Four identity providers — Auth0, Clerk, Keycloak, and Infrai — are compared based on their suitability for portal use cases and their respective trade-offs.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer builds Auricle, a native Windows music player for YouTube Music

A developer frustrated with the high resource usage of mainstream music apps like Spotify and Apple Music on Windows decided to build his own player called Auricle. The app targets YouTube Music's catalogue and is built using a native UI framework rather than Electron or web-based approaches, with goals of low resource consumption and a clean interface. Key challenges beyond basic playback included stream extraction, queue management, and upstream service compatibility. Distributing an early build revealed a separate hurdle: the unsigned installer triggered antivirus flags, even though scans of the installed executables returned clean results. The project is currently Windows-only, with the developer planning to study its real-world performance before considering a Linux port.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How Server-Sent Events Can Replace WebSockets in Next.js Apps

Developers building real-time features in Next.js often default to WebSockets, but Server-Sent Events (SSE) offer a simpler alternative for one-way data streaming. SSE is a browser-native technology that allows servers to push updates to clients over a single HTTP connection. Unlike WebSockets, SSE requires no special protocol and works well for use cases like live feeds, notifications, and progress updates. Next.js supports SSE through its API routes or Route Handlers, making integration relatively straightforward. For applications that only need server-to-client communication, SSE can reduce complexity compared to a full WebSocket implementation.

0 comments Read more at DEV Community