Same AI Model, Different Harness: Performance Gaps Can Exceed 15 Percentage Points

·1 views

A developer testing Moonshot's Kimi K3 model found noticeably different experiences depending on whether it ran inside Kimi Code CLI or Claude Code, despite the underlying model being identical. Moonshot's own benchmark data supports this, showing a score difference between the two harnesses — and the company officially discloses harness conditions in its model card. Research cited in the article indicates that swapping the harness while keeping the model fixed can shift SWE-bench coding performance by up to 15 percentage points, dwarfing the 2–4 point gains typically highlighted in model release announcements. The author argues that an AI agent rests on two pillars — the model and the harness — and that the harness, which manages context, tools, memory, and permissions, is often the weaker and more consequential one. As model capabilities advance rapidly, the harness infrastructure is struggling to keep pace, making harness quality an underappreciated factor in real-world AI performance.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

A Decade in Software Engineering: Key Lessons From Banking, Gov and Tech Projects

A full-stack software engineer with ten years of experience reflects on lessons learned while building systems for a bank, Georgia's Ministry of Healthcare, major online marketplaces, and an international reinsurance platform. The engineer argues that mastering fundamentals like HTTP, database indexing, and problem modelling outlasts any specific framework, as tools are routinely replaced across the industry. Most serious performance issues encountered across sectors traced back to data problems — poor indexing, inefficient queries, or poorly designed schemas — rather than frontend or application logic. Breadth as a full-stack developer proved valuable not for technical superiority in any one area, but for the ability to trace problems across an entire system and resolve disputes between siloed teams. The engineer also highlights that reading and understanding unfamiliar codebases is a critical but underrated skill rarely taught formally, yet essential in real-world engineering roles.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Claude Fable 5 vs GPT-4o: Which Model Writes Better TypeScript Code?

A developer and AI systems architect ran head-to-head TypeScript code generation tests on Claude Fable 5 and GPT-4o using identical prompts and a three-part rubric covering compilation, strict mode compliance, and error handling. Claude Fable 5 performed better with generics, utility types, and async error boundaries, consistently producing output that passed strict TypeScript mode with minimal edits. GPT-4o handled simple and intermediate type inference reliably but struggled with advanced patterns like conditional types and heavily generic helpers. Both models occasionally missed exhaustiveness checks on discriminated unions, a gap the tester flagged as requiring manual human review. The key takeaway is that each model has distinct failure modes, and choosing the right one depends on the complexity and patterns specific to the project being built.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Optimajet Rebuilds Workflow Engine Docs to Let AI Coding Agents Integrate .NET Library

Optimajet has overhauled the documentation for its embeddable .NET workflow library, Workflow Engine, with AI coding agents specifically in mind. The new docs at docs.workflowengine.io publish their full text as a single file so agents can read everything in one request, reducing the guesswork that leads to hallucinated API calls. A dedicated endpoint also allows agents to obtain a trial license key autonomously, removing the human-in-the-loop bottleneck that previously stalled automated evaluations. The documentation spans 57 pages across four sections, and a separate API reference at api.workflowengine.io covers all 20 packages in the library. Workflow Engine is commercial software shipping since 2014, supports .NET Framework 4.6.2 and above, and works with major databases including PostgreSQL, SQL Server, and MongoDB; its latest stable release is version 22.0.0, dated July 16, 2026.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer finds distributed systems lessons hidden inside a simple Rock Paper Scissors game

A developer built an online Rock Paper Scissors game in 2023 as a weekend project, which has since grown to include an iOS app, a Turkish sister site, and a matchmaking queue over three years. Building the game revealed classic distributed systems challenges, including the need to withhold a player's move from their opponent until both picks are locked, preventing any timing-based cheating. A race condition bug emerged at launch when two users joined the same room simultaneously, both seeing an empty seat — a check-then-set flaw fixed using atomic Redis operations. The developer chose to keep all live match state in Redis with a time-to-live expiry, reserving PostgreSQL only for completed match records, which kept the database lean and allowed horizontal scaling. A silent failure in the iOS Live Activity feature, caused by a swallowed error in a token refresh path, went unreported for weeks, highlighting the danger of features that fail quietly without user-facing feedback.

0 comments Read more at DEV Community