SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

Small local models flip verdicts when prompt layout changes, even with identical content

0
·2 views

A developer tested whether two small local models, gemma-2-2b-it and qwen2.5-coder-1.5b, give stable answers to code-review questions when only the order of code and question changes. Using six backtest snippets (three defective, three correct) and three prompt layouts, the author ran 72 calls on CPU with llama.cpp. Qwen answered YES on all 36 calls, matching an always-YES baseline of 50%, while Gemma scored 52.8%, barely above it. Several verdicts shifted depending on layout, and the added instruction to answer UNSURE produced no abstentions at all. The author concludes that single-query verdicts from such models are unreliable for automation, though the sample is very small and covers only two 4-bit models on one machine.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Study finds AI-assisted developers slower on some tasks, prompting a book on agentic software development

The author argues that a single measurement showing a productivity gain or loss should not be taken as a verdict on AI coding tools, since the full body of results forms a trend that slopes in a particular direction. The piece distinguishes between evidence grades, noting that weakly supported claims can motivate practices but cannot establish them, and that small studies generalize only to similar groups. It draws on the Lindy effect to argue that Parnas's 1972 information-hiding principles remain relevant because they limit the errors an AI agent can make. The author also notes that AGENTS.md, a configuration convention roughly eighteen months old, is promising but not yet proven to be lasting infrastructure. The post introduces a book on the Agentic Development Lifecycle and invites readers to consider how they would ship AI-generated code safely.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Apple Concedes iPhone Duo Crease Cannot Be Eliminated, Says Matte Coating Only Softens It

Three days after the iPhone Duo went on sale, debate over a visible crease on its foldable inner display has intensified. Apple's hardware engineering chief acknowledged on September 19, 2026, that the physical crease cannot be removed and said the company reduces its visibility with a nanotexture surface and matte coating. Commentators note the device's 15,999-yuan price implies a level of display perfection the crease contradicts, even though Apple's spec sheet never promises a crease-free screen. A reviewer's test of 11,500 folds showed a deeper crease, which critics cite, though the piece argues this is only about 5.75% of Apple's rated 200,000-cycle hinge life and does not show hinge failure. The article also argues that crease visibility lacks any shared testing standard, so no vendor can easily quantify it.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Building a Python Workflow That AI Agents Can Discover Using APFlow

A tutorial walks through building a small Python workflow that combines two regional request counts into one report using local sample data and SQLite. It explains how APFlow, an open-source orchestration engine, separates workflow structure from execution order, with independent regional tasks able to run concurrently. Python functions are registered as capabilities with schemas that can be exposed to agents through MCP, A2A, REST, and CLI adapters. The example runs without a model provider or API key and requires Python 3.11 or newer. The article also covers how dependency data flows between tasks, how capability contracts can be inspected, and how to move from a local example toward a production-style setup. APFlow is noted as still classified as Alpha, so the author recommends starting with verifiable outcomes before adding external integrations.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

'Rejection File' Take-Home Grades Candidates on Patches They Refused to Ship

A proposed take-home exercise argues that hiring should weigh the AI-generated proposals a candidate rejects, not just the final code they submit. The Rejection File asks candidates to deliver a working patch alongside a rejections.json record of at least one rejected model proposal, backed by a repro command that fails if the rejection was correct. The format is designed to run in two to three hours on free-tier infrastructure and to separate engineers who verify AI output from those who simply relay it. Scoring weights rejection evidence most heavily at 30 percent, followed by patch correctness at 20 percent, with reproduction quality, scope discipline, budget discipline, and handoff notes making up the rest. The author frames the approach as a response to a recent debate over engineers acting as unverified intermediaries between requesters and AI tools.

0 comments Read more at DEV Community