Developer flags harness bug after local 35B model passes task but fails eval on stop token

·3 views

A developer running agentic evaluations on a local 35-billion-parameter quantized model (ornith-1.0-35b-Q8_0) via llama.cpp found a mismatch between actual model behavior and harness-reported results. In one test involving API key rotation, the model chose correct actions for all service classes, avoided trap tools, and produced an honest summary, yet the harness marked the run as failed because the model ended with prose rather than a recognized stop token. A separate database migration test revealed genuine model defects, including repeated discovery-action loops, a double-applied migration, and proceeding with a staging deployment despite a 503 error on the backup check. The developer is now seeking community input on reliable agent-loop termination strategies for llama.cpp, such as grammar-enforced finish tokens or treating a tool-call-free assistant turn as terminal. The core concern is distinguishing true model failures from instrumentation gaps in the evaluation harness itself.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Climate.gov taken offline, but open data efforts help preserve its content

The U.S. government climate information website Climate.gov was shut down, removing public access to years of climate data and resources. Open data advocates and archivists stepped in to preserve and restore much of the site's content before it was lost permanently. The incident highlights the vulnerability of publicly funded scientific resources when government priorities shift. It also underscores the importance of open data practices, which made recovery efforts possible in this case.

0 comments Read more at Hacker News

ProgrammingHacker News ·

Telegram's t.me domain has been suspended

Article URL: https://www.whois.com/whois/t.me Comments URL: https://news.ycombinator.com/item?id=48897878 Points: 5 # Comments: 1

0 comments Read more at Hacker News

ProgrammingDEV Community ·

Green CI checks can mask untested release pipeline steps, developer warns

A developer discovered that two consecutive green CI runs on a release pipeline were misleading — both passed because the tool, python-semantic-release, correctly detected no release was needed and exited early without executing the critical push step. Since the triggering commits were of types like 'docs:' and 'ci:', which do not warrant a version bump, the pipeline short-circuited before ever testing the protected-branch push or the release token. A separate bug also revealed that build artifacts were not automatically staged for commit, meaning they were silently absent from releases despite all job steps exiting with code 0. The author argues that a zero exit code only confirms a command ran without error, not that it performed the intended work. The key takeaway is that engineers should verify actual pipeline effects — such as whether a tag was created or artifacts were staged — rather than relying solely on a passing status badge.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Engram Launches Developer Learning Tool Using Spaced Repetition and First Principles

Engram is a newly launched educational tool designed specifically for developers, combining first-principles curricula, free-recall verification, and FSRS-based spaced repetition to improve knowledge retention. The platform also features 'explorable artifacts,' which let users interact with complex technical concepts in a hands-on, visual manner. Since its launch, Engram has attracted 775 GitHub stars, signalling early interest from the developer community. It positions itself as a more evidence-based alternative to established platforms like Codecademy and Coursera, which it argues lack scientific rigor in their learning frameworks. The tool aims to address the growing demand for personalized, flexible learning strategies among developers managing multiple skill sets.

0 comments Read more at DEV Community