Lessons from running a cheap LLM in production: trust the parser, not the prompt

·1 views

A developer running a language-correction app called Mening discovered that a low-cost LLM tier consistently failed to follow output rules despite repeated prompt refinements. Two recurring bugs — a model incorrectly inserting the Chinese particle 了 and generating explanations in the wrong language — persisted through multiple prompt-engineering attempts. The 了 issue was ultimately resolved not by better prompting but by a parser-level filter in Go that discards no-op and invalid edits before they reach the database. The language-drift bug was reduced to zero failures in testing by moving the language instruction to the end of the user turn, closer to the competing target-language text. A lightweight Unicode script-check guard was also added as a safety net, designed to trigger a repair retry only when an entire response is clearly off-script.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Muse Code and Muse Spark 1.2

Article URL: https://research.meta.ai/blog/introducing-muse-code-and-muse-spark-1-2 Comments URL: https://news.ycombinator.com/item?id=49187575 Points: 6 # Comments: 1

0 comments Read more at Hacker News

ProgrammingDEV Community ·

Strong QA Systems Cut Ambiguity, Speed Release Decisions, Study Finds

A DEV Community guide argues that effective QA work is fundamentally about managing ambiguity rather than simply executing tests, with well-designed systems narrowing decisions before discussions begin. The guide evaluates test-case management tools, recommending teams move beyond spreadsheets to solutions that trace tests to requirements, preserve execution history, and unify manual and automated coverage. It also highlights the importance of scenario-based testing for complex UI patterns like infinite scroll, where bugs often emerge during state transitions rather than in static views. For design-token changes, the guide advises organizing visual test reviews by component scope and masking dynamic content to avoid reviewer fatigue from excessive false diffs. Finally, it recommends that test-reporting dashboards be built around release decisions, surfacing critical-path failures, ownership, and flaky tests rather than overwhelming engineers and executives with raw metrics.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer builds Solidity scanner that prioritizes accuracy over finding volume

A developer has built a smart-contract security scanner for Solidity code designed to eliminate false positives, in contrast to existing tools that often generate hundreds of misleading findings. The scanner uses deterministic checks rather than AI guesswork to verify whether access controls, token whitelists, and return-value handling are genuinely absent before raising an alert. Testing against production protocols and the entire OpenZeppelin library produced zero false positives, with 13 previously flagged findings correctly identified as non-issues and suppressed. Key improvements include recognizing intended role-based access patterns, detecting token curation registries that rule out fee-on-transfer risks, and reading surrounding code lines to avoid misclassifying checked call return values. The developer argues that a smaller, verified set of real findings is more valuable to protocol teams than large noisy reports that erode trust in security tooling.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Choosing a Test Automation Framework Is a Product Decision, Not Just a Technical One

As test automation options have expanded beyond Selenium to include Playwright, AI-generated tests, and managed platforms, teams risk optimizing for speed of setup rather than long-term maintainability. Open-source tools may appear nearly free, but the real costs are hidden in engineering time spent managing infrastructure, fixing flaky tests, and maintaining shared code. AI coding assistants can rapidly generate test scripts, yet someone still must understand the logic, locators, and assumptions embedded in that generated code. The right choice between a code-first framework and a managed testing platform ultimately depends on a team's actual size, skill set, and capacity to own ongoing maintenance. As AI also accelerates frontend development and increases UI churn, the sustainability of whichever testing approach a team selects becomes even more critical.

0 comments Read more at DEV Community