Metamorphic Testing Flags Faulty LLM Outputs Without Needing Labeled Data

·1 views

A developer has shared a technique for detecting incorrect LLM outputs in document extraction tasks where no ground-truth labels exist. The approach borrows from metamorphic testing, checking whether transformations that should not affect a result — such as reordering line items on a receipt — actually leave the output unchanged. When the extracted value shifts after such a neutral transformation, the output is flagged as potentially wrong. In held-out tests, flagged receipts were 2.7 times more likely to contain genuine errors than unflagged ones, all without any labeled examples. The method is available as an open-source Python package called 'wobbly' on GitHub and PyPI.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

BYD Debuts Humanoid Robot as Industry Experts Warn Factory Deployment Still Years Away

Chinese EV giant BYD officially launched its first humanoid robot at its Di Space technology center in Shenzhen in August 2026, entering the robotics sector as a manufacturer rather than an investor. On the same day, analysts at Automate 2026, North America's largest industrial robotics show, cautioned that bipedal general-purpose humanoid robots are still years away from practical deployment on real production lines. The global robotics market is valued at $38 billion, with $23 billion raised in the sector in the first seven months of 2026 alone, and 15,000 humanoid units already operational worldwide. BYD's key advantage lies in its existing supply chains for precision motors, battery packs, and embedded electronics built through years of vertical integration in EV manufacturing. Industry observers note that if BYD applies its cost-leadership strategy to humanoid robotics, it could introduce significant pricing pressure on established players like Figure AI and Boston Dynamics.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Not All Self-Doubt Is Impostor Syndrome — Misdiagnosis Can Make Things Worse

A software developer and conference speaker argues that not every instance of self-doubt qualifies as impostor syndrome, warning that misidentifying the cause can lead to harmful advice. The author draws on personal experiences, including irrational fears of being removed from a speaker lineup and feeling out of place among peers, to illustrate genuine impostor syndrome. However, she contrasts these with areas where her underperformance is real and objective, such as running, where encouragement to 'push harder' would be counterproductive. Her core argument is that distinguishing between unfounded self-doubt and legitimate skill gaps requires an honest, analytical assessment rather than a blanket motivational response. Applying the wrong fix to the wrong problem, she contends, risks burnout and wasted effort instead of meaningful growth.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Safely Mix Singleton and Scoped Services with DbContext in .NET

In .NET dependency injection, combining services with different lifetimes — such as a singleton EventListener and a scoped DbContext — can cause serious runtime issues. Injecting a scoped DbContext directly into a singleton means a single database context is reused for the entire application lifetime, leading to stale data and concurrency errors. The recommended fix is to use IServiceScopeFactory inside the singleton, creating a fresh scope and a new DbContext instance for each event processed. Alternatively, the EventProcessor can be made scoped and resolved per event through the singleton listener using the same scope factory pattern. Following these best practices prevents resource misuse and ensures database operations remain isolated and thread-safe.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How Engineers Can Protect Payment Integrations Against Vendor Acquisitions

Fintech mergers and acquisitions pose a real engineering risk, as seen when Visa's $5.3 billion acquisition of Plaid shifted the product toward enterprise clients and made onboarding harder for smaller teams. Developers who rely on third-party payment APIs are statistically likely to face disruptions from acquisitions, pricing changes, or forced migrations at some point. Experts recommend building a thin abstraction layer so payment logic is isolated behind an internal interface, making provider swaps far less disruptive. Teams should also store their own canonical payment state in an internal database rather than relying solely on the vendor's systems as the source of truth. Early warning signs such as free-tier cutbacks and slower support response times often precede major platform shifts, and budgeting migration time into roadmaps in advance can reduce a forced transition from months of work to just days.

0 comments Read more at DEV Community