SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

Why Unit-Testing LLMs Fails and How a Structured Eval Harness Solves It

0
·2 views

A developer building an internal LLM-assisted tool found that standard unit tests are ineffective for non-deterministic AI outputs, where results vary with every run. To address this, they adopted a two-part strategy: minimizing the LLM's role to a single drafting step while keeping routing, tracking, and safety checks as plain deterministic code. The LLM sits behind a provider interface, allowing the entire deterministic layer to be tested with ordinary unit tests using no API calls. For the irreducible non-deterministic portion, a dedicated eval harness was built to honestly assess what each testing layer can and cannot catch. The approach shifts the goal from chasing exact outputs to verifying system behavior within clearly defined, testable boundaries.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Why Decision Trees Grow Instead of Learn, and Why That Instability Matters

Unlike logistic regression, decision trees are built by greedily selecting the best local data split at each step rather than through gradient descent optimization. Because finding a globally optimal tree is NP-hard, the algorithm commits to each split without backtracking, making the process fast but inherently approximate. This greedy structure also makes trees unstable: a small change in training data can alter the very first split and cascade into a completely different tree structure. In production settings, this means retraining a model monthly could silently shift the organization's stated explanation for outcomes even if prediction accuracy stays the same. Practitioners are advised to version and diff tree structures between retrains, not just track accuracy metrics.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Mendapi 0.5.5 fixes intentionally disclosed bug where scan -h triggered a scan

Open-source CLI tool Mendapi has released version 0.5.5, patching a known bug that was publicly acknowledged in the previous 0.5.4 release notes rather than silently fixed. The issue caused the 'scan' subcommand to ignore the '-h' flag and run a full scan instead of displaying help text, unlike all other subcommands which correctly normalized '-h' to '--help'. The developers chose to document the bug openly before shipping the fix, citing transparency as a core principle of a tool designed to surface upstream changes. Version 0.5.5 standardizes the '-h' flag behavior across all nine subcommands and adds regression tests to ensure the fix remains effective. The update also refreshes the MCP registry entry with an icon and website URL, while all core commands continue to run locally with no network access and zero npm dependencies.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

VPS.org One-Click Templates Ship Hardcoded Passwords, Leaving Servers Open to Takeover

Security researchers disclosed two high-severity vulnerabilities, CVE-2026-16503 and CVE-2026-16504, in VPS.org one-click deployment templates for Supabase and Zulip, published by JVN on August 3, 2026. The Supabase template exposes PostgreSQL on all network interfaces using the hardcoded superuser password 'postgres', potentially bypassing host firewall rules via Docker NAT and allowing unauthenticated remote access. The Zulip template deploys with a known secret key, a default database password, and HTTPS disabled, enabling attackers to forge session tokens and take over accounts. Both flaws stem from templates failing to generate unique, deployment-specific secrets, meaning any internet-exposed instance is immediately vulnerable after setup. No patches are available and no vendor contact has been established, so operators are advised to manually rotate all credentials, restrict database binding, and enforce HTTPS before or instead of using these templates.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Critical N-central Auth Bypass CVE-2026-18577 Exploited to Hijack MSP-Managed Endpoints

N-able has warned of active exploitation of a critical authentication bypass vulnerability, CVE-2026-18577, affecting N-central versions prior to 2026.3.1.7, disclosed on August 3, 2026. Attackers exploit the flaw to gain unauthenticated admin access to the N-central management console, then abuse the legitimate Take Control feature to move laterally to customer-managed endpoints, prioritizing high-value targets such as Domain Controllers. Once on an endpoint, threat actors deploy a Cloudflare Tunnel registered as a Windows service, establishing persistent outbound command-and-control that survives reboots, account revocations, and even patching of the N-central server. Because the attack leverages trusted RMM tooling and spoofs identities resembling N-able support accounts, malicious sessions can be difficult to distinguish from legitimate activity. N-able urges immediate upgrade to version 2026.3.1.7, restriction of management console access via VPN or IP allowlist, and deployment of service-creation monitoring and application allowlisting to limit post-exploitation persistence.

0 comments Read more at DEV Community