Developer Replaces Gut-Feel Model Testing with Automated 30-Minute Scoring Loop

·2 views

A developer frustrated by unreliable informal testing of open-source AI models built a structured evaluation system to replace instinct-based assessments. The core problem identified was threefold: single prompts are unrepresentative, recent demos create anchoring bias, and fluent-sounding wrong answers feel deceptively correct. The solution uses a JSONL task file containing real-world prompts drawn from actual past work, split into auto-scored code tasks verified by assertion checks and manually rubric-scored tasks judged against criteria written before any output is seen. A lightweight Python runner, requiring no third-party libraries, executes the full suite against any OpenAI-compatible endpoint and produces a written verdict rather than a subjective impression. The author argues the entire evaluation loop fits within a coffee break and eliminates the inconsistency of mood-driven model adoption decisions.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Developer Builds Custom JavaScript Promise Class to Demystify Async Internals

A developer published a tutorial on DEV Community explaining how to implement a custom Promise class in JavaScript from scratch. The project, called MyPromise, is modeled on the core principles of the Promises/A+ specification and replicates key native Promise behaviors. The implementation covers Promise states, resolve and reject functions, callback queuing, chaining, error propagation, and asynchronous execution of .then() handlers. A companion test suite was built to compare the custom implementation's behavior directly against JavaScript's native Promise object. The goal is to help developers move beyond treating Promise as a black box and gain a deeper understanding of how asynchronous JavaScript works internally.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

SQLite FTS5 Outperforms Whoosh, But Pure-Python Search Still Has a Place

An AI agent maintaining the whoosh3 Python search library published benchmarks showing SQLite FTS5 indexes and searches roughly 76–78 times faster than Whoosh on a 5,000-document test corpus. The author openly concedes FTS5's speed advantage, attributing it to being a compiled C extension versus pure Python. However, Whoosh remains relevant in environments where FTS5 is unavailable, such as locked-down enterprise systems or minimal containers that ship SQLite without the FTS5 option enabled. Whoosh also offers a richer query model with built-in support for fuzzy matching, wildcards, field-scoped terms, spelling correction, and result highlighting — features that would require custom implementation on top of FTS5. The article concludes that complex search UIs and constrained deployment environments are the clearest practical reasons to choose Whoosh over FTS5.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer launches MCP server for precise financial math in AI agents

A developer has released PrecisionCalc MCP, a Model Context Protocol server designed to give AI agents reliable, high-precision financial calculation capabilities. The tool addresses a known weakness in large language models, which often produce plausible but incorrect results when performing complex arithmetic such as LTV, NPV, or chained financial metrics. PrecisionCalc uses arbitrary-precision decimals instead of floating-point numbers, and every response includes the formula, inputs, and assumptions for full auditability. The server supports 11 financial tools covering metrics like CAC payback, IRR, loan amortization, and currency conversion using live ECB rates. It is available as a remote HTTP server with no installation required, listed in the official MCP Registry, and offers a free tier of 15 calls per day alongside paid plans.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Builds Custom LLM Eval System to Cut Through Model Launch Hype

A software developer has published a personal method for rigorously vetting new large language models before integrating them into their workflow. The approach was motivated by a costly experience adopting a trending model on launch day, only to find it confidently generated hallucinated CLI flags. Rather than relying on public benchmarks or viral demos, the developer built a small deck of adversarial, repo-specific tasks paired with machine-checkable contracts that define what a passing response looks like. A lightweight, dependency-free Node.js runner executes each task against any OpenAI-compatible endpoint, measuring criteria such as required terms, forbidden terms, word count, and success rate across multiple samples. The system is designed to test whether a model can handle unglamorous, real-world coding tasks — such as YAML configs and database migrations — rather than polished benchmark prompts.

0 comments Read more at DEV Community