SShortSingh.
TopTechnologyAI & MLStartupsProgrammingScienceBusinessFinanceCrypto & Web3World
Back to feed
Programming

Spring Boot engineer builds nightly LLM judge to score AI agent response quality

0
·1 views

A Senior Software Engineer at BS23 in Dhaka developed an LLM-as-a-judge evaluation harness for a production e-commerce AI agent built with Spring Boot and Spring AI. The system was created after existing unit tests — which verified tool calls and order state logic — proved unable to assess whether the agent's actual responses to customers were accurate or helpful. The harness runs 40 real conversations from production logs nightly, scoring each against five metrics including answer correctness, factuality, and tool discipline. An LLM model serves as the judge, a pattern documented by Spring AI, with research suggesting such models align with human judgment around 85% — higher than the 81% human-to-human agreement rate. The engineer noted the first evaluation run produced uncomfortable results, which he described as precisely the reason the system was built.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Bitcoin's Four-Year Halving Cycle Persists but ETFs Have Reshaped Its Dynamics

Bitcoin surged to a record above $126,000 in October 2025, roughly 18 months after the April 2024 halving, before falling nearly 48 percent to around $64,963 by August 2026. The cycle's broad shape remained familiar, but its mechanics shifted significantly after U.S. spot ETFs launched, with institutional inflows absorbing over 10,000 BTC per week during much of the 2025 rally — far exceeding the roughly 450 BTC miners produce daily. Bitcoin also broke its 2021 all-time high before the 2024 halving occurred, a first, suggesting ETF-driven demand pulled forward price moves that previously followed the supply cut. When institutional flows reversed, the same regulated channels transmitted redemptions back into spot markets, contributing to the drawdown. Analysts conclude the halving cycle still exists as a supply event and behavioral coordination point, but Bitcoin's price is increasingly correlated with equities and institutional capital flows, making a prolonged supercycle unlikely.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Developer releases duckdiff, a Python tool for comparing large structured data files

A software developer has released duckdiff, an open-source Python package designed to compare large structured data files such as CSV, TSV, and Parquet formats. The tool was built to address limitations of existing solutions like Excel, standard diff utilities, and pandas, which struggle with files exceeding memory limits or require sorted rows. Powered by DuckDB, duckdiff streams data off disk and supports N-way comparison of multiple sources in a single pass, rather than repeated pairwise checks. It also offers features such as fuzzy column mapping, schema intersection, and a key-detection subcommand to identify unique row identifiers. The package is available on PyPI at version 0.1.0, includes 161 tests, and its source code is hosted on GitHub.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Thai researchers release free local-run Thai TTS model with zero-shot voice cloning

ThonburianTTS is an open-source Thai text-to-speech model developed by researchers at Mahidol University's Biomedical and Data Lab in collaboration with Looloo Technology. Built on the F5-TTS flow-matching architecture and fine-tuned on the GigaSpeech2 Thai dataset, it can run entirely on a local machine without requiring any external API. The model supports zero-shot voice cloning using just 5–10 seconds of sample audio, achieving similarity scores of 84–89% according to the original research paper. It was presented at the iSAI-NLP 2025 conference and is freely available on HuggingFace and GitHub under a MIT code licence, though the model weights carry a CC BY-NC-SA 4.0 licence that prohibits commercial use. Researchers note the model struggles with longer text passages, a limitation attributed to its training data favouring short utterances.

0 comments Read more at DEV Community
0
ProgrammingDEV Community ·

Andrew Ng Open-Sources OpenWorker, an AI Desktop Agent That Delivers Completed Work

AI pioneer Andrew Ng has open-sourced OpenWorker, a desktop application that acts as an AI coworker capable of completing tasks across files, terminals, and productivity apps rather than simply responding to chat prompts. The project accumulated over 13,700 GitHub stars and 1,900 forks within three weeks of its July 2026 launch, reflecting strong developer interest in action-oriented AI tools. Unlike conversational AI tools such as ChatGPT or Claude, OpenWorker produces tangible deliverables including drafted documents, updated calendars, and composed Slack replies, pausing for user approval before executing any consequential actions. The app integrates with more than 25 services spanning code platforms like GitHub and Jira, communication tools like Slack and Gmail, and management tools like Notion and Google Calendar, while supporting multiple AI model providers including OpenAI, Anthropic, Google, and local Ollama models. Built on Ng's own aisuite library under an MIT license, OpenWorker runs entirely on the user's machine using their own API keys, keeping data under user control.

0 comments Read more at DEV Community
Spring Boot engineer builds nightly LLM judge to score AI agent response quality · ShortSingh