Developer Discovers RAG System's Full-Text Search Was Broken After First-Ever Benchmark

·2 views

A developer building Langhuan, an open-source knowledge base for AI agents, admitted to never benchmarking its retrieval system despite multiple builds and deployments. Last week, they ran their first formal evaluation using 200 queries sampled from the MIRACL-zh dataset, measuring recall, MRR, and nDCG across vector, full-text, and hybrid search configurations. The benchmarking immediately exposed two critical bugs: a missing foreign key constraint that crashed fresh SQLite instances, and a vector search extension that had never been linked into the production binary. Further results revealed that full-text search (FTS) had zero recall across all 200 queries, while hybrid search scores were identical to vector-only — meaning one of the two retrieval channels had silently never functioned. The developer noted that existing unit and integration tests had missed all these issues because none of them simulated the behavior of a real, first-time user.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

How to Run Effective Ad Hoc Testing Sessions Without Losing Structure

Ad hoc testing lets developers and testers explore software freely without predefined test cases, making it useful for uncovering unexpected behaviour in seemingly stable features. Testers are advised to focus on one feature at a time, complete normal user journeys first, then gradually introduce edge cases using invalid, duplicate, or boundary-value data. Each session should be kept to around 30 to 45 minutes to maintain focus and yield more actionable observations. Despite its informal nature, all meaningful findings should be documented in enough detail for teammates to reproduce the issues. High-risk, repeatable discoveries should be converted into regression test cases to strengthen the overall QA process.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Developer Builds Token Fuel Gauge for Claude Code to Prevent Silent Quality Drops

A developer running an autonomous business setup on Claude Code's MAX plan discovered that token consumption within its 5-hour usage blocks is invisible by default, with no on-screen indicator as the limit approaches. Output quality was found to degrade silently near the 800k output-token threshold, a problem the developer noticed only after reviewing usage data with a tool called ccusage. To address this, the developer built a shell script called token-budget-advisor.sh that aggregates token usage and session costs from two sources and displays a color-coded status — green, yellow, or red — directly in the terminal status line. The system is designed to eliminate the cognitive overhead of manually checking remaining capacity, drawing on the same design principle as a car's fuel gauge. Once integrated into a dashboard update script, the tool automatically surfaces budget state every time a terminal is opened, prompting timely decisions about when to defer heavy workloads to the next usage block.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

Regulators Begin Scrutiny of AI Systems That Target Users at Vulnerable Moments

Digital platforms can now infer users' emotional and financial states in real time and deploy loan offers or persuasive prompts precisely when individuals are most vulnerable, raising serious questions about consent and manipulation. A peer-reviewed paper published in Frontiers in Psychology in April 2026 found that AI-driven design increasingly exploits inferred cognitive and emotional vulnerability to influence user decisions. The UK's Treasury Select Committee warned in January 2026 that British regulators were insufficiently managing AI risks, prompting the FCA's Mills Review to examine how Consumer Duty rules apply to AI personalisation. The EU AI Act's Article 5 prohibitions on AI systems exploiting socio-economic or psychological vulnerabilities became enforceable in February 2025, carrying fines of up to 35 million euros or 7% of global turnover. Meanwhile, the affective computing market — technologies that detect and respond to human emotions — is projected to grow from roughly $100 billion in 2025 to over $300 billion by the early 2030s, far outpacing the legal frameworks meant to govern it.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Diagnose and Fix Nginx 502 and 504 Gateway Errors Effectively

Nginx 502 and 504 gateway errors are common but distinct issues: a 502 Bad Gateway occurs when the upstream process crashes or returns an invalid response, while a 504 Gateway Timeout means nginx stopped waiting for a slow backend. The root cause typically lies in the upstream service — such as Node.js, Gunicorn, or PHP-FPM — rather than in nginx itself. Diagnosing the problem starts with checking nginx error logs for phrases like 'connect() failed' or 'upstream timed out', then verifying whether the upstream process is running and accepting connections. For 504 errors, tuning nginx timeout directives and profiling slow database queries or external API calls is recommended, rather than simply raising timeout values. Configuring upstream failure detection in nginx can also help automatically reroute traffic away from unresponsive servers.

0 comments Read more at DEV Community