Popular AI memory benchmark excludes 22% of questions and bars 'I don't know' answers

·1 views

A widely cited AI memory benchmark, LOCOMO, has a significant methodological gap: its published evaluation harness skips all 446 adversarial questions in Category 5, which are designed to test whether a system can correctly abstain from answering. The harness also instructs the AI judge to 'COMMIT AND ANSWER' and never say 'not specified,' effectively eliminating any measurement of calibrated abstention. The LLM judge used for scoring has been found by an independent audit to accept roughly 63% of deliberately wrong answers, and LOCOMO's own answer key contains errors that cap the realistic maximum honest score at around 93.6. This means a recently reported score of 92.5 — touted as near-perfect memory performance — sits at the ceiling of a flawed benchmark, not at the ceiling of actual capability. The core issue is not fraud but misrepresentation: the metric measures answer accuracy on answerable questions only, yet is routinely interpreted as an overall measure of memory system quality.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

RE-call vs Mem0: No-LLM memory system trades accuracy for cost and privacy

Developer benchmarks comparing RE-call, an open-source memory system that never calls an LLM, against Mem0 show Mem0 outperforms RE-call on temporal reasoning tasks, scoring 0.567 versus 0.408 on the BEAM benchmark. The gap exists because Mem0 distills memories into clean summaries at ingest using an LLM, while RE-call stores raw conversation turns, causing date-related queries to retrieve wrong instances of the same fact. However, RE-call's no-LLM design means zero ingest cost compared to Mem0's $7.29 and 2.6 million tokens spent processing the same workload. RE-call also completed ingestion in 67 seconds versus Mem0's 288 seconds, and keeps all data within the user's own database. The author concludes that neither system is universally better — the right choice depends on whether a user prioritises accuracy on time-sensitive queries or prefers lower cost, faster ingest, and local data control.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

OpenAI Urges Clearer Reporting of AI Benchmark Conditions and Setup Variables

OpenAI has published new guidance calling on researchers, evaluators, and buyers to treat AI benchmark scores as measurements tied to specific testing conditions rather than fixed capability limits. The company argues that factors such as the evaluation harness, compute budget, available tools, and memory management can significantly alter observed model performance, particularly on complex multi-step tasks. OpenAI distinguishes between different types of evaluation claims — including strong elicitation, controlled comparison, and safeguard robustness — noting that each requires different evidence and cannot be used interchangeably. The guidance recommends that benchmark reports disclose the full setup behind a score so that comparisons between models remain meaningful and reliable. OpenAI cited its own GPT-5.5 cyber-range evaluations as an example, where performance visibly improved when the harness used context compaction across extended tasks.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

How to Actually Verify If an AI Crawler Is Legitimate, Not Just Labeled

AI crawlers like GPTBot identify themselves via user-agent strings, but these can be easily faked by anyone in seconds, making them unreliable as sole proof of identity. Verification requires independently confirming a crawler's claimed identity using methods the operator controls, such as reverse DNS lookup, published IP ranges, or cryptographic HTTP message signatures. Reverse DNS remains the most robust method, while IP-range checks are the most widely supported, used by 20 of 41 documented crawlers including OpenAI's GPTBot. Cryptographic signing via HTTP Message Signatures is emerging as the most secure approach, with infrastructure support from Cloudflare, Akamai, and AWS, though operator-side adoption remains limited. The stakes have grown significantly by 2026, as verified crawler identity now governs real privileges like search representation, pay-per-crawl access, and bot-challenge exemptions.

0 comments Read more at DEV Community

ProgrammingDEV Community ·

AI memory systems cannot reliably detect what they don't know, new benchmark shows

A developer investigating AI memory agent reliability found that the ability to abstain from unanswerable questions is not a single capability but varies with how far removed a question is from its supporting evidence. A new benchmark was built with questions generated at controlled 'excision distances,' revealing that discrimination between answerable and unanswerable queries improves sharply as more context is deleted, from near-chance accuracy at close range to near-perfect at full topic removal. This finding explains why two public benchmarks, LOCOMO and BEAM, appeared to contradict each other: both were measuring different points on the same hidden curve and reporting them as a flat capability score. Further analysis showed that for roughly two-thirds of test questions, deleting the gold evidence caused no change in retrieval scores, meaning the correct answer was never the top-ranked result in the first place. The author concludes that abstention failures in AI memory systems are structural rather than tunable, and that benchmark comparisons are misleading without specifying the difficulty distance of unanswerable questions.

0 comments Read more at DEV Community