SShortSingh.
Back to feed

Engrava Posts Reproducible LongMemEval-S Benchmark Scores for Versions 0.5.0 and 0.6.0

0
·1 views

Memory system Engrava published verifiable benchmark results on the 500-question LongMemEval-S test, with version 0.6.0 scoring 81.6% micro in August 2026 and version 0.5.0 scoring 82.4% micro in July 2026. Both runs used the same canonical scorer, standard GPT-4o reader and judge, and a top-k of 20 retrieved turns, with no changes to anything outside the memory layer itself. Unlike many published benchmarks, Engrava released full reproduction artifacts for both runs, including the older, slightly higher-scoring result that predates the current release. The memory pipeline contains no generative language model — ingestion and retrieval rely on deterministic hybrid search over a typed graph, meaning no LLM tokens are consumed on every read or write operation. The four-question gap between the two versions prompted the team to examine run artifacts directly rather than attributing the difference to either regression or statistical noise.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingDEV Community ·

Google Adds Personal Intelligence Layer to Gemini in Chrome for Contextual Assistance

Google is expanding Gemini in Chrome with an opt-in feature called Personal Intelligence, which connects data from selected Google apps such as Gmail, Calendar, Docs, Maps, and YouTube to deliver more contextually relevant assistance. The upgrade aims to reduce manual context-switching by allowing the browser assistant to draw on a user's existing work across Google services, subject to explicit permissions. The rollout is currently focused on desktop Chrome, targeting US users and English-language configurations first, with regional expansion including Latin America planned. Personal Intelligence is in beta and is expected to support more connected apps over time, meaning current capabilities represent an early-stage implementation. Gemini in Chrome is not yet available on Android, and access to certain features may depend on a user's Google account subscription tier.

0
ProgrammingDEV Community ·

Is AI's 'Double-Check Before Use' Warning a Genuine Safeguard or a Liability Shift?

A widely discussed opinion piece on DEV Community questions whether the standard AI disclaimer — advising users to verify outputs before use — places an unfair burden on those least equipped to do so. The author argues that for familiar topics, verification is reasonable, but for unfamiliar ones, users may lack the knowledge needed to detect confident errors in the first place. The piece raises concerns that requiring line-by-line auditing of AI responses erodes the time-saving benefit these tools are meant to provide. The author suggests a more practical standard would be to calibrate verification effort to the level of risk involved, while making AI uncertainty and limitations more transparent to users. Ultimately, the article questions whether the blanket disclaimer is an honest safety measure or a mechanism for AI companies to transfer responsibility onto users.

0
ProgrammingDEV Community ·

How a Solo SaaS Developer Built Idempotent Email Delivery for a Cron-Based Product

A developer behind Upwork Scout, a small SaaS that matches freelancers to job listings via email, has detailed the engineering challenges of building a reliable scheduled email system. The core problem is that unlike database writes, sent emails cannot be undone or retried safely, making duplicate sends a real user-facing failure. To prevent duplicate emails, the developer uses deterministic Firestore document IDs derived from user and job data, turning idempotency checks into simple lookups rather than complex queries. A distributed lock prevents multiple concurrent scan runs from duplicating work and inflating scraping costs. The developer also intentionally chose at-least-once delivery for daily digest emails, reasoning that silently losing a digest is worse than occasionally sending a rare duplicate.

0
ProgrammingDEV Community ·

Junior AppSec Engineer in India Struggles With Billion-Line Codebase and No Mentorship

A junior Application Security Engineer in India has publicly sought help after being assigned to review a codebase of up to two billion lines within a one-month deadline, despite having minimal training and no senior mentor. The engineer, a recent graduate, was given only basic tools such as grep and a free-tier version of Codex, which proved inadequate for detecting complex security flaws at scale. Their only significant find — a critical two-factor authentication flaw involving an OTP stored in a browser cookie — was discovered by chance rather than through any systematic process. With limited proficiency in the relevant programming languages and growing employer pressure over perceived low productivity, the engineer's situation highlights a broader industry problem of organizations assigning high-stakes security work to junior staff without adequate support. Experts warn that such onboarding failures increase the risk of undetected vulnerabilities, data breaches, and long-term reputational damage for the organizations involved.

Engrava Posts Reproducible LongMemEval-S Benchmark Scores for Versions 0.5.0 and 0.6.0 · ShortSingh