SShortSingh.
Back to feed

Memory API Vendor Builds Open Benchmark for Its Own Industry, Publishes Conflict-of-Interest Rules

0
·63 views

A developer at Wontopos, a company that sells memory APIs, has created an open benchmark called Glasshouse to evaluate memory systems, acknowledging the inherent conflict of interest in administering a benchmark they also compete in. To address this, the company has published written rules stating that Wontopos submissions undergo the same third-party approval process as all others, and the firm imposes a 14-day embargo on publishing its own results after any new version. The benchmark dataset is built around a single simulated person's 17-month conversation history, spanning over 103,000 turns and roughly 1.9 million tokens, designed to test memory recall across varying context sizes. A key design choice is a three-tier scoring system for outdated facts, distinguishing between a correct updated answer, an honest admission of uncertainty, and a confidently wrong stale answer — treating the last as a distinct failure mode. The full codebase is released under the Apache 2.0 license, allowing any party to fork and independently operate the benchmark if Wontopos itself becomes a source of bias.

Read the full story at DEV Community

This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)

Log in to join the discussion and vote.

Log in

Related stories

0
ProgrammingHacker News ·

South Korea to Issue Nationwide Emergency Alerts Over College Entrance Exam Cheating

South Korea plans to send nationwide emergency alerts in response to cheating on its college entrance exam. The measure targets fraud linked to the national higher education admissions test. Officials are reportedly concerned about the misuse of technology to gain unfair advantages during the exam. The alerts are intended to warn the public and deter further irregularities. Details beyond the headline are limited in the source material provided.

0
ProgrammingDEV Community ·

AMARACUS 2026 hackathon offers GTA 6 or $100 for best post-quantum crypto app

The AMARACUS Cryptographic Innovation Hackathon 2026 invites developers to build projects that call the company's API, with the top entry winning a copy of Grand Theft Auto VI or $100 in cash, at the winner's choice. Eligible projects span encrypted chat apps, database encryption plugins, enterprise key-management tools, and post-quantum features such as ML-KEM-768 key exchange. Participants may enter solo or in teams of up to three and must be at least 13 years old. Submissions require a public GitHub repository or Drive archive whose code genuinely uses the API, with entry requiring a paid AMARACUS plan starting at $2 per month. The deadline is December 15, 2026, at 23:59 UTC, and full rules are published on the company's website.

0
ProgrammingDEV Community ·

Static analysis finds null-check and memory leak bugs in GmSSL cryptographic library

A reader suggested examining GmSSL, an open-source C cryptographic library from Peking University that implements China's SM2/SM3/SM4/SM9 standards and TLCP and TLS protocols. A PVS-Studio analysis found no especially critical bugs, but it did flag common errors. In one case, a connection pointer is dereferenced before being checked for null, which should be validated at the start of the function. Several test functions also exit early on allocation failure without freeing previously allocated buffers, causing memory leaks. The report notes that these are familiar mistakes that remain problematic despite being routine.

0
ProgrammingDEV Community ·

Why Overbuilt Signup Forms Drive Users Away

Signup flows are often treated as a design issue, but the real problem is usually that the form asks for too much too early. Many forms request extra details like company size, role, and phone number before the user has any reason to stay. Friction such as password rules revealed only after an error, confirm-password fields, and late email verification adds to the drop-off. The article argues that each unnecessary field erodes trust and that data can be collected later, in context. It recommends reducing signup to minimal essentials, such as email alone or email with a password.