Memory API Vendor Builds Open Benchmark for Its Own Industry, Publishes Conflict-of-Interest Rules

A developer at Wontopos, a company that sells memory APIs, has created an open benchmark called Glasshouse to evaluate memory systems, acknowledging the inherent conflict of interest in administering a benchmark they also compete in. To address this, the company has published written rules stating that Wontopos submissions undergo the same third-party approval process as all others, and the firm imposes a 14-day embargo on publishing its own results after any new version. The benchmark dataset is built around a single simulated person's 17-month conversation history, spanning over 103,000 turns and roughly 1.9 million tokens, designed to test memory recall across varying context sizes. A key design choice is a three-tier scoring system for outdated facts, distinguishing between a correct updated answer, an honest admission of uncertainty, and a confidently wrong stale answer — treating the last as a distinct failure mode. The full codebase is released under the Apache 2.0 license, allowing any party to fork and independently operate the benchmark if Wontopos itself becomes a source of bias.