Turning Production Agent Sessions into Regression Tests with ZizkaDB

Many AI agent test suites are written by hand and miss the odd phrasings, timeouts, and mid-conversation changes that occur in real use. The article argues that production logs are a better source, since they record what users actually did and the full path an agent took. It recommends defining regression checks that cover both final outcomes and the steps leading to them, with path checks catching cases where an agent reaches a correct-looking answer by skipping a safeguard. Sessions should be selected from failures, escalations, and disputes, with a set of fifty to two hundred well-chosen cases preferred over thousands of random ones. The guide also stresses verifying record integrity with hash-chain checks before use, and notes that keyword-based search is the self-hosted default unless embeddings are enabled.