Confirm Two Independent CLI Runs Match Before Freezing a Transcript Baseline
A DEV Community article argues that a repository's behavior should only be frozen as a golden baseline once two independent runs yield identical transcripts, since disagreement means there is no reliable baseline. It recommends testing at the process boundary, capturing exit codes, stdout, stderr, and a SHA-256 digest of the output file tree rather than function return values. The author describes using fixture directories, isolated temp copies, and a short, explicitly documented set of regex normalizers for nondeterministic fields like UUIDs and timestamps. To expose ordering nondeterminism, the process runs twice under different PYTHONHASHSEED values, and any divergence is fixed by sorting the data rather than pinning the seed. The article includes a roughly sixty-line dependency-free Python harness for recording and checking these golden files.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)
Log in to join the discussion and vote.
Log in