Playwright --update-snapshots can overwrite a good baseline with an unstable frame and still pass

A Playwright issue, #44128, reports that running snapshot updates on a page that never becomes visually stable can replace a correct baseline with a random frame while the test passes. The author reproduced this in tests across Playwright versions 1.40.0 to 1.65.1, using pages with a CSS animation, a JavaScript animation, and a live clock. With the update flag, every run on the moving pages passed and overwrote the baseline, and the next normal run then failed, while runs without the flag failed as expected. Masking the moving element did not prevent the problem for the animation, but masking its container did. The behavior traces to toHaveScreenshot, which writes the last captured frame as the baseline on any error when updating, without checking the error type. The author has published the proof-of-concept code and results.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)
Log in to join the discussion and vote.
Log in