Performance budget in CI switched to median timings after flaky failures
A CI performance budget kept failing on changes that could not have slowed the app, and rerunning the job often made it pass. The author decided to keep the budget but make each measurement distinguish app slowness from machine noise. The app is an issue tracker with a local IndexedDB cache and WebSocket sync, and the tests time how long a cached workspace takes to open and how quickly a rename appears in a second browser. Each test uses the median of five runs, runs in an isolated CI job with one worker and no retries, and reads timing marks recorded inside the app to avoid counting test-driver round trips. Budgets were set at roughly three times the measured open time and a looser multiple for propagation, and the author tightens them as measured results show more headroom.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.

Discussion (0)
Log in to join the discussion and vote.
Log in