Silent CI Pipeline Hid Test Failures for 11 Weeks Due to Two Shell Scripting Errors

A software team discovered their integration pipeline had been silently swallowing test failures for eleven weeks, only noticing when a colleague pushed a deliberately broken test that still passed. The root cause was a single shell step combining a pipe without 'set -o pipefail' — causing the exit status of 'tee' to mask failures — and a '|| true' appended months earlier to suppress an unrelated cleanup error. Further investigation uncovered eight similarly broken steps across four repositories, using various patterns that discarded command exit statuses. Fixes included enforcing 'set -euo pipefail' at the start of every script step, banning '|| true' outside an allowlisted set of documented exceptions, and replacing piped log capture with sequential redirection. The team also added a canary job to every pipeline that intentionally exits with an error, verifying the pipeline can actually detect and report failures on each build.