Dev builds deterministic CI gate to test robotics software on degraded networks
A developer behind Ganglion, a connectivity layer for robotics fleets, has built a required CI gate that tests software behavior across five simulated degraded-network profiles before any code merges to main. The system was created because existing tests ran only on clean loopback connections, missing the real-world failure modes seen on warehouse Wi-Fi, carrier CGNAT, and restrictive firewalls. A key finding during development was that Linux's netem tool cannot pin a random seed, making packet-loss profiles non-deterministic and unsuitable for blocking gates. The final design uses only fully deterministic mechanisms — fixed delays, token-bucket rate caps, and nth-packet drop rules — while randomized chaos testing runs nightly in a non-blocking job. The writeup emphasizes that a gate which fails randomly is worse than no gate at all, as it trains engineers to simply re-run jobs until they pass.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in