Dev Team Learns Hard Lessons After Building Multi-Repo Project with Git Submodules

A development team designed their project around five separate Git repositories and submodules from day one, prioritizing CI isolation, access control, and clean history. In practice, the setup demanded strict two-step push discipline — once inside the submodule and once in the parent repo — and forgetting either step caused silent failures that only surfaced during fresh CI checkouts. On one occasion, three submodules simultaneously had unpushed local commits, while a fourth required an interactive rebase to recover. The core 'engine' repository also caused CI conflicts by behaving differently as a standalone repo versus as a nested submodule, breaking path-resolution logic and missing cross-repo dependencies. These recurring failures pushed the team to move beyond one-off patches and establish a shared convention for managing the dual-context build environment.