Colibri Runs Massive AI Models on CPU and SSD by Loading Only Needed Experts

A project called Colibri allows large Mixture of Experts (MoE) language models to run on ordinary computers using only a CPU and SSD, without requiring expensive GPUs or loading the entire model into memory. Instead of keeping all model parameters resident in RAM, Colibri's router selects which expert modules are needed at each inference step, loads them from the SSD, runs the computation, then discards them. This approach reframes storage as an active participant in inference rather than a passive archive of model weights. The key bottleneck identified is the repeated SSD-to-RAM-to-CPU data transfer that occurs thousands of times per generation, making I/O latency the primary constraint rather than raw compute power. The author proposes that predictive prefetching — anticipating which experts will be needed next based on observed token patterns — could significantly reduce that waiting time and further improve the architecture's efficiency.