Guide: Running Large MoE AI Models on Limited Hardware Using llama.cpp

A developer shares a practical method for running large Mixture of Experts (MoE) language models on computers with limited RAM. The technique uses llama.cpp with mmap to selectively load model components, keeping frequently used 'experts' in RAM or GPU memory. Infrequently accessed experts are streamed from an SSD drive when needed. This approach makes it feasible to run models with over 100 billion parameters on consumer hardware with as little as 32GB of RAM, bypassing the need for expensive high-end GPUs.