AirLLM Can Run a 70B Model on 4GB VRAM, But Speed Is the Real Catch

AirLLM is a tool that enables 70B large language models to run on a single 4GB GPU by streaming model layers one at a time from disk rather than loading the entire model into VRAM, keeping memory use to roughly 1.75GB per layer at FP16 precision. Unlike most memory-saving techniques, it does not require quantization or pruning, meaning the full-precision model remains intact. The approach is technically verified and has been independently reproduced, but the performance trade-off is steep — on an RTX 6000 Ada, generation runs at approximately 292 seconds per token, meaning a 100-token response could take over eight hours. Community benchmarks on more common hardware show speeds ranging from 5 to 35 seconds per token for 70B models on NVMe drives, far slower than running a quantized model on a high-end GPU like the RTX 4090, which delivers 8–15 tokens per second. AirLLM is a legitimate engineering solution for memory-constrained environments, but users should weigh the severe speed limitations before treating it as a practical inference option.