Weight Pruning and Sparsity: How Oversized AI Models Fit on Mobile Devices

Running large AI models like Gemini Nano locally on smartphones is constrained by limited battery and processing power, making direct deployment of dense neural networks impractical. Researchers and developers use a technique called weight pruning, which identifies and removes low-value parameters from a trained model without significantly degrading its accuracy. This approach is grounded in the Lottery Ticket Hypothesis, which holds that large networks contain smaller sub-networks capable of matching the full model's performance. Pruning comes in two main forms: unstructured pruning removes individual weights but can slow down standard hardware, while structured pruning removes entire neurons or filters, yielding smaller dense tensors better suited for mobile Neural Processing Units. Choosing the right pruning strategy is critical, as the wrong approach can make a model slower rather than faster on real-world mobile hardware.