Dev Shrinks DistilBERT Model 4x for Full Browser Inference Using ONNX and Int8

A developer fine-tuned DistilBERT on the Banking77 dataset to classify 77 banking intents, achieving 92.2% accuracy, but faced a deployment challenge with a 256 MB model too large for a serverless static site. By exporting the model to ONNX format using Hugging Face Optimum, inference speed improved 1.58x over standard PyTorch without any accuracy loss. Applying dynamic int8 quantization then reduced the model size from 256 MB to 64 MB, though it unexpectedly increased latency by 8% on Apple Silicon due to quantize/dequantize overhead. The final model runs entirely client-side in the browser via Transformers.js, downloading once at 64 MB and requiring no backend server or infrastructure. The developer notes that ONNX export and quantization are distinct performance levers, with export delivering speed gains and quantization delivering size reduction rather than further speed improvements.