Study Quantifies How Much Information Each LLM Parameter Can Store

A joint paper by Meta, DeepMind, Cornell University, and NVIDIA, accepted at ICML 2026, investigates the information storage capacity of large language model parameters. Researchers found that GPT-style Transformer models store approximately 3.6 bits of information per parameter in BF16 format, rising to 3.83 bits under FP32 precision. To isolate rote memorization from generalization, the team trained models ranging from 500,000 to 1.5 billion parameters on random, patternless bit strings. The study identifies a critical phase transition point where training data volume matches model capacity, at which memorization declines and generalization ability rises sharply — coinciding with the well-known double-descent phenomenon. The findings also highlight that while average training samples are unlikely to be leaked from large models, rare or sensitive content such as proprietary code or non-English text remains at high risk of memorization and exposure.