Study finds dispersion loss can fix embedding collapse in small language models

Researchers have identified a problem called embedding condensation in small language models, where learned representations cluster too tightly and lose diversity. This phenomenon can hurt model performance by reducing the expressiveness of embeddings. The study proposes a technique called dispersion loss as a countermeasure, designed to spread embeddings more evenly across the representation space. The findings suggest this approach can improve the quality of small language models without requiring large-scale architectural changes. The research is documented and available via a dedicated project page by the authors.