Researcher Proposes Using Trained Model Weights as a New AI Training Data Modality

Damian Borth, an AI and machine learning professor at the University of St. Gallen, has introduced a concept called Weight Space Learning (WSL), which treats the weights of trained neural networks as a standalone data modality for further training. Speaking on the TWIML AI Podcast, Borth argued that each trained model encodes thousands of GPU-hours of optimization experience, and that this information can be extracted and reused without accessing original training data or model outputs. His team developed an autoencoder-based framework that compresses model weights into a low-dimensional latent space, enabling tasks such as predicting a network's accuracy without any test data. In one experiment, WSL-generated weight initialization reduced training compute from 12,000 GPU-hours to roughly 350 GPU-hours for a remote sensing model. Borth distinguishes WSL from knowledge distillation, noting that distillation learns from a teacher model's behavioral outputs, whereas WSL learns directly from the geometric structure of parameter spaces across large collections of open-source models.