Large AI writes training data for small model, boosting translation accuracy by 42 points

A developer replaced human-written training data with output generated by a large 27-billion-parameter AI model to train a smaller 4-billion-parameter specialist translation model. The technique, known as distillation, involves having the larger "teacher" model produce example translations that the smaller "student" model then imitates. Applied to Japanese-to-English translation — the direction the small model struggled with most — accuracy jumped from 42% to 84% using around 5,000 training examples derived from the teacher's output. The experiment revealed a key pitfall: providing the teacher model with surrounding context caused the student to learn to fabricate subjects, suggesting training data should mirror production input conditions. The author concludes that performance ceilings in such setups often reflect the student model's capacity limits rather than flaws in the teacher's output quality.