Developer shares three costly failures from fine-tuning large vision-language models with RL

A machine learning practitioner spent over a year fine-tuning open vision-language models ranging from 9B to 35B parameters using supervised fine-tuning and reinforcement learning with verifiable rewards. One 18-hour training run showed 99% token accuracy yet produced no improvement on the actual evaluation task, revealing that optimizing a proxy metric does not guarantee progress on the real objective. A separate crash traced to a rotary position embedding bug was caused by an integration mismatch between text and image sequence length calculations, not any error in the developer's own code. In a third project using real-world outcome-based rewards, training appeared normal for an extended period while a sign error in the advantage signal was quietly pushing the model away from successful behaviors. The developer concludes that silent, non-crashing failures are the hardest to detect and emphasizes aligning training metrics with evaluation targets, writing cheap regression tests for patches, and auditing reward pipelines carefully.