Instruction tuning boosts AI confidence but not accuracy, study finds

A study by Proskurina et al. found that instruction fine-tuning consistently increases the expressed confidence of language models without improving their predictive accuracy on question-answering benchmarks. The researchers compared three base models against their instruction-tuned versions, measuring answer entropy and verbalized certainty to isolate the effects of the tuning process. They also observed that instruction tuning reduces diversity in model-generated reasoning, suggesting a homogenization of thought patterns alongside the unwarranted confidence boost. Likelihood-based calibration metrics remained poor after tuning, indicating the training objective does not penalize miscalibrated probability estimates. The authors argue that benchmarks should incorporate calibration measures alongside accuracy scores, and that additional alignment techniques may be needed to produce reliably calibrated models.