Why LLM Confidence Scores in Data Extraction Are Often Unreliable

A technical analysis published on DEV Community warns that adding a confidence score field to structured extraction schemas is largely ineffective, as large language models tend to output uniformly high, round numbers like 0.95 regardless of whether their answers are correct. Research by Kadavath et al. (Anthropic, 2022) found that meaningful calibration signals exist within a model's token probability distribution, but not in self-reported text outputs. OpenAI's GPT-4 technical report further showed that post-RLHF alignment training — which makes models more conversational — noticeably degrades confidence calibration. More reliable alternatives include reading log-probabilities directly from the token distribution using the logprobs API parameter, or repeatedly sampling the same extraction and measuring how often the model produces the same answer across runs.