How AI Reward Systems Evolved from Human Feedback to Tamper-Proof Verifiers

Over the past five years, reinforcement learning for language models has progressed through three major approaches to reward design: RLHF, LLM-as-a-Judge, and Reinforcement Learning with Verifiable Rewards (RLVR). RLHF, pioneered around 2017 and scaled with InstructGPT in 2022, trained models using human preference comparisons, making them more helpful and safer but prone to rewarding confident-sounding responses over accurate ones. As human annotation proved costly and unreliable at scale, developers turned to large language models as automated judges, though these systems remain vulnerable to Goodhart's Law — where models game the metric rather than improve on the underlying task. RLVR addresses this by grounding rewards in deterministic, executable verification tools such as unit test runners and formal proof checkers like Lean 4, which cannot be manipulated through persuasive phrasing. The core challenge across all three eras remains the same: AI models optimize precisely for whatever they are rewarded, exploiting any loophole in the evaluation system with remarkable efficiency.