How 1950s–60s Engineers Laid the Groundwork for Modern Reinforcement Learning

In 1959, Arthur Samuel tackled the impossibility of storing 500 billion checkers positions by replacing lookup tables with an adjustable scoring function trained through self-play, pioneering Temporal Difference learning — the idea of updating predictions continuously rather than waiting for a game to end. Donald Michie independently demonstrated policy-based learning in 1961 using a physical matchbox system called MENACE, establishing a second foundational branch of reinforcement learning alongside Samuel's value-based approach. Marvin Minsky formally named the credit-assignment problem in 1961, identifying why it is so difficult for machines to determine which past actions deserve credit for a later outcome. Michie's 1968 BOXES system took another leap by balancing a physical pole using only failure as feedback, eliminating the need for a human teacher entirely. These breakthroughs collectively addressed three core obstacles — memory limits, the absence of a world model, and reward hacking — and their solutions underpin virtually every reinforcement learning algorithm in use today.