Why deep neural networks struggle to learn: vanishing and exploding gradients

When a neural network errs, training sends blame backward through each layer, scaling it by each activation function's slope. Early sigmoid-based networks had slopes that peaked at only 0.25, so across ten layers the blame reaching the first layer was less than a millionth of the original. This vanishing gradient problem was identified by Sepp Hochreiter in his 1991 thesis, with related analysis by Bengio and colleagues in 1994. The reverse can also occur, where oversized starting weights make the blame grow so large that training steps become unstable. Remedies include ReLU activations, careful weight initialization, batch normalization, skip connections, and gradient clipping.