How Batch Normalization Became a Cornerstone Technique in Deep Learning

Batch Normalization (BatchNorm) is a mathematical technique used in deep neural networks that normalizes intermediate activations during training by subtracting the mini-batch mean and dividing by its standard deviation. It was originally motivated by the concept of Internal Covariate Shift, where changing network weights cause activation distributions in deeper layers to shift continuously, slowing down training. To preserve the network's representational flexibility, BatchNorm also applies learned scale and shift parameters, allowing the model to adapt the normalization as needed. During inference, BatchNorm relies on running estimates of mean and variance accumulated during training rather than batch statistics, ensuring stable and consistent predictions. A key practical benefit is that BatchNorm enables training with larger learning rates by smoothing the optimization landscape and controlling the scale of signals flowing through the network.