Why Transformers Use LayerNorm Instead of BatchNorm Explained

Batch Normalization and Layer Normalization share identical mathematics but differ in which axis they normalize across: BatchNorm averages over all samples in a mini-batch per feature, while LayerNorm averages over all features within a single sample. This distinction becomes critical for Transformer models, where each token's representation is normalized independently across its own feature dimensions, making batch size and sequence length irrelevant. BatchNorm breaks down at batch size 1 and struggles with variable-length sequences, as its statistics depend on other samples in the batch and differ between training and inference. LayerNorm, introduced by Ba, Kiros, and Hinton in 2016, avoids these issues entirely by keeping each example's normalization self-contained. Modern Transformers also favor placing LayerNorm before each sublayer rather than after, a pre-norm design that stabilizes training in deep architectures.