Spectral Normalisation Bounds Lipschitz Constants to Stabilise Deep Networks
Deep networks can be accurate yet fragile, with outputs shifting sharply under tiny input changes and training gradients that can explode or vanish. Spectral normalisation addresses this by limiting how much each layer can stretch its input, which bounds the Lipschitz constant of the whole network. For linear layers this constant equals the largest singular value of the weight matrix, so rescaling each weight matrix by that value makes every layer 1-Lipschitz. Because computing the singular value exactly is costly, practitioners use power iteration, which needs only one extra matrix-vector product per layer per training step. The method adds no new architecture and needs no extra hand-tuned loss term.
This is an AI-generated summary. ShortSingh links to the original source for the complete article.
Discussion (0)
Log in to join the discussion and vote.
Log in