Why Loss Functions Are Derived From Probability, Not Chosen by Intuition

Loss functions are the mathematical core of neural network learning, converting raw predictions into training signals via backpropagation. Rather than being arbitrarily selected, they are derived from maximum likelihood estimation by assuming a probability distribution over the target labels. Mean squared error emerges naturally from assuming Gaussian noise on continuous outputs, while cross-entropy loss follows from a categorical (multinoulli) distribution over discrete class labels. One-hot encoding is essential for classification tasks because it avoids implying false ordinal relationships between classes and maps cleanly onto the multinoulli formulation. In the one-hot case, cross-entropy simplifies to the negative log-probability assigned to the correct class, making it both mathematically grounded and computationally efficient.