Bayesian Neural Networks Explained: Uncertainty, Priors, and Posterior Approximation

Bayesian Neural Networks (BNNs) address a key limitation of standard deep neural networks called underspecification, where multiple parameter settings fit training data equally well but generalize differently. By computing a posterior predictive distribution through Bayesian model averaging, BNNs can improve both accuracy and uncertainty estimation. Key challenges include defining suitable priors and efficiently computing the posterior given large models and datasets. Practical approximation methods include Monte Carlo Dropout, which applies random dropout at test time, and the Bayesian Last Layer approach, which applies Bayesian inference only to the final layer's weights. More rigorous posterior approximations are achieved through MCMC methods like Hamiltonian Monte Carlo, though stochastic variants such as SGLD are preferred for scalability with large datasets.