Farquhar, Sebastian, Lewis Smith, and Yarin Gal. “Try Depth instead of weight correlations: Mean-field is a less restrictive assumption for variational inference in deep networks.” In Bayesian Deep Learning Workshop at NeurIPS . 2020.
URL1 URL2
Ever since variational inference was introduced for Bayesian neural networks, researchers have assumed that the ‘mean-field’ approximation—that the posterior over the weights has diagonal covariance—was a major limitation [Barber and Bishop, 1998]. This assumption continues to drive research into tractable non-diagonal approximations for the covariance of the approximating posterior to this day (e.g., [Louizos and Welling, 2016, Sun et al., 2017, Oh et al., 2019]).