Priors in Bayesian Deep Learning: A Review

Introduction

Bayesian models have gained a stable popularity in data analysis and machine learning . Especially in recent years, the interest in combining these models with deep learning has surgedAs attested, for instance, by the growing interest in the Bayesian Deep Learning workshop at NeurIPS.. The main idea of Bayesian modeling is to infer a posterior distribution over the parameters ${\bm{\theta}}$ of the model given some observed data ${\mathcal{D}}$ using Bayes’ theorem as

where $p({\mathcal{D}}|{\bm{\theta}})$ is the likelihood, $p({\mathcal{D}})$ is the marginal likelihood (or evidence), and $p({\bm{\theta}})$ is the prior. The prior can often be parameterized by hyperparameters $\bm{\psi}$, in which case we will write it as $p({\bm{\theta}};\bm{\psi})$ if we want to highlight this dependence. This posterior can then be used to model new unseen data ${\mathcal{D}}^{*}$ using the posterior predictive

The integral in Eq. 2 is also called the Bayesian model average, because it averages the predictions of all plausible models weighted by their posterior probability. This is in contrast to standard maximum-likelihood learning, where only one parameter ${\bm{\theta}}^{*}$ is used for the predictions as

While much previous work has focused on the properties of the posterior predictive , the approximation of the integrals in Eq. 1 and Eq. 2 , or the use of the marginal likelihood for Bayesian model selection , in this thesis we want to shed some light on the often-neglected term in Eq. 1: the prior $p({\bm{\theta}})$.

In orthodox Bayesianism, the prior should be chosen in a way such that it accurately reflects our beliefs about the parameters ${\bm{\theta}}$ before seeing any data . This has been described as being the most crucial part of Bayesian model building, but also the hardest one, since it is often not trivial to map the subjective beliefs of the practitioner unambiguously onto tractable probability distributions . However, in practice, choosing the prior is often rather seen as a nuisance, and there have been many attempts to try to avoid having to choose a meaningful prior, for instance, through objective priors , reference priors , empirical Bayes , or combinations of these . One problem with these methods is that in Bayesian deep learning, they are often not tractable due to the high dimensionality of the inference problem, since they either require computing the Fisher information matrix, solving a series of increasingly high-dimensional integrals, or splitting the model parameters into “parameters of interest” and “nuisance parameters” . Especially in Bayesian deep learning, it is therefore common practice to choose a (seemingly) “uninformative” prior, such as a standard Gaussian [c.f., 19].

This trend is troubling, because choosing a bad prior can have detrimental consequences for the whole inference endeavor. While the choice of uninformative (or weakly informative) priors is often being motivated by invocation of the asymptotic consistency guarantees of the Bernstein-von-Mises theorem , this theorem does not in fact hold in many applications, since its regularity conditions are not satisfied . Moreover, in the non-asymptotic regime of our practical inferences, especially in high-dimensional settings, the prior can have a strong influence on the posterior, often forcing the probability mass onto arbitrary subspaces of the parameter space . This means, for instance, that the seemingly innocuous standard Gaussian prior is not uninformative at all , but forces the posterior mass onto a thin spherical subspace, which in most cases does not reflect any useful prior knowledge and can severely bias the inference .

Worse yet, prior misspecification can undermine the very properties that compel us to use Bayesian inference in the first place. For instance, marginal likelihoods can become meaningless under prior misspecification, leading us to choose suboptimal models when using Bayesian model selection . Moreover, de Finetti’s famous Dutch book argument can be extended to cases where we can be convinced to take wagers that lose money in expectation when using bad priors, which even holds for the aforementioned objective (Jeffreys) priors . In a similar vein, Savage’s theorem , which promises us optimal decisions under Bayesian decision theory, breaks down under prior misspecification . Finally, it can even be shown that PAC-Bayesian inference can exceed the Bayesian one in terms of generalization performance when the prior is misspecified .

On a more optimistic note, the no-free-lunch theorem states that no learning algorithm is universally superior, or in other words, that different learning algorithms outperform each other on different datasets. Applied to Bayesian learning, this means that there is also no universally preferred prior, but that each task is potentially endowed with its own optimal prior. Finding (or at least approximating) this optimal prior then offers the potential for significantly improving the performance of the inference or even enabling successful inference in cases where it otherwise would not have been possible.

All these observations should at least motivate us to think a bit more carefully about our priors than is often done in practice. But do we really have reason to believe that the commonly used priors in Bayesian deep learning are misspecified? One recent piece of evidence is the fact that in Bayesian linear models, it can be shown that prior misspecification leads to the necessity to temper the posterior for optimal performance (i.e., use a posterior $p_{T}({\bm{\theta}}\,|\,{\mathcal{D}})\propto p({\bm{\theta}}\,|\,{\mathcal{D}})^{1/T}$ for some $T<1$) . And indeed, this need for posterior tempering has also been observed empirically in modern Bayesian deep learning models [e.g., 35, 36, 37, 19].

Based on all these insights, it is thus high time that we critically reflect upon our choices of priors in Bayesian deep learning models. Luckily for us, there are many alternative priors that we could choose over the standard uninformative ones. This survey shall attempt to provide an overview of them. We will review existing prior designs for (deep) Gaussian processes in Section 2, for variational autoencoders in Section 3, and for Bayesian neural networks in Section 4. We will then finish by giving some brief outline of methods for learning priors from data in Section 5.

Priors in (Deep) Gaussian Processes

Gaussian processes (GPs) have a long history in Bayesian machine learning and enjoy many useful properties . They are nonparametric models, which means that we are not actually specifying a prior over parameters $p({\bm{\theta}})$, but instead a prior over functions $p(f)$. This prior can also have hyperparameters $\bm{\psi}$, which parameterize a mean function $m_{\bm{\psi}}$ and a kernel function $k_{\bm{\psi}}$ as

This prior is called a Gaussian process because it has the property that when evaluating the function at any finite set of points ${\bm{x}}$, the function values ${\bm{f}}:=f({\bm{x}})$ are distributed as $p({\bm{f}})={\mathcal{N}}({\bm{m}}_{\bm{x}},{\bm{K}}_{{\bm{x}}{\bm{x}}})$, where ${\bm{m}}_{\bm{x}}=m_{\bm{\psi}}({\bm{x}})$ is the vector of mean function outputs, the $(i,j)$’th element of the kernel matrix ${\bm{K}}_{{\bm{x}}{\bm{x}}}$ is given by $k_{\bm{\psi}}(x_{i},x_{j})$, and the $d$-dimensional multivariate Gaussian ${\mathcal{N}}({\bm{f}};{\bm{\mu}},{\bm{\Sigma}})$ is

The Gaussian process can also be seen as an infinite-dimensional version of this multivariate Gaussian distribution, following the Kolmogorov extension theorem .

This model is often combined with a Gaussian observation likelihood $p({\bm{y}}\,|\,{\bm{f}})={\mathcal{N}}({\bm{f}},\sigma^{2}{\bm{I}})$, since it then allows for a closed-form posterior inference on unseen data points $({\bm{x}}^{*},{\bm{y}}^{*})$ as

While these models are not deep per se, there are many ways in which they connect to Bayesian deep learning, which merits their appearance in this thesis. In the following, we are going to present how GP priors can be parameterized by deep neural networks (Section 2.1), how GPs can be stacked to build deeper models (Section 2.2), and how deep neural networks can themselves turn into GPs or be approximated by GPs (Section 2.3).

As mentioned above, the GP prior is determined by the parameterized functions $m_{\bm{\psi}}$ and $k_{\bm{\psi}}$. It will come as no surprise that these functions can be chosen to be deep neural networks (DNNs). In the case of deep kernels , however, one has to proceed with care, since most neural network functions will not actually yield proper kernels. One option to get a kernel out of a neural network is to use the last layer of the network as a feature space and define the kernel to be the inner product in this space, that is, $k_{\bm{\psi}}(x,x^{\prime})=\langle\phi(x;\bm{\psi}),\phi(x^{\prime};\bm{\psi})\rangle$, where $\phi(\cdot\,;\bm{\psi})$ is the neural network with parameters $\bm{\psi}$ and $\langle\cdot,\cdot\rangle$ is the inner product. This actually leads to a Bayesian linear regression in the feature space of the network and is also sometimes called a Bayesian last layer (BLL) model .

Another option to develop deep kernels is to start with a base kernel $k_{\text{base}}(\cdot,\cdot)$, for instance a radial basis function (RBF) kernel $k_{\text{RBF}}(x,x^{\prime})=\exp(-\lambda(x-x^{\prime})^{2})$ with lengthscale $\lambda$. This kernel can then be applied in the DNN feature space, yielding the kernel

If one chooses the linear kernel $k_{\text{lin}}(x,x^{\prime})=\langle x,x^{\prime}\rangle$ as the base kernel, this reduces to the BLL model above. However, when choosing a kernel like the RBF, this model still yields an infinite-dimensional reproducing kernel Hilbert space (RKHS) and thus offers a full GP that does not reduce to a finite Bayesian linear regression. These approaches can not only lead to very expressive models, but have also been shown to improve properties such as adversarial robustness .

When using deep mean functions instead of (or in combination with) deep kernels, less precautions have to be taken, since virtually any function is a valid GP mean function. Thus, the neural network can simply be used as the mean function itself, as $m_{\bm{\psi}}(x)=\phi(x;\bm{\psi})$. Moreover, deep mean functions in GPs have been related to other popular learning paradigms, such as functional PCA . However, the main problem with these, as with the deep kernels above, is the question how to choose them. Since DNNs are notoriously hard to interpret, choosing their parameters truly a priori, that is, before seeing any data, seems like an impossible task. These approaches are thus usually used in combination with some additional learning algorithms, which set their parameters based on some objective function. We will provide further details on these algorithms in Section 5. As an additional sidenote, we would also like to mention a specific type of GP kernel, namely the convolutional kernel , which is not itself parameterized by a neural networks, but inspired by convolutional neural networks (CNNs) in its construction, leading to improved performance on images.

2 Deep Gaussian processes

While GPs can be combined with deep neural networks, as we saw in the previous section, they can also be used to construct deep models in their own right. This is done by adding $k$ additional latent functions $\{f_{1},\dots,f_{k}\}$ with function outputs $\{{\bm{f}}_{1},\dots,{\bm{f}}_{k}\}$ and latent variables $\{{\bm{z}}_{1},\dots,{\bm{z}}_{k-1}\}$, where each function uses the previous latent variable as its inputs, that is, ${\bm{f}}_{i+1}=f_{i+1}({\bm{z}}_{i})$ and ${\bm{f}}_{1}=f_{1}({\bm{x}})$. In the simplest case, all these latent GPs still have Gaussian latent likelihoods $p({\bm{z}}_{i}\,|\,{\bm{f}}_{i})={\mathcal{N}}({\bm{f}}_{i},\sigma^{2}_{i}{\bm{I}})$ and a Gaussian output likelihood $p({\bm{y}}\,|\,{\bm{f}}_{k})={\mathcal{N}}({\bm{f}}_{k},\sigma^{2}_{k}{\bm{I}})$. If each of these functions is endowed with a GP prior $p(f_{i})=\mathcal{GP}(m_{\bm{\psi}_{i}}(\cdot),k_{\bm{\psi}_{i}}(\cdot,\cdot))$, this model is called a deep Gaussian process . Similarly to deep neural networks, these models can represent increasingly complex distributions with increasing depth, but unlike neural networks, they still offer a fully Bayesian treatment. Crucially, in contrast to standard GPs, deep GPs can model a larger class of output distributions , which includes distributions with non-Gaussian marginals . For increased flexibility, these models can also be coupled with warping functions between the GP layers .

While these models seem to be strictly superior and preferable to standard GPs, their additional flexibility comes at a price: the posterior inference is not tractable in closed form anymore. This means that the posterior has to be estimated using approximate inference techniques, such as variational inference , expectation propagation , or amortized inference . A very popular approximate inference technique for GPs is based on so-called inducing points, which are chosen to be a subset of the training points or generally of the training domain . This technique can also be extended to inference in deep GPs or replaced by variational random features . Moreover, it has recently been shown that for certain choices of kernels, neural networks can be trained as point estimates for deep GP posteriors .

In contrast to the inference techniques, the choice of priors for deep GPs has generally been understudied. While a deep GP as a whole can model a rather complex prior over functions, the priors for the single layers in terms of $m_{\bm{\psi}_{i}}$ and $k_{\bm{\psi}_{i}}$ are often chosen to be quite simple, for instance, RBF kernels with different lengthscales . An exception to this are combinations of deep GPs with the convolutional GP kernels mentioned above, which yield models that are similar in spirit to deep CNNs . Moreover, recent software packages for deep GP inference have made it easier to experiment with different priors . One can thus be carefully optimistic that research into better deep GP priors will blossom in the years to come.

3 Gaussian process limits of neural networks

Another way to connect GPs to DNNs is via neural network limits. It has been known for some time now that the function-space prior $p(f)$ induced by a Bayesian neural network (BNN) with a single hidden layer and any independent finite-variance parameter prior $p({\bm{\theta}})$ converges in the limit of infinite width to a GP, due to the central limit theorem . The limiting GP prior is given by

with the prior weight and bias variances $\sigma^{2}_{w_{1}},\sigma^{2}_{b_{1}}$ in the first layer, $\sigma^{2}_{w_{2}},\sigma^{2}_{b_{2}}$ in the second layer, and nonlinear activation function $\varphi(\cdot)$. Note that here it is usually assumed that the weight variances are set as $\sigma^{2}_{w_{i}}\propto 1/n_{i}$, where $n_{i}$ is the number of units in the $i$’th layer. The kernel $k_{\text{NN}}(\cdot,\cdot)$ is then called the neural network GP (NNGP) kernel. This result has recently been extended to BNNs with ReLU activations and deep BNNs , where the lower layer GP kernel takes the same form as above and the kernel for the higher layers assumes the recursive form

While these results only hold for independent finite-variance priors, they can be extended to dependent priors, where they yield GPs that are marginalized over a hyperprior , and to infinite-variance priors, where they lead to $\alpha$-stable processes . Excitingly, it has been shown that this convergence of the BNN prior to a stochastic process also implies the convergence of the posterior under mild regularity assumptions . While these results have typically been derived manually, the recent theoretical framework of tensor programs allows to rederive them in a unified way, including for recurrent architectures and batch-normalization . Moreover, it allows to derive limits for networks where only a subset of the layers converge to infinite width, which recovers the models’ ability to learn latent features .

Not only infinitely wide BNNs can lead to GP limits, but this is also true for infinitely wide standard DNNs. Crucially however, in this case, the GP arises not as a function-space prior at initialization, but as a model of training under gradient descent . Specifically, neural networks under gradient descent training can be shown to follow the kernel gradient of their functional loss with respect to the so-called neural tangent kernel (NTK) which is

where ${\bm{J}}_{\bm{\theta}}({\bm{x}})$ is the Jacobian of the neural network with respect to the parameters ${\bm{\theta}}$ evaluated at input ${\bm{x}}$. In the limit of infinite width, this kernel becomes stable over training and can be recursively computed as

In the case of finite width, this kernel will not model the training behavior exactly, but there exist approximate corrections . Interestingly, this same kernel can also be derived from approximate inference in the neural network, leading to an implicit linearization . This linearization can also be made explicit and can then be used to improve the performance of BNN predictives and for fast domain adaptation in multi-task learning . Moreover, when using the NTK in a kernel machine, such as a support vector machine, it can outperform the original neural network it was derived from, at least in the small-data regime . Similarly to the aforementioned NNGP kernels, the NTKs for different architectures can also be rederived using the framework of tensor programs and there exist practical Python packages for the efficient computation of NNGP kernels and NTKs . Finally, it should be noted that this linearization of neural networks has also been linked to the scaling of the parameters and described as lazy training, which has been argued to be inferior to standard neural network training .

Priors in Variational Autoencoders

Moving on from GPs, another popular class of Bayesian deep learning model is the variational autoencoder (VAE) . VAEs are Bayesian latent variable models which assume a generative process in which the observations ${\bm{x}}$ are generated from unobserved latent variables ${\bm{z}}$ through a likelihood $p({\bm{x}}\,|\,{\bm{z}})$. In the case of VAEs, this likelihood is parameterized by a neural network which is trained on the observed data. Since the nonlinearity of this neural network renders exact inference on the posterior $p({\bm{z}}\,|\,{\bm{x}})$ intractable, it is approximated with a variational approximation $q({\bm{z}}\,|\,{\bm{x}})$, which is typically also parameterized by a neural network. The whole model is then trained by optimizing the evidence lower bound (ELBO)

The likelihood and approximate posterior are usually chosen to be Gaussian. While the prior is typically also chosen to be standard Gaussian, that is $p({\bm{z}})={\mathcal{N}}({\bm{0}},{\bm{I}})$, there are many other possible choices, which we will explore in the following. Particularly, we will look at some proper probability distributions that can directly replace the standard Gaussian (Section 3.1), at some structural priors that also require changes to the architecture (Section 3.2), and finally at a particularly interesting VAE model with idiosyncratic architecture and prior, namely the neural process (Section 3.3).

We will use the term distributional priors to refer to distributions $p({\bm{z}})$ that can be plugged into the standard VAE architecture described above without changing the rest of the model. However, note that often it can be beneficial to also change the functional form of the variational posterior $q({\bm{z}}\,|\,{\bm{x}})$ to fit the prior better. The first type of prior that has been shown to yield some benefits compared to the standard Gaussian one is a spherical prior, namely a von-Mises-Fisher (vMF) prior , that is

for a $d$-dimensional latent space, where ${\bm{\mu}}$ is the mean, $\kappa$ is a concentration parameter, and ${\mathcal{I}}_{k}$ is the modified Bessel function of the first kind of order $k$. This distribution can be seen as a version of the multivariate Gaussian distribution that is supported on the hypersphere. However, its main disadvantage is that the modified Bessel function cannot generally be computed in closed form and thus has to be approximated numerically.

These hyperspherical priors have been shown to improve VAE performance on benchmark data over standard Gaussian ones, however mostly in low-dimensional latent spaces (up to $d\approx 20$) . This could be due to the Gaussian annulus theorem [108, Thm. 2.9], which states that the measure of a multivariate Gaussian in high dimensions concentrates on a hypersphere anyway. For higher-dimensional latent spaces, it has thus been proposed to replace the vMF prior with a product of lower-dimensional vMF distributions .

To overcome the numerical issues of the modified Bessel functions, the power-spherical distribution has been suggested as a replacement for the vMF . Its $d$-dimensional density is given by

where ${\bm{\mu}}$ is again the mean, $\kappa$ the concentration parameter, and $\Gamma(\cdot)$ is the Gamma function. Since the Gamma function is easier to evaluate than the modified Bessel function, this density allows for closed-form evaluation and reparameterizable sampling. Empirically, it yields the same performance in VAEs as the vMF prior, while being numerically more stable .

Another type of priors are mixture priors , typically mixtures of Gaussian of the form

with $K$ mixture components where $\pi_{i}$ are the mixture weights that are often set to $\pi_{i}=1/K$ in the prior. These priors have been motivated by the idea that the data might consist of clusters, which should also be disjoint in the latent space , and they have been shown to outperform many other clustering methods on challenging datasets . However, similarly to many other clustering methods, one challenge is to choose the number of clusters $K$ a priori. This can also be optimized automatically, for instance by specifying a stick-breaking or Dirichlet process hyperprior , albeit at the cost of more involved inference.

Finally, most of these priors assume independence between data points. If we have prior knowledge about potential similarity between data points and we can encode it into a kernel function, a Gaussian process can be a powerful prior for a VAE . The prior is usually defined as

where ${\bm{Z}}=({\bm{z}}_{1},\dots,{\bm{z}}_{n})$ is the matrix of latent variables and ${\bm{K}}_{{\bm{z}}{\bm{z}}}$ is again the kernel matrix with $(i,j)$’th element $k({\bm{z}}_{i},{\bm{z}}_{j})$ for some suitable kernel function $k(\cdot,\cdot)$. These models have been shown to excel at conditional generation , time series modeling , missing data imputation , and disentanglement . It should be noted that this comes at additional computational cost compared to standard VAEs, since it requires the ${\mathcal{O}}(n^{3})$ inversion of the kernel matrix (see Eq. 6). However, this operation can be made more scalable, either through the use of inducing point methods (c.f., Section 2.2) or through factorized kernels . Moreover, depending on the prior knowledge of the generative process, these models can also be extended to use additive GP priors or tensor-valued ones .

2 Structural VAE priors

In contrast to the distributional priors discussed above, we will use the term structural priors to refer to priors that do not only change the actual prior distribution $p({\bm{z}})$ in the VAE model, but also the model architecture itself. Some of these structural priors are extensions of the distributional priors mentioned above. For instance, the aforementioned Gaussian mixture priors can be extended with a mixture-of-experts decoder, that is, a factorized generative likelihood, where each factor only depends on one of the latent mixture components . Another example are the Gaussian process priors, which are defined over the whole latent dataset ${\bm{Z}}$ and thus benefit from a modified encoder (i.e., inference network), which encodes the complete dataset ${\bm{X}}$ jointly .

In addition to these distributional priors with modified architectures, there are also structural priors which could not be realized with the standard VAE architecture. One example are hierarchical priors , such as

We see here that instead of having a single latent variable ${\bm{z}}$, these models feature $K$ different latent variables $\{{\bm{z}}_{i},\dots,{\bm{z}}_{K}\}$, which depend on each other hierarchically. These models require additional generative networks to parameterize the conditional probabilities in Eq. 17 or Eq. 18, which then enable them to better model data with intrinsically hierarchical features and to reach state-of-the-art performance in image generation with VAEs .

Another type of structural priors are discrete latent priors, such as the VQ-VAE prior

where $E$ is a finite dictionary of prototypes and ${\bm{z}}_{e}$ is a continuous latent variable that is then discretized to ${\bm{z}}_{q}$. Crucially, the prior is not placed over the continuous ${\bm{z}}_{e}$, but over the discrete ${\bm{z}}_{q}$, namely as a uniform prior over the dictionary $E$. These discrete latent variables can then be saved very cheaply and thus lead to much stronger compression than standard VAEs . When combining these models with the hierarchical latent variables described above, they can also reach competitive image generation performance . Moreover, these discrete latent variables can be extended to include neighborhood structures such as self-organizing maps , leading to more interpretable latent representations that can also be used for clustering . Finally, similar topological priors can also be induced on continuous latent variables using ideas from persistent homology .

3 Neural processes

To conclude this section, we will look at a structural VAE prior that has spawned a lot of interest in recent years and thus deserves its own subsection: the neural process (NP). This model has been independently proposed under the names of partialVAE and (conditional) neural process , but the latter nomenclature has caught on in the literature. The main novelty of this VAE architecture is that it not only models the distribution of one type of observed variable ${\bm{x}}$, but of two variables $({\bm{x}},{\bm{y}})$, which can be split into a context and target set $({\bm{x}},{\bm{y}})=({\bm{x}}_{c},{\bm{y}}_{c})\cap({\bm{x}}_{t},{\bm{y}}_{t})$. These sets are conditionally independent given ${\bm{z}}$, that is, $p({\bm{x}},{\bm{y}}\,|\,{\bm{z}})=p({\bm{x}}_{c},{\bm{y}}_{c}\,|\,{\bm{z}})\,p({\bm{x}}_{t},{\bm{y}}_{t}\,|\,{\bm{z}})$. This then allows to infer an unobserved ${\bm{y}}_{t}$ based on the other variables using a variational approximation $q({\bm{z}}\,|\,{\bm{x}}_{c},{\bm{y}}_{c})$ and the conditional likelihood $p({\bm{y}}_{t}\,|\,{\bm{z}},{\bm{x}}_{t})$. Thus, the model can be used for missing data imputation and regression tasks. Note that, since the likelihood is typically conditioned on ${\bm{x}}_{t}$ instead of just on ${\bm{z}}$, this model can be framed as a conditional VAE .

One remarkable feature of this model is the used prior, which is namely

This means that instead of using an unconditional prior $p({\bm{z}})$ for the full posterior $p({\bm{z}}\,|\,{\bm{x}},{\bm{y}})$, a part of the data (the context set) is used to condition the prior, which is in turn approximated by the variational posterior with reduced conditioning set. While this is atypical for classical Bayesian inference and generally frowned upon by orthodox Bayesians, it bears resemblence to the data-dependent oracle priors that can be used in PAC-Bayesian bounds and have been shown to make those bounds tighter .

The NP model has been heavily inspired by stochastic processes (hence the name) and has been shown to constitute a stochastic process itself under some assumptions . Moreover, when the conditional likelihood $p({\bm{y}}_{t}\,|\,{\bm{z}},{\bm{x}}_{t})$ is chosen to be an affine transformation, the model is actually equivalent to a Gaussian process with neural network kernel .

Since their inception, NP models have been extended in expressivity in different ways, both in terms of their inference and their generative model. On the inference side, there are attentive NPs , which endow the encoder with self-attention (and thus make it Turing complete ), and convolutional (conditional) NPs , which add translation equivariance to the model. On the generative side, there are functional NPs , which introduce dependence between the predictions by learning a relational graph structure over the latents ${\bm{z}}$, and Gaussian NPs , which achieve a similar property by replacing the generative likelihood with a Gaussian process, the mean and kernel of which are inferred based on the latents.

Priors in Bayesian Neural Networks

Bayesian neural networks are neural network models in which the parameters are determined through Bayesian inference (see Eq. 1) and predictions are made using the posterior predictive (see Eq. 2). These models have gained increasing popularity in recent years , mostly due to their uncertainty calibration properties . While many different priors have been proposed for these models [e.g., 151, and references therein], it has often been argued that standard Gaussian priors over the parameters are sufficient and that the modeler’s inductive biases should be represented through the choice of architecture instead . This view had been supported by preliminary studies on small networks and simple problems that did not find conclusive evidence for the misspecification of Gaussian priors .

However, in recent work, the adequacy of Gaussian priors has been put into question, particularly by the discovery that Gaussian priors can cause a cold posterior effect which is not caused by some other priors . Following the general considerations regarding prior misspecification (see above), it thus seems advisable to also consider alternative priors for BNNs. In the following, we will review priors defined in the weight space (Section 4.1) and in the function-space (Section 4.2) and will also show how to extend these ideas to (Bayesian) ensembles of neural networks (Section 4.3).

As mentioned before, the most widely used priors for BNNs are isotropic Gaussian (see Eq. 5) priors [e.g., 78, 154, 155, 156, 99, 157]. When these priors are used in combination with ReLU nonlinearities, it has been shown that the distributions of activations within the network grow more heavy-tailed with increasing depth . However, it has also been shown that these networks converge to GPs in the infinite limit (see Section 2.3), which has famously led Dave MacKay to inquire whether we have “thrown the baby out with the bath water” , since we usually choose BNN models for their increased expressivity over GPs (where we mean with expressivity the ability to approximate different distributions over function space in their respective predictives). Moreover, Gaussian priors have recently been shown to cause a cold posterior effect in BNNs. That is, the tempered posterior $p_{T}({\bm{\theta}}\,|\,{\mathcal{D}})\propto p({\bm{\theta}}\,|\,{\mathcal{D}})^{1/T}$ with $T\ll 1$ performs better than the true Bayesian posterior, suggesting prior misspecification .

A simple extension of standard Gaussian priors are matrix-valued Gaussians, which allow for additional correlations between weights . Their density is given by

Another way to improve the expressiveness of Gaussian priors is to combine them with hierarchical hyperpriors , which has already been proposed in early work on BNNs as

where $p({\bm{\Sigma}})$ is a hyperprior over the covariance. An example of such a hyperprior is the inverse Wishart distribution [e.g., 47], which is in $d$ dimensions given by

where $\nu$ are the degrees of freedom and ${\bm{K}}$ is the mean of $p({\bm{\Sigma}})$. When marginalizing the prior in Eq. 22 over the hyperprior in Eq. 23, it turns out that one gets a $d$-dimensional multivariate Student-t distribution with $\nu$ degrees of freedom , namely

Such distributions have been shown to model the predictive variance more flexibly in stochastic processes and BNNs . Moreover, in BNNs, it has been shown that priors like these, which are heavy-tailed (also including Laplace priors ) and allow for weight correlations, can reduce the cold posterior effect , suggesting that they are less misspecified than isotropic Gaussians. Finally, when using Student-t priors, it has been shown that one can obtain expressive BNN posteriors even when forcing the posterior mean of the weights to be zero , which highlights the flexibility of these distributions.

Another Gaussian scale mixture prior is the horseshoe prior , which is

where $b_{0}$ and $b_{1}$ are scale parameters and ${\mathcal{C}}^{+}$ is the half-Cauchy distribution

In BNNs, the horseshoe prior can encourage sparsity and enable interpretable feature selection . It can also be used to aid compression of the neural network weights . Moreover, in application areas such as genomics, where prior knowledge about the signal-to-noise ratio is available, this knowledge can be encoded in such sparsity-inducing hierarchical priors . Interestingly, the popular neural network regularization technique dropout can also be understood as an approximation to these types of priors and they can also be used to explicitly model uncertainty over the network architecture, using doubly stochastic inference techniques . Finally, Indian buffet process priors can also be used to similarly encourage sparsity and select smaller numbers of weights .

Another interesting prior is the radial-directional prior, which disentangles the direction of the weight vector from its length . It is given by

In the specific case of convolutional neural networks for vision tasks, early research has already noted that the weight distributions of the convolutional filters follow the statistics of natural images . Based on this insight, weight priors have been suggested for Bayesian CNNs that either use correlated Gaussians to encourage weights that are similar for neighboring pixels or Gabor function priors for the whole filters to encourage, for instance, edge detection .

In terms of even more expressive priors, it has been proposed to model the parameters in terms of the units of the neural network instead of the weights themselves . The weight $\theta_{ij}$ between units $i$ and $j$ would then have the prior

where the function $g$ can be either parameterized by a neural network or by a Gaussian process . A similarly implicit model, with even more flexibility, has been proposed by Atanov et al. and is simply given by

In both of these priors, the main challenge is to choose the function $g$. Since this is hard to do manually, the function is usually (meta-)learned (see Section 5.3). Finally, recent work on software packages for BNN inference (e.g., using gradient-guided MCMC inference ) has made it easier to try different weight-space priors, thus fostering research to discover better prior distributions .

2 Function-space priors

As we saw, there are many different weight-space priors that one can choose for Bayesian neural networks. When using certain non-standard architectures, such as radial basis function networks , desired functional properties (e.g., lengthscale or amplitude) can be directly encoded into those priors . However, when using standard BNNs, choosing the right parameter prior can be challenging, since we often have better intuitions about the functions we would expect rather than the parameters themselves. The trouble is then that the mapping from parameters to functions in neural networks is highly non-trivial due to their many weight-space symmetries and complex function-space geometries . This has led to an alternative approach to prior specification in BNNs, namely to specify the priors directly in function space, such that

where $p(f)$ is the function-space prior, $\phi(\cdot\,;{\bm{\theta}})$ is the function implemented by a neural network with parameters ${\bm{\theta}}$ and $\delta(\cdot)$ is the Dirac delta measure (in function space).

As we have seen before (c.f., Section 2), Gaussian processes offer an excellent model class to encode functional prior knowledge through the choice of kernel and mean functions, that is, $p(f)=\mathcal{GP}(m(\cdot),k(\cdot,\cdot))$. It is thus a natural idea to use GP priors as function-space priors for BNNs. If one applies this idea in the most straightforward way, one can just optimize a posterior that now depends on the KL divergence between the BNN posterior and the GP prior. However, since this KL is defined in an infinite-dimensional space, it requires approximations, such as Stein kernel gradient estimators . Alternatively, one can first optimize a weight-space distribution on a BNN to minimize the KL divergence with the desired GP prior (e.g., using Monte Carlo estimates) and then use this optimized weight prior as the BNN prior during inference .

While both of these approaches seem reasonable at first sight, it has been discovered that GP and BNN function-space distributions do not actually have the same support and that the true KL divergence is thus infinite (or undefined) . It has therefore recently been proposed to use the Wasserstein distance instead, although this also requires approximations . If one wants to forego the need for a well-defined divergence, one can also use a hypernetwork as an implicit distribution of BNN weights and then train the network to match the GP samples on a certain set of function outputs . Finally, it has recently been discovered that the ridgelet transform can be used to approximate GP function-space distributions with BNN weight-space distributions . As a sidenote, it should be noted that the reverse can actually be achieved more easily, namely fitting a GP to the outputs of a BNN , which can also be of interest in certain applications.

If one does not want to use a GP prior in function space, one can still encode useful functional prior knowledge into BNN priors. For instance, through the study of the infinite-width limits of BNNs (see Section 2.3), one finds that the activation function of the network has a strong influence on the functions being implemented and one can, for instance, modulate the smoothness or periodicity of the BNN output by choosing different activation functions . Moreover, one can directly define priors over the BNN outputs, which can encode strong prior assumptions about the values that the functions are allowed to take in certain parts of the input space , that is,

This prior can encode the belief that the epistemic uncertainty should grow away from the in-distribution data and can thus also lead to more GP-like behavior in BNN posteriors. Finally, if we have the prior belief that the BNN functions should not be much more complex than the ones of a different function class (e.g., shallower or even linear models), we can use this other class as a functional reference prior and thus regularize the predictive complexity of the model .

3 Bayesian neural network ensembles

Deep neural network ensembles, or deep ensembles, are a frequentist method similar to the bootstrap that has been used to gain uncertainty estimates in neural networks . However, it has been recently argued that these ensembles actually approximate the BNN posterior predictive , that is

where ${\bm{\theta}}_{i}$ are the weights of $K$ independently trained ensemble members of the same architecture. For linear models, ensembles can actually be made to sample exactly from the posterior , while in deeper models they can at least provide lower bounds on the marginal likelihood of the true posterior . These models can also be extended to ensembles with different hyperparameters , thus also approximating a hierarchical hyperposterior. Moreover, they can be made more parameter-efficient by sharing certain parameters between ensemble members , which can then also be used for approximate BNN inference . While these models have performed well in many practical tasks , they can still severely overfit in some scenarios , leading to ill-calibrated uncertainties . However, it has been shown recently that each ensemble member can be combined with a random function that is sampled from a function-space prior , and that this can indeed yield uncertainties that are conservative with respect to the Bayesian ones . More specifically, the uncertainties of such ensembles are with high probability at least as large as the ones from a Gaussian process with the corresponding NNGP kernel (see Section 2.3). These results can also be extended to the NTK .

Another way of making these deep ensembles more Bayesian and incorporating priors are particle-based approximate inference methods, such as Stein variational gradient descent (SVGD) . In SVGD, the ensemble members (or particles) are updated according to

where $\eta$ is a step-size and $k(\cdot,\cdot)$ is a kernel function in weight space. With the right step-size schedule, this update rule converges asymptotically to the true posterior and even enjoys some non-asymptotic guarantees . Moreover, note that it only requires sample-based access to the gradient of the log posterior (and thus also the log prior), which allows it to be used with different weight-space priors and even function-space priors, such as GPs . Finally, standard deep ensembles can also be directly extended with a kernelized repulsive force, similar to the one in SVGD, which then also leads to asymptotic convergence to the true Bayesian posterior .

(Meta-)Learning Priors

So far, we have explored different types of distributions and methods to encode our prior knowledge into Bayesian deep learning models. But what if we do not have any useful prior knowledge to encode? While orthodox Bayesianism would prescribe an uninformative prior in such a case , there are alternative ways to elicit priors, namely by learning them from data. If we go the traditional route of Bayesian model selection using the marginal likelihood (the term $p({\mathcal{D}})$ in Eq. 1), we can choose a functional form $p({\bm{\theta}};\bm{\psi})$ for the prior and optimize its hyperparameters $\bm{\psi}$ with respect to this quantity. This is called empirical Bayes or type-II maximum likelihood (ML-II) estimation . While there are reasons to be worried about overfitting in such a setting, there are also arguments that the marginal likelihood automatically trades off the goodness of fit with the model complexity and thus leads to model parsimony in the spirit of Occam’s razor principle .

In the case where we have previously solved tasks that are related to the task at hand (so-called meta-tasks), we can alternatively also rely on the framework of learning to learn or meta-learning . If we apply this idea to learning priors for Bayesian models in a hierarchical Bayesian way, we arrive at Bayesian meta-learning . This can then also be extended to modern gradient-based methods .

While these ML-II optimization and Bayesian meta-learning ideas can in principle be used to learn hyperparameters for most of the priors discussed above, we will briefly review some successful examples of their application below. Following the general structure from above, we will explore learning priors for Gaussian processes (Section 5.1), variational autoencoders (Section 5.2), and Bayesian neural networks (Section 5.3).

Following the idea of ML-II optimization, we can use the marginal likelihood to select hyperparameters for the mean and kernel functions of GPs. Conveniently, the marginal likelihood for GPs (with Gaussian observation likelihood) is available in closed form as

with $N$ being the number of data points, ${\bm{K}}_{{\bm{x}}{\bm{x}}}$ the kernel matrix on the data points, and $\sigma^{2}$ the noise of the observation likelihood. We can see that the first term measures the goodness of fit, while the second term (the log determinant of the kernel matrix) measures the complexity of the model and thus incorporates the Occam’s razor principle .

While this quantity can be optimized to select the hyperparameters of simple kernels, such as the lengthscale of an RBF kernel, it can also be used for more expressive ones. For instance, one can define a spectral mixture kernel in the Fourier domain and then optimize the basis functions’ coefficients using the marginal likelihood, which can recover a range of different kernel functions . To make the kernels even more expressive, we can also allow for addition and multiplication of different kernels , which can ultimately lead to an automatic statistician , that is, a model that can choose its own problem-dependent kernel combination based on the data and some kernel grammar. While this model naïvely scales rather unfavorably due to the size of the combinatorial search space, it can be made more scalable through cheaper approximations or by making the kernel grammar differentiable .

Another avenue, which was already alluded to above (see Section 2.1), is to use a neural network to parameterize the kernel. The first attempt at this trained a deep belief network on the data and then used it as the kernel function , but later approaches optimized the neural network kernel directly using the marginal likelihood , often in combination with sparse approximations or stochastic variational inference for scalability (see Eq. 7). In this vein, it has recently been proposed to regularize the Lipschitzness of the used neural network, in order for the learned kernel to preserve distances between data points and thus improve its out-of-distribution uncertainties . While all these approaches still rely on the log determinant term in Eq. 35 to protect them from overfitting, it has been shown that this is unfortunately not effective enough when the employed neural networks are overparameterized . However, this can be remedied by adding a prior over the neural network parameters, thus effectively turning them into BNNs and the whole model into a proper hierarchical Bayesian model. It should be noted that these techniques cannot only be used to learn GP priors that work well for a particular task, but also to learn certain invariances from data or to fit GP priors to other (implicit) function-space distributions (c.f., Section 4.2).

As mentioned above, if we have related tasks available, we can use them to meta-learn the GP prior. This can be applied to the kernel as well as the mean function , by optimizing the marginal likelihood on these meta-tasks as

where ${\mathcal{D}}_{M}$ is the set of meta-tasks. Note that the mean function can only safely be optimized in this meta-learning setting, but not in the ML-II setting, since Eq. 35 does not provide any complexity penalty on the mean function and it would thus severely overfit. While meta-learning does not risk overfitting on the actual training data (since it is not used), it might overfit on the meta-tasks, if there are too few of them, or if they are too similar to each other . In the Bayesian meta-learning setting, this can be overcome by specifying a hierarchical hyperprior, which turns out to be equivalent to optimizing a PAC-Bayesian bound . This has been shown to successfully meta-learn GP priors from as few as five meta-tasks.

2 Learning VAE priors

Variational autoencoders are already trained using the ELBO (see Eq. 12), which is a lower bound on the marginal likelihood. Moreover, their likelihood $p({\bm{x}}\,|\,{\bm{z}})$ is trained on this objective, as opposed to being fixed a priori as in most other Bayesian models. One could thus expect that VAEs would be well suited to also learn their prior using their ELBO. Indeed, the ELBO can be further decomposed as

As mentioned above, a more expressive family of prior distributions than the common standard Gaussian priors are Gaussian mixture priors (see Section 3.1). In particular, with an increasing number of components, these mixtures can approximate any smooth distribution arbitrarily closely . These VAE priors can be optimized using the ELBO , however it has been found that this can severely overfit , highlighting again that the marginal likelihood (or its lower bound) cannot always protect against overfitting (see Section 5.1). Instead, it has been proposed to parameterize the mixture components as variational posteriors on certain inducing points, that is

where the ${\bm{x}}_{i}$’s are learned . This can indeed improve the VAE performance without overfitting, and since the prior is defined in terms of inducing points in data space, it can also straightforwardly be used with hierarchical VAEs .

Since mixture models can exacerbate the computation of the KL divergence and require the difficult choice of a number of components $K$, an alternative are implicit priors which are parameterized by learnable functions. One specific example for image data has been proposed for VAEs in which the latent space preserves the shape of the data, that is, the ${\bm{z}}$’s are not just vectors, but 2D or 3D tensors. In such models, one can define a hierarchical prior over ${\bm{z}}$, which is parameterized by learnable convolutions over the latent dimensions . Another way of specifying a learnable hierarchical prior is to use memory modules, where the prior is then dependent on the stored memories and the memory is learned together with the rest of the model . More generally, one can define implicit prior distributions in VAEs as

where $g(\cdot\,;\bm{\psi})$ is a learnable diffeomorphism, such as a normalizing flow . This has been successfully demonstrated with RealNVP flows , where it has been shown that the VAE can learn very expressive latent representations even with a single latent dimension . Moreover, it has been shown that using an autoregressive flow in this way for the prior is equivalent to using an inverse autoregressive flow as part of the decoder .

Finally, one can also reshape some base prior by a multiplicative term, that is

3 Learning BNN priors

Finally, we will consider learning priors for Bayesian neural networks. Due to the large dimensionality of BNN weight spaces and the complex mapping between weights and functions (see Section 4.1), learning BNN priors has not been attempted very often in the literature. A manual prior specification procedure that may be loosely called “learning” is the procedure in Fortuin et al. , where the authors train standard neural networks using gradient descent and use their empirical weight distributions to inform their prior choices. When it comes to proper ML-II optimization, BNNs pose an additional challenge, because their marginal likelihoods are typically intractable and even lower bounds are hard to compute. Learning BNN priors using ML-II has therefore so far only focused on learning the parameters of Gaussian priors in BNNs with Gaussian approximate posteriors, where the posteriors were computed either using moment-matching or using the Laplace-Generalized-Gauss-Newton method , that is

where $q({\mathcal{D}})$ is the marginal likelihood of a Laplace approximation, ${\bm{\theta}}^{*}=\operatorname*{arg\,max}_{{\bm{\theta}}}p({\bm{\theta}}\,|\,{\mathcal{D}})$ is the maximum a posteriori (MAP) estimate of the parameters, $\hat{{\bm{H}}}_{{\bm{\theta}}^{*}}$ is an approximate Hessian around ${\bm{\theta}}^{*}$, ${\bm{J}}_{{\bm{\theta}}^{*}}$ is the Jacobian of the BNN outputs with respect to the parameters, ${\bm{H}}_{{\bm{\theta}}^{*}}^{L}$ is the Hessian of the log likelihood, and ${\bm{H}}_{{\bm{\theta}}^{*}}^{P}$ is the Hessian of the log prior. Using this approximation, the marginal likelihood is actually differentiable with respect to the prior hyperparameters $\bm{\psi}$, such that they can be trained together with the BNN posterior .

Again, if meta-tasks are available, one can try to meta-learn the BNN prior. For CNNs, one can for instance train standard neural networks on the meta-tasks and then learn a generative model (e.g., a VAE) for the filter weights. This generative model can then be used as a BNN prior for convolutional filters . In the case of only few meta-tasks, one can also again use PAC-Bayesian bounds to avoid meta-overfitting, at least when meta-learning Gaussian BNN priors . Finally, if we do not have access to actual meta-tasks, but we are aware of invariances in our data, we can construct meta-tasks using data augmentation and use them to learn a prior that is (approximately) invariant to these augmentations , that is

Conclusion

We have argued that choosing good priors in Bayesian models is crucial to actually achieve the theoretical and empirical properties that they are commonly celebrated for, including uncertainty estimation, model selection, and optimal decision support. While practitioners in Bayesian deep learning currently often resort to the option of isotropic Gaussian (or similarly uninformative) priors, we have also highlighted that these priors are usually misspecified and can lead to several unintended negative consequences during inference. On the other hand, well chosen priors can improve performance and even enable novel applications. Luckily, a plethora of alternative prior choices is available for popular Bayesian deep learning models, such as (deep) Gaussian processes, variational autoencoders, and Bayesian neural networks. Moreover, in certain cases, useful priors for these models can even be learned from data alone.

We hope that this survey—while necessarily being incomplete in certain ways—has provided the interested reader with a first overview of the existing literature on priors for Bayesian deep learning and with some guidance on how to choose them. We also hope to encourage practitioners in this field to consider their prior choices a bit more carefully, and to potentially choose one of the priors presented here instead of the standard Gaussian ones, or better yet, to use inspiration from these priors and come up with even better suited ones for their own models. If only a small fraction of the time usually spent thinking about increasingly elaborate inference techniques will be instead spent on thinking about the priors used, this effort will have been worthwhile.

We acknowledge funding from the Swiss Data Science Center through a PhD fellowship. We thank Alex Immer, Adrià Garriga-Alonso, and Claire Vernade for helpful feedback on the draft and Arnold Weber for constant inspiration.

References