Generalized Energy Based Models

Introduction

Generative Adversarial Networks (GANs) (Goodfellow et al.,, 2014) are a particular way to enforce low dimensional structure in a model. They rely on an implicit model, the generator, to produce samples supported on a low-dimensional manifold by mapping a pre-defined latent noise to the sample space using a trained function. GANs have been very successful in generating high-quality samples on various tasks, especially for unsupervised image generation (Brock et al.,, 2018). The generator is trained adversarially against a discriminator network whose goal is to distinguish samples produced by the generator from the target data. This has inspired further research to extend the training procedure to more general losses (Nowozin et al.,, 2016; Arjovsky et al.,, 2017; Li et al.,, 2017; Bińkowski et al.,, 2018; Arbel et al.,, 2018) and to improve its stability (Miyato et al.,, 2018; Gulrajani et al.,, 2017; Nagarajan and Kolter,, 2017; Kodali et al.,, 2017). While the generator of a GAN has effectively a low-dimensional support, it remains challenging to refine the distribution of mass on that support using pre-defined latent noise. For instance, as shown by Cornish et al., (2020) for normalizing flows, when the latent distribution is unimodal and the target distribution possesses multiple disconnected low-dimensional components, the generator, as a continuous map, compensates for this mismatch using steeper slopes. In practice, this implies the need for more complicated generators.

In the present work, we propose a new class of models, called Generalized Energy Based Models (GEBMs), which can represent distributions supported on low-dimensional manifolds, while offering more flexibility in refining the mass on those manifolds. GEBMs combine the strength of both implicit and explicit models in two separate components: a base distribution (often chosen to be an implicit model) which learns the low-dimensional support of the data, and an energy function that can refine the probability mass on that learned support. We propose to train the GEBM by alternating between learning the energy and the base, analogous to $f$-GAN training (Goodfellow et al.,, 2014; Nowozin et al.,, 2016). The energy is learned by maximizing a generalized notion of likelihood which we relate to the Donsker-Varadhan lower-bound (Donsker and Varadhan,, 1975) and Fenchel duality, as in (Nguyen et al.,, 2010; Nowozin et al.,, 2016). Although the partition function is intractable in general, we propose a method to learn it in an amortized fashion without introducing additional surrogate models, as done in variational inference (Kingma and Welling,, 2014; Rezende et al.,, 2014) or by Dai et al., 2019a ; Dai et al., 2019b . The resulting maximum likelihood estimate, the KL Approximate Lower-bound Estimate (KALE), is then used as a loss for training the base. When the class of energies is rich and smooth enough, we show that KALE leads to a meaningful criterion for measuring weak convergence of probabilities. Following recent work by Chu et al., (2020); Sanjabi et al., (2018), we show that KALE possesses well defined gradients w.r.t. the parameters of the base, ensuring well-behaved training. We also provide convergence rates for the empirical estimator of KALE when the variational family is sufficiently well behaved, which may be of independent interest.

The main advantage of GEBMs becomes clear when sampling from these models: the posterior over the latents of the base distribution incorporates the learned energy, putting greater mass on regions in this latent space that lead to better quality samples. Sampling from the GEBM can thus be achieved by first sampling from the posterior distribution of the latents via MCMC in the low-dimensional latent space, then mapping those latents to the input space using the implicit map of the base. This is in contrast to standard GANs, where the latents of the base have a fixed distribution. We focus on a class of samplers that exploit gradient information, and show that these samplers enjoy fast convergence properties by leveraging the recent work of Eberle et al., (2017). While there has been recent interest in using the discriminator to improve the quality of the generator during sampling (Azadi et al.,, 2019; Turner et al.,, 2019; Neklyudov et al.,, 2019; Grover et al.,, 2019; Tanaka,, 2019; Wu et al., 2019b, ), our approach emerges naturally from the model we consider.

We begin in Section 2 by introducing the GEBM model. In Section 3, we describe the learning procedure using KALE, then derive a method for sampling from the learned model in Section 4. In Section 5 we discuss related work. Finally, experimental results are presented in Section 6 with code available at https://github.com/MichaelArbel/GeneralizedEBM.

Generalized Energy-Based Models

While EBMs have been shown recently to be powerful models for representing complex high dimensional data distributions, they still unavoidably lead to a blurred model whenever data are concentrated on a lower-dimensional manifold. This is the case in Figure 1(a), where the ground truth distribution is supported on a 1-D line and embedded in a 2-D space. The EBM in Figure 1(d) learns to give higher density to a halo surrounding the data, and thus provides a blurred representation. That is a consequence of EBM having a density defined over the whole space, and can result in blurred samples for image models.

GANs are popular instances of these models, and are trained adversarially (Goodfellow et al.,, 2014). When the latent space $\mathcal{Z}$ has a smaller dimension than the input space $\mathcal{X}$, the IGM will be supported on a lower dimensional manifold of $\mathcal{X}$, and thus will not possess a Lebesgue density on $\mathcal{X}$ (Bottou et al.,, 2017). IGMs are therefore good candidates for modelling low dimensional distributions. While GANs can accurately learn the low-dimensional support of the data, they can have limited power for representing the distribution of mass on the support. This is illustrated in Figure 1(b).

In order to hold, Proposition 1 does not need the generator ${{G}}$ to be invertible. We provide a proof in Section C.1 which relies on a characterization of probability distribution using generalized moments. We will see later in Section 4 how equation Equation 5 can be used to provide practical sampling algorithms from the GEBM. Next we discuss the advantages of GEBMs.

Learning GEBMs

In this section we describe a general procedure for learning GEBMs. We decompose the learning procedure into two steps: an energy learning step and a base learning step. The overall learning procedure alternates between these two steps, as done in GAN training (Goodfellow et al.,, 2014).

2 Base learning

The universal approximation assumption holds in particular when $\mathcal{E}$ contains feedforward networks. In fact networks with a single neuron are enough, as shown in (Zhang et al.,, 2017, Theorem 2.3). The Lipschitz assumption holds when additional regularization of the energy is enforced during training by methods such as spectral normalization (Miyato et al.,, 2018) or additional regularization $I(\psi)$ on the energy $E_{\psi}$ such as the gradient penalty (Gulrajani et al.,, 2017) as done in Section 6.

Estimating KALE. According to Arora et al., (2017), accurate finite sample estimates of divergences that result from an optimization procedures (such as in Equation 11) depend on the richness of the class $\mathcal{E}$; and richer energy classes can result in slower convergence. Unlike divergences such as Jensen-Shannon, KL and the Wasserstein distance, which result from optimizing over a non-parametric and rich class of functions, KALE is restricted to a class of parametric energies $E_{\psi}$. Thus, (Arora et al.,, 2017, Theorem 3.1) applies, and guarantees good finite sample estimates, provided optimization is solved accurately. In Appendix B, we provide an analysis for the more general case where energies are not necessarily parametric but satisfy some further smoothness properties; we emphasize that our rates do not require the strong assumption that the density ratio is bounded above and below as in (Nguyen et al.,, 2010).

Under Assumptions (I), (II) and (III) of Section C.2, sub-gradient methods on $\mathcal{K}$ converge to local optima. Moreover, $\mathcal{K}$ is Lipschitz and differentiable for almost all $\theta\in\Theta$ with:

Estimating the gradient in Equation 12 is achieved by first optimizing over $E_{\psi}$ and $A$ using Equation 10, with additional regularization $I(\psi)$. The resulting estimators $\hat{E}^{\star}$ and $\hat{A}^{\star}$ are plugged in Equation 13 to estimate $\nabla\mathcal{K}(\theta)$ using samples $(Z_{m})_{1:M}$ from $\eta$. Unlike for learning the energy $E^{\star}$, which benefits from using the amortized estimator of the log-partition function, we found that using the empirical log-partition for learning the base was more stable. We summarize the training procedure in Algorithm 1, which alternates between learning the energy and the base in a similar fashion to adversarial training.

Sampling from GEBMs

Overdamped samplers are obtained as a time-discretization of the Overdamped Langevin dynamics:

where $w_{t}$ is a standard Brownian motion. The simplest sampler arising from Equation 14 is the Unadjusted Langevin Algorithm (ULA):

Kinetic samplers arise from the Kinetic Langevin dynamics which introduce a momentum variable:

with friction coefficient $\gamma\geq 0$, inverse mass $u\geq 0$, momentum vector $v_{t}$ and standard Brownian motion $w_{t}$. When the mass $u^{-1}$ becomes negligible compared to the friction coefficient $\gamma$, i.e. $u\gamma^{-2}\approx 0$, standard results show that Equation 16 recovers the Overdamped dynamics Equation 14. Discretization in time of Equation 16 leads to Kinetic samplers similar to Hamiltonian Monte Carlo (Cheng et al.,, 2017; Sachs et al.,, 2017). We consider a particular algorithm from Sachs et al., (2017) which we call Kinetic Langevin Algorithm (KLA) (see Algorithm 3 in Appendix F). Kinetic samplers were shown to better explore the modes of the invariant distribution $\nu$ compared to Overdamped ones (see (Neal,, 2010; Betancourt et al.,, 2017) for empirical results and (Cheng et al.,, 2017) for theory), as also confirmed empirically in Appendix D for image generation tasks using GEBMs. Next, we provide the following convergence result:

where $c$ and $C$ are positive constants independent of $t$, with $c=O(\exp(-dim(\mathcal{Z})))$.

Proposition 6 is proved in Section C.1 using (Eberle et al.,, 2017, Corollary 2.6), and implies that $(x_{t})_{t\geq 0}$ converges at the same speed as $(z_{t})_{t\geq 0}$. When the dimension $q$ of $\mathcal{Z}$ is orders of magnitude smaller than the input space dimension $d$, the process $(x_{t})_{t\geq 0}$ converges faster than typical sampling methods on $\mathcal{X}$, for which the exponent controlling the convergence rate is of order $O(\exp(-d))$.

Related work

Generative Adversarial Networks. Recent work proposes using the discriminator of a trained GAN to improve the generator quality. Rejection sampling (Azadi et al.,, 2019) and Metropolis-Hastings correction (Turner et al.,, 2019; Neklyudov et al.,, 2019) perform sampling directly on the high-dimensional input space without using gradient information provided by the discriminator. Moreover, the data distribution is assumed to admit a density w.r.t. the generator. Ding et al., (2019) perform sampling on the feature space of some auxiliary pre-trained network; while Lawson et al., (2019) treat the sampling procedure as a model on its own, learned by maximizing the ELBO. In our case, no auxiliary model is needed. In the present work, sampling doesn’t interfere with training, in contrast to recently considered methods to optimize over the latent space during training Wu et al., 2019b ; Wu et al., 2019a . In Tanaka, (2019), the discriminator is viewed as an optimal transport map between the generator and the data distribution and is used to compute optimized samples from latent space. This is in contrast to the diffusion-based sampling that we consider. In (Xie et al., 2018b, ; Xie et al., 2018a, ), two independent models, a full support EBM and a generator network, are trained cooperatively using MCMC. By contrast, in the present work, the energy and base are part of the same model, and the model support is lower-dimensional than the target space $\mathcal{X}$. While we do not address the mode collapse problem, Xu et al., (2018); Nguyen et al., (2017) showed that KL-based losses are resilient to it thanks to the zero-avoiding property of the KL, a good sign for KALE which is derived from KL by Fenchel duality.

The closest related approach appears in a study concurrent to the present work (Che et al.,, 2020), where the authors propose to use Langevin dynamics on the latent space of a GAN generator, but with a different discriminator to ours (derived from the Jensen-Shannon divergence or a Wasserstein-based divergence). Our theory results showing the existence of the loss gradient (Theorem 5), establishing weak convergence of distributions under KALE (Proposition 4), and demonstrating consistency of the KALE estimator (Appendix B) should transfer to the JS and Wasserstein criteria used in that work. Subsequent to the present work, an alternative approach has been recently proposed, based on normalising flows, to learn both the low-dimensional support of the data and the density on this support (Brehmer and Cranmer,, 2020). This approach maximises the explicit likelihood of a data projection onto a learned manifold, and may be considered complementary to our approach.

Experiments

Experimental setting. We train a GEBM on unsupervised image generation tasks, and compare the quality of generated samples with other methods using the FID score (Heusel et al.,, 2017) computed on $5\times 10^{4}$ generated samples. We consider CIFAR-10 (Krizhevsky,, 2009), LSUN (Yu et al.,, 2015), CelebA (Liu et al.,, 2015) and ImageNet (Russakovsky et al.,, 2014) all downsampled to 32x32 resolution to reduce computational cost. We consider two network architectures for each of the base and energy, a smaller one (SNGAN ConvNet) and a larger one (SNGAN ResNet), both of which are from Miyato et al., (2018). For the base we used the SNGAN generator networks from Miyato et al., (2018) with a $100$-dimensional Gaussian for the latent noise $\eta$. For the energy we used the SNGAN discriminator networks from Miyato et al., (2018). (Details of the networks in Section G.1).

2 Density Estimation

Motivation. We next consider the particular setting where the likelihood of the model is well-defined, and admits a closed form expression. This is intended principally as a sanity check that our proposed training method in Algorithm 1 succeeds in learning maximum likelihood solutions. Outside of this setting, closed form expressions of the normalizing constant are not available for generic GEBMs. While this is not an issue (since the proposed method doesn’t require a closed form expression for the normalizing constant), in this experiment only, we want to have access to closed form expressions, as they enable a direct comparison with other density estimation methods.

Results. Table 3 reports the Negative Log-Likelihood (NLL) evaluated on the test set and corresponding to the best performance on the validation set. Training the GEBM using Algorithm 1 leads to comparable performance to (CD) and (ML). As shown in Figure 7 of Appendix E, (KALE-DV) and (KALE-F) maintain a small error gap between the training and test NLL and, as discussed in Sections 3.1 and F, (KALE-F) leads to more accurate estimates of the log-partition function, with a relative error of order $0.1\%$ compared to $10\%$ for (KALE-DV).

Acknowledgments

We thank Mihaela Rosca for insightful discussions and Song Liu, Bo Dai and Hanjun Dai for pointing us to important related work.

Appendix A KL Approximate Lower-bound Estimate

Nguyen et al., (2010); Nowozin et al., (2016) derived a variational formulation for the KL using Fenchel duality. By the duality theorem (Rockafellar,, 1970), the convex and lower semi-continuous function $\zeta:u\mapsto u\log(u)$ that appears in Equation 18 can be expressed as the supremum of a concave function:

Nguyen et al., (2010) provided the variational formulation for the reverse KL using a different choice for $\zeta$: ($\zeta(u)=-\log(u)$). We refer to (Nowozin et al.,, 2016) for general $f$-divergences. Choosing a smaller set of functions $\mathcal{H}$ in the variational objective Equation 20 will lead to a lower bound on the KL. This is the KL Approximate Lower-bound Estimate (KALE):

We defer the consistency analysis of Equation 23 to Appendix B where we provide convergence rates in a setting where the set of functions $\mathcal{H}$ is a Reproducing Kernel Hilbert Space and under weaker assumptions that were not covered by the framework of Nguyen et al., (2010).

Appendix B Convergence rates of KALE

Any function $h$ in $\mathcal{H}$ satisfies the reproducing property $f(x)=\langle f,k(x,.)\rangle$ for any $x\in\mathcal{X}$.

The proposed estimator is obtained by solving a regularized empirical problem,

Finally, we introduce $D(h,\delta)$ and $\Gamma(h,\delta)$:

The empirical versions of $D(h,\delta)$ and $\Gamma(h,\delta)$ are denoted $\hat{D}(h,\delta)$ and $\hat{\Gamma}(h,\delta)$. Later, we will show that $D(h,\delta)$ $\hat{D}(h,\delta)$ are in fact the gradients of $\mathcal{F}(h)$ and $\widehat{\mathcal{F}}(h)$ along the direction $\delta$.

The supremum of $\mathcal{F}$ over $\mathcal{H}$ is attained at $h_{0}$.

The following quantities are finite for some positive $\epsilon$:

For any $h\in\mathcal{H}$, if $D(h,\delta)=0$ for all $\delta$ then $h=h_{0}$.

Fix any $1>\eta>0$. Under Assumptions (i), (ii) and (iii), and provided that $\lambda=\frac{1}{\sqrt{N}}$, it holds with probability at least $1-2\eta$ that

for a constant $M^{\prime}(\eta,h_{0})$ that depends only on $\eta$ and $h_{0}$.

The assumptions in Theorem 7 essentially state that the kernel associated to the RKHS $\mathcal{H}$ needs to satisfy some integrability requirements. That is to guarantee that the gradient $\delta\mapsto\nabla\mathcal{F}(h)(\delta)$ and its empirical version are well-defined and continuous. In addition, the optimality condition $\nabla\mathcal{F}(h)=0$ is assumed to characterize the global solution $h_{0}$. This will be the case if the kernel is characteristic Simon-Gabriel and Scholkopf, (2018). The proof of Theorem 7, in Section B.2, takes advantage of the Hilbert structure of the set $\mathcal{H}$, the convexity of the functional $\mathcal{F}$ and the optimality condition $\nabla\widehat{\mathcal{F}}(\hat{h})=\lambda\hat{h}$ of the regularized problem, all of which turn out to be sufficient for controlling the error of Equation 23.

B.2 Proofs

We state now the proof of Theorem 7 with subsequent lemmas and propositions.

We begin with the following inequalities:

The first inequality is by definition of $\hat{h}$ while the second is obtained by concavity of $\widehat{\mathcal{F}}$. For simplicity we write $\mathcal{B}=\|\hat{h}-h_{0}\|$ and $\mathcal{C}=\|\nabla\widehat{\mathcal{F}}(h_{0})-\mathcal{L}(h_{0})\|$. Using Cauchy-Schwarz and triangular inequalities, it is easy to see that

Moreover, by triangular inequality, it holds that

Lemma 11 ensures that $\mathcal{A}(\lambda)=\|h_{\lambda}-h_{0}\|$ converges to as $\lambda\rightarrow 0$. Furthermore, by Proposition 12, we have $\|\hat{h}-h_{\lambda}\|\leq\frac{1}{\lambda}\mathcal{D}$ where $\mathcal{D}(\lambda)=\|\nabla\widehat{\mathcal{F}}(h_{\lambda})-\nabla\mathcal{L}(h_{\lambda})\|$. Now choosing $\lambda=\frac{1}{\sqrt{N}}$ and applying Chebychev inequality in Lemma 8, it follows that for any $1>\eta>0,$ we have with probability greater than $1-2\eta$ that both

where $C(\|h_{0}\|,\eta)$ is defined in Lemma 8. This allows to conclude that for any $\eta>0$, it holds with probability at least $1-2\eta$ that $|\widehat{\mathcal{F}}(\hat{h})-\widehat{\mathcal{F}}(h_{0})|\leq\frac{M^{\prime}(\eta,h_{0})}{\sqrt{N}}$ where $M^{\prime}(\eta,h_{0})$ depends only on $\eta$ and $h_{0}$.

We proceed using the following lemma, which provides an expression for $D(h,\delta)$ and $\hat{D}(h,\delta)$ along with a probabilistic bound:

Under Assumptions (i) and (ii), for any $h\in\mathcal{H}$ such that $\|h\|\leq\|h_{0}\|+\epsilon$, there exists $\mathcal{D}(h)$ in $\mathcal{H}$ satisfying

and for any $h\in\mathcal{H}$, there exists $\widehat{\mathcal{D}}(h)$ satisfying

Moreover, for any $0<\eta<1$ and any $h\in\mathcal{H}$ such that $\|h\|\leq\|h_{0}\|+\epsilon:=M$, it holds with probability greater than $1-\eta$ that

where $C(M,\eta)$ depends only on $M$ and $\eta$.

Finally, the probabilistic inequality is a simple consequence of Chebychev’s inequality. ∎

The next lemma states that $\mathcal{F}(h)$ and $\widehat{\mathcal{F}}(h)$ are Frechet differentiable.

Under Assumptions (i) and (ii) , $h\mapsto\mathcal{F}(h)$ is Frechet differentiable on the open ball of radius $\|h_{0}\|+\epsilon$ while $h\mapsto\widehat{\mathcal{F}}(h)$ is Frechet differentiable on $\mathcal{H}$. Their gradients are given by $\mathcal{D}(h)$ and $\widehat{\mathcal{D}}(h)$ as defined in Lemma 8,

The empirical functional $\widehat{\mathcal{F}}(h)$ is differentiable since it is a finite sum of differentiable functions, and its gradient is simply given by $\widehat{\mathcal{D}}(h)$. For the population functional, we use second order Taylor expansion of $\exp$ with integral remainder, which gives

By Assumption (ii) we know that $\frac{\Gamma(h,\delta)}{\|\delta\|}$ converges to as soon as $\|\delta\|\rightarrow 0$. This allows to directly conclude that $\mathcal{F}$ is Frechet differentiable, with differential given by $\delta\mapsto D(h,\delta)$. By Lemma 8, we conclude the existence of a gradient $\nabla\mathcal{F}(h)$ which is in fact given by $\nabla\mathcal{F}(h)=\mathcal{D}(h)$.

From now on, we will only use the notation $\nabla\mathcal{F}(h)$ and $\nabla\widehat{\mathcal{F}}(h)$ to refer to the gradients of $\mathcal{F}(h)$ and $\widehat{\mathcal{F}}(h)$. The following lemma states that Equations 29 and 30 have a unique global optimum, and gives a first order optimality condition.

The problems Equations 29 and 30 admit unique global solutions $\hat{h}$ and $h_{\lambda}$ in $\mathcal{H}$. Moreover, the following first order optimality conditions hold:

For Equation 29, existence and uniqueness of a minimizer $\hat{h}$ is a simple consequence of continuity and strong concavity of the regularized objective. We now show the existence result for Equation 30. Let’s introduce $\mathcal{G}_{\lambda}(h)=-\mathcal{F}(h)+\frac{\lambda}{2}\|h\|^{2}$ for simplicity. Uniqueness is a consequence of the strong convexity of $\mathcal{G}_{\lambda}$. For the existence, consider a sequence of elements $f_{k}\in\mathcal{H}$ such that $\mathcal{G}_{\lambda}(f_{k})\rightarrow\inf_{h\in\mathcal{H}}\mathcal{G}_{\lambda}(h)$. If $h_{0}$ is not the global solution, then it must hold for $k$ large enough that $\mathcal{G}_{\lambda}(f_{k})\leq\mathcal{G}_{\lambda}(h_{0})$. We also know that $\mathcal{F}(f_{k})\leq\mathcal{F}(h_{0})$, hence, it is easy to see that $\|f_{k}\|\leq\|h_{0}\|$ for $k$ large enough. This implies that $f_{k}$ is a bounded sequence, therefore it admits a weakly convergent sub-sequence by weak compactness. Without loss of generality we assume that $f_{k}$ weakly converges to some element $h_{\lambda}\in\mathcal{H}$ and that $\|f_{k}\|\leq\|h_{0}\|$. Hence, $\|h_{\lambda}\|\leq\lim\inf_{k}\|f_{k}\|\leq\|h_{0}\|$. Recall now that by definition of weak convergence, we have $f_{k}(x)\rightarrow_{k}h_{\lambda}(x)$ for all $x\in\mathcal{X}$. By Assumption (ii), we can apply the dominated convergence theorem to ensure that $\mathcal{F}(f_{k})\rightarrow\mathcal{F}(h_{\lambda})$. Taking the limit of $\mathcal{G}_{\lambda}{f_{k}}$, the following inequality holds:

Finally, by Lemma 9 we know that $\mathcal{F}$ is Frechet differentiable, hence we can use Ekeland and Témam, (1999) (Proposition 2.1) to conclude that $\nabla\mathcal{F}(h_{\lambda})=\lambda h_{\lambda}$. We use exactly the same arguments for Equation 29. ∎

Next, we show that $h_{\lambda}$ converges towards $h_{0}$ in $\mathcal{H}$.

Under Assumptions (i), (iii) and (ii) it holds that:

We will first prove that $h_{\lambda}$ converges weakly towards $h_{0},$ and then conclude that it must also converge strongly. We start with the following inequalities:

These are simple consequences of the definitions of $h_{\lambda}$ and $h_{0}$ as optimal solutions to Equations 29 and 25. This implies that $\|h_{\lambda}\|$ is always bounded by $\|h_{0}\|$. Consider now an arbitrary sequence $(\lambda_{m})_{m\geq 0}$ converging to . Since $\|h_{\lambda_{m}}\|$ is bounded by $\|h_{0}\|$, it follows by weak-compactness of balls in $\mathcal{H}$ that $h_{\lambda_{m}}$ admits a weakly convergent sub-sequence. Without loss of generality we can assume that $h_{\lambda_{m}}$ is itself weakly converging towards an element $h^{*}$. We will show now that $h^{*}$ must be equal to $h_{0}$. Indeed, by optimality of $h_{\lambda_{m}}$, it must hold that

This implies that $\nabla\mathcal{F}(h_{m})$ converges weakly to . On the other hand, by Assumption (ii), we can conclude that $\nabla\mathcal{F}(h_{m})$ must also converge weakly towards $\nabla\mathcal{F}(h^{*})$, hence $\nabla\mathcal{F}(h^{*})=0$. Finally by Assumption (iii) we know that $h_{0}$ is the unique solution to the equation $\nabla\mathcal{F}(h)=0$ , hence $h^{*}=h_{0}$. We have shown so far that any subsequence of $h_{\lambda_{m}}$ that converges weakly, must converge weakly towards $h_{0}$. This allows to conclude that $h_{\lambda_{m}}$ actually converges weakly towards $h_{0}$. Moreover, we also have by definition of weak convergence that:

Recalling now that $\|h_{\lambda_{m}}\|\leq\|h_{0}\|$ it follows that $\|h_{\lambda_{m}}\|$ converges towards $\|h_{0}\|$. Hence, we have the following two properties:

$h_{\lambda_{m}}$ converges weakly towards $h_{0}$,

$\|h_{\lambda_{m}}\|$ converges towards $\|h_{0}\|$.

This allows to directly conclude that $\|h_{\lambda_{m}}-h_{0}\|$ converges to . ∎

By definition of $\hat{h}$ and $h_{\lambda}$ the following optimality conditions hold:

Now introducing $\delta:=\hat{h}-h_{\lambda}$ and $E:=\nabla\widehat{\mathcal{F}}(\hat{h})-\nabla\widehat{\mathcal{F}}(h_{\lambda})$ for simplicity and taking the squared norm of the above equation, it follows that

By concavity of $\widehat{\mathcal{F}}$ on $\mathcal{H}$ we know that $-\langle\hat{h}-h_{\lambda},E\rangle\geq 0$. Therefore:

Appendix C Latent noise sampling and Smoothness of KALE

Here we prove Proposition 6 for which we make the assumptions more precise:

$\log\eta$ is strongly concave and admits a Lipschitz gradient.

There exists a non-negative constant $L$ such that for any $x,x^{\prime}\in\mathcal{X}$ and $z,z^{\prime}\in\mathcal{Z}$:

Throughout this section, we introduce $U(z):=-\log(\eta(z))+E({{G}}(z))$ for simplicity.

Let $\pi_{t}$ be the probability distribution of $(z_{t},v_{t})$ at time $t$ of the diffusion in Equation 16, which we recall that

We call $\pi_{\infty}$ its corresponding invariant distribution given by

By Lemma 13 we know that $U$ is dissipative, bounded from below, and has a Lipschitz gradient. This allows to directly apply (Eberle et al.,, 2017)(Corollary 2.6.) which implies that

where $c$ is a positive constant and $C$ only depends on $\pi_{\infty}$ and the initial distribution $\pi_{0}$. Moreover, the constant $c$ is given explicitly in (Eberle et al.,, 2017, Theorem 2.3) and is of order $0(e^{-q})$ where $q$ is the dimension of the latent space $\mathcal{Z}$.

The second line uses the definition of $(x_{t},x)$ as joint samples obtained by mapping $(z_{t},z)$. The third line uses the assumption that $B$ is $L$-Lipschitz. Finally, the last line uses that $\Pi_{t}$ is an optimal coupling between $\pi_{t}$ and $\pi_{\infty}$. ∎

Under 1, there exists $A>0$ and $\lambda\in(0,\frac{1}{4}]$ such that

where $\gamma$ and $u$ are the coefficients appearing in Equation 16. Moreover, $U$ is bounded bellow and has a Lipschitz gradient.

For simplicity, let’s call $u(z)=-\log\eta(z)$, $w(z)=E^{\star}\circ B_{\theta^{\star}}(z),$ and denote by $M$ an upper-bound on the Lipschitz constant of $w$ and $\nabla w$ which is guaranteed to be finite by assumption. Hence $U(z)=u(z)+w(z)$. Equation Equation 69 is equivalent to having

Using that $w$ is Lipschitz, we have that $w(z)\leq w(0)+M\|z\|$ and $-z^{\top}\nabla w(z)\leq M\|z\|$. Hence, $2\lambda w(z)-z^{\top}\nabla w(z)-2A\leq 2\lambda w(0)+(2\lambda+1)M\|z\|-2A$. Therefore, a sufficient condition for Equation 70 to hold is

We will now rely on the strong convexity of $u,$ which holds by assumption, and implies the existence of a positive constant $m>0$ such that

This allows to write the following inequality,

Combining the previous inequality with Equation 71 and denoting $M^{\prime}=\|\nabla u(0)\|$ , it is sufficient to find $A$ and $\lambda$ satisfying

The l.h.s. in the above equation is a quadratic function in $\|z\|$ and admits a global minimum when $\lambda<\left(m+\frac{\gamma^{2}}{2u}\right)^{-1}$. The global minimum is always positive provided that $A$ is large enough.

To see that $U$ is bounded below, it suffice to note, by Lipschitzness of $w$, that $w(z)\geq w(0)-M\|z\|$ and by strong convexity of $u$ that

Hence, $U$ is lower-bounded by a quadratic function in $\|z\|$ with positive leading coefficient $\frac{m}{2}$, hence it must be lower-bounded by a constant. Finally, by assumption, $u$ and $w$ have Lipschitz gradients, which directly implies that $U$ has a Lipschitz gradient. ∎

C.2 Topological and smoothness properties of KALE

Topological properties of KALE. Denseness and smoothness of the energy class $\mathcal{E}$ are the key to guarantee that KALE is a reliable criterion for measuring convergence. We thus make the following assumptions on $\mathcal{E}$:

For all $E\in\mathcal{E}$, $-E\in\mathcal{E}$ and there is $C_{E}>0$ such that $cE\in\mathcal{E}$ for $0\leq c\leq C_{E}$. For any continuous function $g$, any compact support $K$ in $\mathcal{X}$ and any precision $\epsilon>0$, there exists a finite linear combination of energies $G=\sum_{i=1}^{r}a_{i}E_{i}$ such that $\sup_{x\in K}|f(x)-G(x)|\leq\epsilon.$

All energies $E$ in $\mathcal{E}$ are Lipschitz in their input with the same Lipschitz constant $L>0$.

Assumption (A) holds in particular when $\mathcal{E}$ contains feedforward networks with a given number of parameters. In fact networks with a single neuron are enough, as shown in (Zhang et al.,, 2017, Theorem 2.3). Assumption (B) holds when additional regularization of the energy is enforced during training by methods such as spectral normalization Miyato et al., (2018) or gradient penalty Gulrajani et al., (2017) as done in Section 6. Proposition 4 states the topological properties of KALE ensuring that it can be used as a criterion for weak convergence. A proof is given in Section C.2.1 and is a consequence of (Zhang et al.,, 2017, Theorem B.1).

Under Assumptions (A) and (B) it holds that:

In this section we prove Proposition 4. We first start by recalling the required assumptions and make them more precise:

For all $E\in\mathcal{E}$, $-E\in\mathcal{E}$ and there is $C_{E}>0$ such that $cE\in\mathcal{E}$ for $0\leq c\leq C_{E}$. For any continuous function $g$, any compact support $K$ in $\mathcal{X}$ and any precision $\epsilon>0$, there exists a finite linear combination of energies $G=\sum_{i=1}^{r}a_{i}E_{i}$ such that $|f(x)-G(x)|\leq\epsilon$ on $K$.

All energies $E$ in $\mathcal{E}$ are Lipschitz in their input with the same Lipschitz constant $L>0$.

We proceed by proving the separation properties ($1^{st}$ statement), then the metrization of the weak topology ($2^{nd}$ statement).

Assume that for any $\epsilon>0$ and any $h$ and $h^{\prime}$ in $\mathcal{E}$ there exists $f$ in $2\mathcal{E}$ such that $\|h+h^{\prime}-f\|_{\infty}\leq\epsilon$ then there exists a constant $C$ such that:

C.2.2 Smoothness properties of KALE

We will now prove Theorem 5. We begin by stating the assumptions that will be used in this section:

$\mathcal{E}$ is parametrized by a compact set of parameters $\Psi$.

Functions in $\mathcal{E}$ are jointly continuous w.r.t. $(\psi,x)$ and are $L$-lipschitz and $L$-smooth w.r.t. the input $x$:

$(\theta,z)\mapsto{{G}}_{\theta}(z)$ is jointly continuous in $\theta$ and $z$, with $z\mapsto{{G}}_{\theta}(z)$ uniformly Lipschitz w.r.t. $z$:

Moreover, $a$ and $b$ are integrable in the following sense:

To show that sub-gradient methods converge to local optima, we only need to show that $\mathcal{K}$ is Lipschitz continuous and weakly convex. This directly implies convergence to local optima for sub-gradient methods, according to Davis and Drusvyatskiy, (2018); Thekumparampil et al., (2019). Lipschitz continuity ensures that $\mathcal{K}$ is differentiable for almost all $\theta\in\Theta,$ and weak convexity simply means that there exits some positive constant $C\geq 0$ such that $\theta\mapsto\mathcal{K}(\theta)+C\|\theta\|^{2}$ is convex. We now proceed to show these two properties.

We will first prove that $\theta\mapsto\mathcal{K}(\theta)$ is weakly convex in $\theta$. By Lemma 15, we know that for any $E\in\mathcal{E}$, the function $\theta\mapsto\mathcal{L}_{\theta}(E)$ is $M$-smooth for the same positive constant $M$. This directly implies that it is also weakly convex and the following inequality holds:

Taking the supremum w.r.t. $E$, it follows that

This means precisely that $\mathcal{K}$ is weakly convex in $\theta$.

To prove that $\mathcal{K}$ is Lipschitz, we will also use Lemma 15, which states that $\mathcal{L}_{\theta}(E)$ is Lipschitz in $\theta$ uniformly on $\mathcal{E}$. Hence, the following holds:

Again, taking the supremum over $E$, it follows directly that

We conclude that $\mathcal{K}$ is Lipschitz by exchanging the roles of $\theta$ and $\theta^{\prime}$ to get the other side of the inequality. Hence, by the Rademacher theorem, $\mathcal{K}$ is differentiable for almost all $\theta$.

We will now provide an expression for the gradient of $\mathcal{K}$. By Lemma 16 we know that $\psi\mapsto\mathcal{L}_{\theta}(E_{\psi})$ is continuous and by Assumption (I) $\Psi$ is compact. Therefore, the supremum $\sup_{E\in\mathcal{E}}\mathcal{L}_{\theta}(E)$ is achieved for some function $E^{\star}_{\theta}$. Moreover, we know by Lemma 15 that $\mathcal{L}_{\theta}(E)$ is smooth uniformly on $\mathcal{E}$, therefore the family $(\partial_{\theta}\mathcal{L}_{\theta}(E))_{E\in\mathcal{E}}$ is equi-differentiable. We are in position to apply Milgrom and Segal, (2002)(Theorem 3) which ensures that $\mathcal{K}(\theta)$ admits left and right partial derivatives given by

Under Assumptions (I), (II) and (III), the functional $\mathcal{L}_{\theta}(E)$ is Lipschitz and smooth in $\theta$ uniformly on $\mathcal{E}$:

By Lemma 16, we have that $\mathcal{L}_{\theta}(E)$ is differentiable, and that

Lemma 16 ensures that $\|\partial_{\theta}\mathcal{L}_{\theta}(E)\|$ is bounded by some positive constant $C$ that is independent from $E$ and $\theta$. This implies in particular that $\mathcal{L}_{\theta}(E)$ is Lipschitz with a constant $C$. We will now show that it is also smooth. For this, we need to control the difference

The first term can be upper-bounded using $L_{e}$-smoothness of $E$ and the fact that ${{G}}_{\theta}$ is Lipschitz in $\theta$:

The last inequality was obtained by Lemma 17. Similarly, using that $\nabla_{\theta}{{G}}_{\theta}$ is Lipschitz, it follows by Lemma 17 that

Finally, for the last term $III$, we first consider a path $\theta_{t}=t\theta+(1-t)\theta^{\prime}$ for $t\in,$ and introduce the function $s(t):=p_{E,\theta_{t}}\circ{{G}}_{\theta_{t}}$. We will now control the difference $p_{E,\theta}\circ{{G}}_{\theta}-p_{E,\theta^{\prime}}\circ{{G}}_{\theta^{\prime}},$ also equal to $s(1)-s(0)$. Using the fact that $s_{t}$ is absolutely continuous we have that $s(1)-s(0)=\int_{0}^{1}s^{\prime}(t)dt$. The derivative $s^{\prime}(t)$ is simply given by $s^{\prime}(t)=(\theta-\theta^{\prime})^{\top}(M_{t}-\bar{M}_{t})s(t)$ where $M_{t}=(\nabla_{x}E\circ B_{\theta_{t}})\nabla_{\theta}{{G}}_{\theta_{t}}$ and $\bar{M}_{t}=\int M_{t}p_{E,\theta_{t}}\circ{{G}}_{\theta_{t}}d\eta$. Hence,

We also know that $M_{t}$ is upper-bounded by $La(z),$ which implies

where the last inequality is obtained using Lemma 17. This allows us to conclude that $\mathcal{L}_{\theta}(E)$ is smooth for any $E\in\mathcal{E}$ and $\theta\in\Theta$. ∎

Under Assumptions (II) and (III), it holds that $\psi\mapsto\mathcal{L}_{\theta}(E_{\psi})$ is continuous, and that $\theta\mapsto\mathcal{L}_{\theta}(E_{\psi})$ is differentiable in $\theta$ with gradient given by

Moreover, the gradient is bounded uniformly in $\theta$ and $E$:

To show that $\psi\mapsto\mathcal{L}_{\theta}(E_{\psi})$ is continuous, we will use the dominated convergence theorem. We fix $\psi_{0}$ in the interior of $\Psi$ and consider a compact neighborhood $W$ of $\psi_{0}$. By assumption, we have that $(\psi,x)\mapsto E_{\psi}(x)$ and $(\psi,z)\mapsto E_{\psi}({{G}}_{\theta}(z))$ are jointly continuous. Hence, $|E_{\psi}(0)|$ and $|E_{\psi}({{G}}_{\theta}(0))|$ are bounded on $W$ by some constant $C$. Moreover, by Lipschitz continuity of $x\mapsto E_{\psi}$, we have

To show that $\theta\mapsto\mathcal{L}_{\theta}(E_{\psi})$ is differentiable in $\theta$, we will use the differentiation lemma in (Klenke,, 2008, Theorem 6.28). We first fix $\theta_{0}$ in the interior of $\Theta$, and consider a compact neighborhood $V$ of $\theta_{0}$. Since $\theta\mapsto|E({{G}}_{\theta}(0))|$ is continuous on the compact neighborhood $V$ it admits a maximum value $C$; hence we have using Assumptions (II) and (III) that

Along with the integrability assumption in Assumption (III), this ensures that $z\mapsto\exp(-E({{G}}_{\theta}(z)))$ is integrable w.r.t $\eta$ for all $\theta$ in $V$. We also have that $\exp(-E({{G}}_{\theta}(z)))$ is differentiable, with gradient given by

Using that $E$ is Lipschitz in its inputs and ${{G}}_{\theta}(z)$ is Lipschitz in $\theta$, and combining with the previous inequality, it follows that

where $a(z)$ is the location dependent Lipschitz constant introduced in Assumption (III). The r.h.s. of the above inequality is integrable by Assumption (III) and is independent of $\theta$ on the neighborhood $V$. Thus (Klenke,, 2008, Theorem 6.28) applies, and it follows that

We can now directly compute the gradient of $\mathcal{L}_{\theta}(E)$,

Since $E$ and ${{G}}_{\theta}$ are Lipschitz in $x$ and $\theta$ respectively, it follows that $\|\nabla_{x}E({{G}}_{\theta_{0}}(z))\|\leq L_{e}$ and $\|\nabla_{\theta}{{G}}_{\theta_{0}}(z)\|\leq a(z)$. Hence, we have

Finally, Lemma 17 allows us to conclude that $\|\nabla_{\theta}\mathcal{L}_{\theta}(E)\|$ is bounded by a positive constant $C$ independently from $\theta$ and $E$. ∎

Under Assumptions (II) and (III), there exists a constant $C$ independent from $\theta$ and $E$ such that

By Lipschitzness of $E$ and ${{G}}_{\theta}$, we have $\exp(-L_{e}L_{b}\|z\|)\leq\exp(E({{G}}_{\theta}(0))-E({{G}}_{\theta}(z))\leq\exp(L_{e}L_{b}\|z\|)$, thus introducing the factor $\exp(E(B_{\theta_{0}}(0))$ in Equation 132 we get

The r.h.s. of both inequalities is independent of $\theta$ and $E,$ and finite by the integrability assumptions in Assumption (III). ∎

Appendix D Image Generation

Figures 3 and 4 show sample trajectories using Algorithm 3 with no friction $\gamma=0$ for the 4 datasets. It is clear that along the same MCMC chain, several image modes are explored. We also notice the transition from a mode to another happens almost at the same time for all chains and corresponds to the gray images. This is unlike Langevin or when the friction coefficient $\gamma$ is large as in Figure 5. In that case each chain remains within the same mode.

Table 4 shows further comparisons with other methods on Cifar10 and ImageNet 32x32.

Appendix E Density Estimation

Figure Figure 7 (left) shows the error in the estimation of the log-partition function using both methods (KALE-DV and KALE-F). KALE-DV estimates the negative log-likelihood on each batch of size $100$ and therefore has much more variance than KALE-F which maintains the amortized estimator of the log-partition function.

Figure Figure 7 (right) shows the evolution of the negative log-likelihood (NLL) on both training and test sets per epochs for RedWine and Whitewine datasets. The error decreases steadily in the case of KALE-DV and KALE-F while the error gap between the training and test set remains controlled. Larger gaps are observed for both direct maximum likelihood estimation and Contrastive divergence although the training NLL tends to decrease faster than for KALE.

Appendix F Algorithms

In Algorithm 1, we describe the general algorithm for training a GEBM which alternates between gradient steps on the energy and the generator. An additional regularization, denoted by $I(\psi)$ is used to ensure conditions of Propositions 4 and 5 hold. $I(\psi)$ can include $L_{2}$ regularization over the parameters $\psi$, a gradient penalty as in Gulrajani et al., (2017) or Spectral normalization Miyato et al., (2018). The energy can be trained either using the estimator in Equation 8 (KALE-DV) or the one in Equation 10 (KALE-F) depending on the variable $\mathcal{C}$.

In Algorithm 3, we describe the MCMC sampler proposed in Sachs et al., (2017) which is a time discretization of Equation 16.

Appendix G Experimental details

In all experiments, we use regularization which is a combination of $L_{2}$ norm and a variant of the gradient penalty Gulrajani et al., (2017). For the image generation tasks, we also employ spectral normalization Miyato et al., (2018). This is to ensure that the conditions in Propositions 4 and 5 hold. We pre-condition the gradient as proposed in Simsekli et al., (2020) to stabilize training, and to avoid taking large noisy gradient steps due to the exponential terms in Equations 8 and 10. We also use the second-order updates in Equation 136 for the variational constant $c$ whenever it is learned.

Table 6 and Table 6 show the network architectures used for the GEBM in the case of SNGAN ConvNet. Table 6 and Table 6 show the network architectures used for the GEBM in the case of SNGAN ResNet. The residual connections of each residual block consists of two convolutional layers proceeded by a BatchNormalization and ReLU activation: BN+ReLU+Conv+BN+ReLU+Conv as in (Miyato et al.,, 2018, Figure 8).

We train both base and energy by alternating $5$ gradient steps to learn the energy vs $1$ gradient step to learn the base. For the first two gradient iterations and after every $500$ gradient iterations on base, we train the energy for $100$ gradient steps instead of $5$. We then train the model up to $150000$ gradient iterations on the base using a batch-size of $128$ and Adam optimizer Kingma and Ba, (2014) with initial learning rate of $10^{-4}$ and parameters $(0.5,.999)$ for both energy and base.

We decrease the learning rate using a scheduler that monitors the FID score in a similar way as in Bińkowski et al., (2018); Arbel et al., (2018). More precisely, every $2000$ gradient iterations on the base, we evaluate the FID score on the training set using $50000$ generated samples from the base and check if the current score is larger than the score $20000$ iterations before. The learning rate is decreased by a factor of $0.8$ if the FID score fails to decrease for $3$ consecutive times.

For (DOT) Tanaka, (2019), we use the following objective:

where $z_{y}$ is sampled from a standard Gaussian, $\epsilon$ is a perturbation meant to stabilize sampling and $k_{eff}$ is the estimated Lipschitz constant of $E\circ B$. Note that Equation 137 uses a flipped sign for the $E\circ B$ compared to Tanaka, (2019). This is because $E$ plays the role of $-D$ where $D$ is the discriminator in Tanaka, (2019). Introducing the minus sign in Equation 137 leads to a degradation in performance. We perform $1000$ gradient iterations with a step-size of $0.0001$ which is also decreased by a factor of $10$ every $200$ iterations as done for the proposed method. As suggested by the authors of Tanaka, (2019) we perform the following projection for the gradient before applying it:

We set the perturbation $\epsilon$ to $0.001$ and $k_{eff}$ to $1$ which was also shown in Tanaka, (2019) to perform well. In fact, we found that estimating the Lipschitz constant by taking the maximum value of $\|\nabla E\circ{{G}}(z)\|$ over $1000$ latent samples according to $\eta$ lead to higher values for $k_{eff}$: ( Cifar10: $9.4$, CelebA : $7.2$, ImageNet: $4.9$, Lsun: $3.8$). However, those higher values did not perform as well as setting $k_{eff}=1$.

For (IHM) Turner et al., (2019) we simply run the MCMC chain for $1000$ iterations.

G.2 Density estimation

We use code and pre-processing steps from Wenliang et al., (2019) which we describe here for completeness. For RedWine and WhiteWine, we added uniform noise with support equal to the median distances between two adjacent values. That is to avoid instabilities due to the quantization of the datasets. For Hepmass and MiniBoone, we removed ill-conditioned dimensions as also done in Papamakarios et al., (2017). We split all datasets, except HepMass into three splits. The test split consists of $10\%$ of the total data. For the validation set, we use $10\%$ of the remaining data with an upper limit of $1000$ to reduce the cost of validation at each iteration. For HepMass, we used the sample splitting as done in Papamakarios et al., (2017). Finally, the data is whitened before fitting and the whitening matrix was computed on at most $10000$ data points.

We set the regularization parameter to $0.1$ and use a combination of $L_{2}$ norm and a variant of the gradient penalty Gulrajani et al., (2017):

For both base and energy, we used an NVP Dinh et al., (2016) with 5 NVP layers each consisting of a shifting and scaling layer with two hidden layers of $100$ neurons. We do not use Batch-normalization.

In all cases we use Adam optimizer with learning rate of $0.001$ and momentum parameters $(0.5,0.9)$. For both KALE-DV and KALE-F, we used a batch-size of $100$ data samples vs $2000$ generated samples from the base in order to reduce the variance of the estimation of the energy. We alternate $50$ gradient steps on the energy vs $1$ step on the base and further perform $50$ additional steps on the energy for the first two gradient iterations and after every $500$ gradient iterations on base. For Contrastive divergence, each training step is performed by first producing $100$ samples from the model using $100$ Langevin iterations with a step-size of $10^{-2}$ and starting from a batch of $100$ data-samples. The resulting samples are then used to estimate the gradient of the of the loss.

For (CD), we used 100 Langevin iterations for each learning step to sample from the EBM. This translates into an improved performance at the expense of increased computational cost compared to the other methods. All methods are trained for 2000 epochs with batch-size of 100 (1000 on Hepmass and Miniboone datasets) and fixed learning rate $0.001$, which was sufficient for convergence.

G.3 Illustrative example in Figure 1

with $\theta=(W,B,W^{\prime},b)$. we also call $\theta^{\star}=(1,1,1,0)$. In addition, we consider a sigmoid like function $h$ from $$to$$ of the form:

: To generate a data point $X=(X_{1},X_{2})$, we consider the following simple generative model:

Apply the distortion function $h$ to get a latent sample $Y=h(Z)$.

Generate point $X$ using $X_{1}=G_{\theta^{\star}}^{(1)}(Y)$ and $X_{2}=G_{\theta^{\star}}^{(2)}(Y)$.

Hence, the data are supported on the $1$-d line defined by the equation $X_{2}=G_{\theta^{\star}}^{(2)}(X_{1})$.

For the generator we sample $Z$ uniformly from $$then generate a sample$(X_{1},X_{2})=(G_{\theta}^{(1)}(Z),G_{\theta}^{(2)}(Z))$. The goal is to learn$\theta$.

For the discriminator, we used an MLP with $6$ layers and $10$ hidden units.

For the base we use the same generator as in the GAN model. For the energy we use the same MLP as discriminator of the GAN model.

To ensure tractability of the likelihood, we use the following model:

$MoG((\mu_{1},\sigma_{1}),(\mu_{2},\sigma_{2}))$ refers to a Mixture of two gaussians with mean and variances $\mu_{i}$ and $\sigma_{i}$. We learn each of the parameters $(\theta,\sigma_{0},\mu_{1},\sigma_{1},\mu_{2},\sigma_{2})$ by maximizing the likelihood.

Both GAN and GEBM have the capacity to recover the the exact support by finding the optimal parameter $\theta^{\star}$. For the EBM, when $\theta=\theta^{\star}$, the mean $G_{\theta^{\star}}(X_{1})$ of the conditional gaussian $X_{2}|X_{1}$ draws a line which matches the data support exactly, i.e.: $X_{2}=G_{\theta^{\star}}^{(2)}(X_{1})$.

G.4 Base/Generator complexity

To investigate the effect of model complexity of the performance gap between GANs and GEBMs, we performed additional experiments using the setting of Figure 1. Now we allow the generator/base network to better model the hidden transformation $h$ that produces the first coordinate $X_{1}$ given the latent noise. We choose $G_{\theta}^{(1)}$ to be either a one hidden layer network or an MLP with 3 hidden layers both with leaky ReLU activation, instead of a simple linear transform as previously done in Section G.3. The network has universal approximation capability that depends on the number of units. This provides a direct control over the complexity of the generator/base. We then varied the number of hidden units from $1$ to $5*10^{4}$ units for the one hidden layer network and from $10$ to $5*10^{3}$ units per layer for the MLP. Note that the MLP with $5*10^{3}$ units per layer stores a matrix of size $2.5*10^{7}$ and thus contains 2 orders of magnitudes more parameters than the widest shallow network with $5*10^{4}$ units. We then compared the performance of the GAN and GEBM using the Sinkhorn divergence Feydy et al., (2019) between each model and the data distribution. In all experiments, we used the same discriminator/energy network described in Section G.3. Results are provided in Figure 8.

The Sinkhorn is computed using $6000$ samples from the data and the model, with squared euclidean distance as a ground cost and using a regularization $\epsilon=1e-3$. We then repeat the procedure $5$ times and average the result to get the final estimate of the Sinkhorn distance for a given run.

Each run optimizes the parameters of the model using Adam optimizer $(\beta_{1}=.5,\beta_{2}=.99)$, learning rate $lr=1e-4$ for the energy/discriminator and $lr=1e-5$ for the base/generator and weight decay of $1e-2$ for the base/generator. Training is performed using KALE for $2000$ epochs using a batch size of $5000$ and $10$ gradient iterations for the energy/discriminator per base/generator iteration. We use the gradient penalty for the energy/discriminator with a penalty parameter of $0.01$. We then perform early stopping and retain the best performing model on a validation set.

We make the following observations from Figure 8: the GAN generator indeed improves when we increase the number of hidden units. The performance of the GEBM remains stable as the number of hidden units increases. The performance of the GEBM is always better than the GAN, although we can see the GAN converging towards the GEBM. GEBM with a simpler base already outperforms the GAN with more powerful generators. The gap between the GEBM and the GAN reduces as the GAN becomes more expressive. Using a deeper network further reduces the gap compared to a shallow network.

These observations support the prior discussion that the energy witnesses a remaining difference between the generator and training samples, as long as it is not flat. This information allows the GEBM to perform better than a GAN that ignores it. The performance gap between the GEBM and the GAN reduces as the generator becomes more powerful and forces the energy to be more flat. This is consistent with the result in Proposition 3.