From optimal transport to generative modeling: the VEGAN cookbook

Introduction

The field of representation learning was initially driven by supervised approaches, with impressive results using large labelled datasets. Unsupervised generative modeling, in contrast, used to be a domain governed by probabilistic approaches focusing on low-dimensional data. Recent years have seen a convergence of those two approaches. In the new field that formed at the intersection, variational autoencoders (VAEs) form one well-established approach, theoretically elegant yet with the drawback that they tend to generate blurry images. In contrast, generative adversarial networks (GANs) turned out to be more impressive in terms of the visual quality of images sampled from the model, but have been reported harder to train. There has been a flurry of activity in assaying numerous configurations of GANs as well as combinations of VAEs and GANs. A unifying theory relating GANs to VAEs in a principled way is yet to be discovered. This forms a major motivation for the studies underlying the present paper.

Following , we approach generative modeling from the optimal transport point of view. The optimal transport (OT) cost is a way to measure a distance between probability distributions and provides a much weaker topology than many others, including $f$-divergences associated with the original GAN algorithms. This is particularly important in applications, where data is usually supported on low dimensional manifolds in the input space $\mathcal{X}$. As a result, stronger notions of distances (such as $f$-divergences, which capture the density ratio between distributions) often max out, providing no useful gradients for training. In contrast, the optimal transport behave nicer and may lead to a more stable training .

In this work we aim at minimizing the optimal transport cost $W_{c}(P_{X},P_{G})$ between the true (but unknown) data distribution $P_{X}$ and a latent variable model $P_{G}$. We do so via the primal form and investigate theoretical properties of this optimization problem. Our main contributions are listed below; cf. also Figure 1.

We derive an equivalent formulation for the primal form of $W_{c}(P_{X},P_{G})$, which makes the role of latent space $\mathcal{Z}$ and probabilistic encoders $Q(Z|X)$ explicit (Theorem 1).

Unlike in VAE, we arrive at an optimization problem where the $Q$ are constrained. We relax the constraints by penalization, arriving at the penalized optimal transport (POT) objective (13) which can be minimized with stochastic gradient descent by sampling from $P_{X}$ and $P_{G}$.

We show that for squared Euclidean cost $c$ (Section 4.1), POT coincides with the objective of adversarial auto-encoders (AAE) . We believe this provides the first theoretical justification for AAE, showing that they approximately minimize the 2-Wasserstein distance $W_{2}(P_{X},P_{G})$. We also compare POT to VAE, adversarial variational Bayes (AVB) , and other methods based on the marginal log-likelihood. In particular, we show that all these methods necessarily suffer from blurry outputs, unlike POT or AAE.

When $c$ is the Euclidean distance (Section 4.2), POT and WGAN both minimize the 1-Wasserstein distance $W_{1}(P_{X},P_{G})$. They approach this problem from primal/dual forms respectively, which leads to a different behaviour of the resulting algorithms.

Finally, following a somewhat well-established tradition of using acronyms based on the “GAN” suffix, and because we give a recipe for blending VAE and GAN (using POT), we propose to call this work a VEGAN cookbook.

The authors of address computing the OT cost in large scale using stochastic gradient descent (SGD) and sampling. They approach this task either through the dual formulation, or via a regularized version of the primal. They do not discuss any implications for generative modeling. Our approach is based on the primal form of OT, we arrive at regularizers which are very different, and our main focus is on generative modeling. The Wasserstein GAN minimizes the 1-Wasserstein distance $W_{1}(P_{X},P_{G})$ for generative modeling. The authors approach this task from the dual form. Unfortunately, their algorithm cannot be readily applied to any other OT cost, because the famous Kantorovich duality holds only for $W_{1}$. In contrast, our algorithm POT approaches the same problem from the primal form and can be applied for any cost function $c$.

The present paper is structured as follows. In Section 2 we introduce our notation and discuss existing generative modeling techniques, including GANs, VAEs, and AAE. Section 3 contains our main results, including a theoretical analysis of the primal form of OT and the novel POT objective. Section 4 discusses the implications of our new results. Finally, we discuss future work in Section 5. Proofs may be found in Appendix B.

Notations and preliminaries

We use calligraphic letters (i.e. $\mathcal{X}$) for sets, capital letters (i.e. $X$) for random variables, and lower case letters (i.e. $x$) for their values. We denote probability distributions with capital letters (i.e. $P(X)$) and densities with lower case letters (i.e. $p(x)$). By $\delta_{x}$ we denote the Dirac distribution putting mass 1 on $x\in\mathcal{X}$, and $\mathbf{supp}\,P$ denotes the support of $P$.

where $c(x,y)\colon\mathcal{X}\times\mathcal{X}\to\mathcal{R}_{+}$ is any measurable cost function and $\mathcal{P}(X\sim P,Y\sim Q)$ is a set of all joint distributions of $(X,Y)$ with marginals $P$ and $Q$ respectively. A particularly interesting case is when $(\mathcal{X},d)$ is a metric space and $c(x,y)=d^{p}(x,y)$ for $p\geq 1$. In this case $W_{p}$, the $p$-th root of $W_{c}$, is called the $p$-Wasserstein distance. Finally, the Kantorovich-Rubinstein theorem establishes a duality for the 1-Wasserstein distance, which holds under mild assumptions on $P$ and $Q$:

where $\mathcal{F}_{L}$ is the class of all bounded 1-Lipschitz functions on $(\mathcal{X},d)$. Note that the same symbol is used for $W_{p}$ and $W_{c}$, but only $p$ is a number and thus the above $W_{1}$ refers to the Wasserstein distance.

Even though GANs and VAEs are quite different—both in terms of the conceptual frameworks and empirical performance—they share important features: (a) both can be trained by sampling from the model $P_{G}$ without knowing an analytical form of its density and (b) both can be scaled up with SGD. As a result, it becomes possible to use highly flexible implicit models $P_{G}$ defined by a two-step procedure, where first a code $Z$ is sampled from a fixed distribution $P_{Z}$ on a latent space $\mathcal{Z}$ and then $Z$ is mapped to the image $G(Z)\in\mathcal{X}=\mathcal{R}^{d}$ with a (possibly random) transformation $G\colon\mathcal{Z}\to\mathcal{X}$. This results in latent variable models $P_{G}$ defined on $\mathcal{X}$ with density of the form

assuming all involved densities are properly defined. These models are indeed easy to sample and, provided $G$ can be differentiated analytically with respect to its parameters, $P_{G}$ can be trained with SGD. The field is growing rapidly and numerous variations of VAEs and GANs are available in the literature. Next we introduce and compare several of them.

The original generative adversarial network (GAN) approach minimizes

Variational auto-encoders (VAE) utilize models $P_{G}$ of the form (3) and minimize

Minimizing the primal of optimal transport

We have argued that minimizing the optimal transport cost $W_{c}(P_{X},P_{G})$ between the true data distribution $P_{X}$ and the model $P_{G}$ is a reasonable goal for generative modeling. We now will explain how this can be done in the primal formulation of the OT problem (1) by reparametrizing the space of couplings (Section 3.1) and relaxing the marginal constraint (Section 3.2), leading to a formulation involving expectations over $P_{X}$ and $P_{G}$ that can thus be solved using SGD and sampling.

We will consider certain sets of joint probability distributions of three random variables $(X,Y,Z)\in\mathcal{X}\times\mathcal{X}\times\mathcal{Z}$. The reader may wish to think of $X$ as true images, $Y$ as images sampled from the model, and $Z$ as latent codes. We denote by $P_{G,Z}(Y,Z)$ a joint distribution of a variable pair $(Y,Z)$, where $Z$ is first sampled from $P_{Z}$ and next $Y$ from $P_{G}(Y|Z)$. Note that $P_{G}$ defined in (3) and used throughout this work is the marginal distribution of $Y$ when $(Y,Z)\sim P_{G,Z}$.

In the optimal transport problem, we consider joint distributions $\Gamma(X,Y)$ which are called couplings between values of $X$ and $Y$. Because of the marginal constraint, we can write $\Gamma(X,Y)=\Gamma(Y|X)P_{X}(X)$ and we can consider $\Gamma(Y|X)$ as a non-deterministic mapping from $X$ to $Y$. In this section we will show how to factor this mapping through $\mathcal{Z}$, i.e., decompose it into an encoding distribution $Q(Z|X)$ and the generating distribution $P_{G}(Y|Z)$.

In order to give a more intuitive explanation of this decomposition, consider the case where all probability distributions have densities with respect to the Lebesgue measure. In this case our results show that some elements $\Gamma$ of $\mathcal{P}(X\sim P_{X},Y\sim P_{G})$ have densities of the form

where the density of the conditional distribution $Q(Z|X)$ satisfies $q_{Z}(z):=\int_{\mathcal{X}}q(z|x)p_{X}(x)dx=p_{Z}(z)$ for all $z\in\mathcal{Z}$. Equation (8) allows to express the search space (couplings) of the OT problem in terms of the probabilistic encoders $Q(Z|X)$. Unlike VAE, AVB, and other methods based on the marginal log-likelihood, where the $Q$ are not constrained, the encoders of the OT problem need to match the aggregated posterior $Q_{Z}$ to the prior $P_{Z}$.

As in Section 2, $\mathcal{P}(X\sim P_{X},Y\sim P_{G})$ denotes the set of all joint distributions of $(X,Y)$ with marginals $P_{X},P_{G}$, and likewise for $\mathcal{P}(X\sim P_{X},Z\sim P_{Z})$. The set of all joint distributions of $(X,Y,Z)$ such that $X\sim P_{X}$, $(Y,Z)\sim P_{G,Z}$, and (Y\mathchoice{\mathrel{\hbox to0.0pt{\displaystyle\perp\hss}\mkern 2.0mu{\displaystyle\perp}}}{\mathrel{\hbox to0.0pt{\textstyle\perp\hss}\mkern 2.0mu{\textstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptstyle\perp\hss}\mkern 2.0mu{\scriptstyle\perp}}}{\mathrel{\hbox to0.0pt{\scriptscriptstyle\perp\hss}\mkern 2.0mu{\scriptscriptstyle\perp}}}X)|Z will be denoted by $\mathcal{P}_{X,Y,Z}$. Finally, we denote by $\mathcal{P}_{X,Y}$ and $\mathcal{P}_{X,Z}$ the sets of marginals on $(X,Y)$ and $(X,Z)$ (respectively) induced by distributions in $\mathcal{P}_{X,Y,Z}$. Note that $\mathcal{P}(P_{X},P_{G})$, $\mathcal{P}_{X,Y,Z}$, and $\mathcal{P}_{X,Y}$ depend on the choice of conditional distributions $P_{G}(Y|Z)$, while $\mathcal{P}_{X,Z}$ does not. In fact, it is easy to check that $\mathcal{P}_{X,Z}=\mathcal{P}(X\sim P_{X},Z\sim P_{Z})$. From the definitions it is clear that $\mathcal{P}_{X,Y}\subseteq\mathcal{P}(P_{X},P_{G})$ and we get the following upper bound:

If $P_{G}(Y|Z)$ are Dirac measures (i.e., $Y=G(Z)$), the two sets are actually coincide, thus justifying the reparametrization (8) and the illustration in Figure 1(b), as demonstrated in the following theorem:

If $P_{G}(Y|Z=z)=\delta_{G(z)}$ for all $z\in\mathcal{Z}$, where $G\colon\mathcal{Z}\to\mathcal{X}$, we have

where $Q_{Z}$ is the marginal distribution of $Z$ when $X\sim P_{X}$ and $Z\sim Q(Z|X)$.

The r.h.s. of (10) is the optimal transport $W_{c_{G}}(P_{X},P_{Z})$ between $P_{X}$ and $P_{Z}$ with the cost function c_{g}(x,z):=c\bigl{(}x,G(z)\bigr{)} defined on $\mathcal{X}\times\mathcal{Z}$. If $P_{G}(Y|Z)$ corresponds to a deterministic mapping $G\colon\mathcal{Z}\to\mathcal{X}$, the resulting model $P_{G}$ is the push-forward of $P_{Z}$ through $G$. Theorem 1 states that in this case the two OT problems are equivalent, $W_{c}(P_{X},P_{G})=W_{c_{G}}(P_{X},P_{Z})$.

The conditional distributions $Q$ in (11) are constrained to ensure that when $X$ is sampled from $P_{X}$ and then $Z$ from $Q(Z|X)$, the resulting marginal distribution of $Z$ (which was denoted $Q_{Z}$ and called aggregated posterior in Section 2.1) coincides with $P_{Z}$. A very similar result also holds for the case when $P_{G}(Y|Z)$ are not necessarily Dirac. Nevertheless, for simplicity we will focus on the Dirac case for now and shortly summarize the more general case in the end of this section.

2 Relaxing the constraints

Minimizing $W_{c}(P_{X},P_{G})$ boils down to a min-min optimization problem. Unfortunately, the constraint on $Q$ makes the variational problem even harder. We thus propose to replace the constrained optimization problem (11) with its relaxed unconstrained version in a standard way. Namely, use any convex penalty $F\colon Q\to\mathcal{R}_{+}$, such that $F(Q)=0$ if and only if $P_{Z}=Q_{Z}$, and for any $\lambda>0$, consider the following relaxed unconstrained version of $W_{c}^{\dagger}(P_{X},P_{G})$:

It is well known that under mild conditions adding a penalty as in (12) is equivalent to adding a constraint of the form $F(Q)\leq\mu_{\lambda}$ for some $\mu_{\lambda}>0$. As $\lambda$ increases, the corresponding $\mu_{\lambda}$ decreases, and as $\lambda\to\infty$, the solutions of (12) reach the feasible region where $P_{Z}=Q_{Z}$. This shows that $W_{c}^{\lambda}(P_{X},P_{G})\leq W_{c}(P_{X},P_{G})$ for all $\lambda\geq 0$ and the gap reduces with increasing $\lambda$.

The above discussion assumed Dirac measures $P_{G}(Y|Z)$. If this is not the case, we can still upper bound the 2-Wasserstein distance $W_{2}(P_{X},P_{G})$, corresponding to $c(x,y)=\|x-y\|^{2}$, in a very similar way to Theorem 1. The case of Gaussian decoders $P_{G}(Y|Z)$, which will be particularly useful when discussing the relation to VAEs, is summarized in the following remark:

Implications: relations to AAE, VAEs, and GANs

Thus far we showed that the primal form of the OT problem can be relaxed to allow for efficient minimization by SGD. We now discuss the main implications of these results.

Consider the squared Euclidean cost function $c(x,y)=\|x-y\|^{2}$, for which $W_{c}$ is the squared 2-Wasserstein distance $W_{2}^{2}$. The goal of this section is to compare the minimization of $W_{2}(P_{X},P_{G})$ to other generative modeling approaches. Let us focus our attention on generative distributions $P_{G}(Y|Z)$ typically used in VAE, AVB, and AAE, i.e., Gaussians P_{G}(Y|Z)=\mathcal{N}\bigl{(}Y;G(Z),\sigma^{2}\!\cdot\!I_{d}\bigr{)}. In order to verify the differentiability of $\log p_{G}(x|z)$ all three methods require $\sigma^{2}>0$ and have problems handling the case of deterministic decoders ($\sigma^{2}=0$). To emphasize the role of the variance $\sigma^{2}$ we will denote the resulting latent variable model $P_{G}^{\sigma}$.

The analysis of Section 3 shows that the value of $W_{2}(P_{X},P_{G}^{\sigma})$ is upper bounded by $W_{c}^{\dagger}(P_{X},P_{G}^{\sigma})$ of the form (16) and the two coincide when $\sigma^{2}=0$. Next we summarize properties of solutions $G$ minimizing these two values $W_{2}$ and $W_{c}^{\dagger}$:

Let $\mathcal{X}=\mathcal{R}^{d}$ and assume $c(x,y)=\|x-y\|^{2}$, P_{G}(Y|Z)=\mathcal{N}\bigl{(}Y;G(Z),\sigma^{2}\!\cdot\!I\bigr{)} with any function $G\colon\mathcal{X}\to\mathcal{R}$. If $\sigma^{2}>0$ then the functions $G^{*}_{\sigma}$ and $G^{\dagger}$ minimizing $W_{c}(P_{X},P_{G}^{\sigma})$ and $W_{c}^{\dagger}(P_{X},P_{G}^{\sigma})$ respectively are different: $G^{*}_{\sigma}$ depends on $\sigma^{2}$, while $G^{\dagger}$ does not. The function $G^{\dagger}$ is also a minimizer of $W_{c}(P_{X},P_{G}^{0})$.

In contrast, Proposition 2 shows that for the same Gaussian models with any given $\sigma^{2}\geq 0$ we can minimize $W_{c}^{\dagger}(P_{X},P_{G}^{\sigma})$ and the solution $G^{\dagger}$ will be indeed the one resulting in the smallest 2-Wasserstein distance between $P_{X}$ and the noiseless implicit model $G(Z)$, $Z\sim P_{Z}$ used in practice.

Relation to AAE

The authors of tried to establish a connection between AAE and log-likelihood maximization. They argued that AAE is “a crude approximation” to AVB. Our results suggest that AAE is in fact attempting to minimize the 2-Wasserstein distance between $P_{X}$ and $P_{G}^{\sigma}$, which may explain its good empirical performance reported in .

Blurriness of VAE and AVB

We next add to the discussion regarding the blurriness commonly attributed to VAE samples. Our argument shows that VAE, AVB, and other methods based on the marginal log-likelihood necessarily lead to an averaging in the input space if $P_{G}(Y|Z)$ are Gaussian.

This overlap necessarily happens in VAEs, which use Gaussian encoders $Q(Z|X)$ supported on the entire $\mathcal{Z}$. When probabilistic encoders $Q$ are allowed to be flexible enough, as in AVB, for any fixed $P_{G}(Y|Z)$ the optimal $Q^{*}$ will try to invert the decoder (see Appendix A) and take the form

This approximation becomes exact in the nonparametric limit of $Q$. When $P_{G}(Y|Z)$ is Gaussian we have $p_{G}(y|z)>0$ for all $y\in\mathcal{X}$ and $z\in\mathcal{Z}$, showing that $\mathbf{supp}\,Q^{*}(Z|X=x)=\mathbf{supp}\,P_{Z}$ for all $x\in\mathcal{X}$. This will again lead to the overlap of encoders if $\mathbf{supp}\,P_{Z}=\mathcal{Z}$. In contrast, the optimal encoders of AAE and POT do not necessarily overlap, as they are not inverting the decoders.

2 The 1-Wasserstein distance: relation to WGAN

This means we can now approach the problem of optimizing $W_{1}$ in two distinct ways, taking gradient steps either in the primal or in the dual forms. Denote by $Q^{*}$ the optimal encoder in the primal and $f^{*}$ the optimal witness function in the dual. By the envelope theorem, gradients of $W_{1}$ with respect to $G$ can be computed by taking a gradient of the criteria evaluated at the optimal points $Q^{*}$ or $f^{*}$.

Despite the theoretical equivalence of both approaches, practical considerations lead to different behaviours and to potentially poor approximations of the real gradients. For example, in the dual formulation, one usually restricts the witness functions to be smooth, while in the primal formulation, the constraint on $Q$ is only approximately enforced. We will study the effect of these approximations.

There exists a constant $C>0$ such that for any $\epsilon>0$, one can construct distributions $P_{X}$, $P_{G}$ and pick witness functions $f_{\epsilon}\in\mathcal{F}_{L}$ and $h_{\epsilon}\in\mathcal{H}$ that are $\epsilon$-optimal $|J_{D}(f_{\epsilon})-J_{D}(f^{*})|\leq\epsilon$, $|J_{D}(h_{\epsilon})-J_{D}(h^{*})|\leq\epsilon$, but which give (at some point $z\in\mathcal{Z}$) gradients whose direction is at least $C$-wrong: $A(f_{\epsilon},f^{*})\leq 1-C$, $A(h_{0},h^{*})\leq 1-C$, and $A(h_{\epsilon},h^{*})\leq 0$.

Imperfect posterior in the primal (i.e., for POT)

In the primal formulation, when the constraint is violated, that is the aggregated posterior $Q_{Z}$ is not matching $P_{Z}$, there can be two kinds of negative effects: (i) the gradient of the criterion is only computed on a (possibly small) subset of the latent space reached by $Q_{Z}$; (ii) several input points could be mapped by $Q(Z|X)$ to the same latent code $z$, thus giving gradients that encourage $G(z)$ to be the average/median of several inputs (hence encouraging a blurriness). See Section 4.1 for the details and Figure 1 for an illustration.

Conclusion

This work proposes a way to fit generative models by minimizing any optimal transport cost. It also establishes novel links between different popular unsupervised probabilistic modeling techniques. Whilst our contribution is on the theoretical side, it is reassuring to note that the empirical results of show the strong performance of our method for the special case of the 2-Wasserstein distance. Experiments with other cost functions $c$ are beyond the scope of the present work and left for the future studies.

Acknowledgments

The authors are thankful to Mateo Rojas-Carulla and Fei Sha for stimulating discussions. CJSG is supported by a Google European Doctoral Fellowship in Causal Inference.

References

Appendix A Further details on VAEs and GANs

For models $P_{G}$ of the form (3) and any conditional distribution $Q(Z|X)$ it can be easily verified that

Here the conditional distribution $P_{G}(Z|X)$ is induced by a joint distribution $P_{G,Z}(X,Z)$, which is in turn specified by the 2-step latent variable procedure: (a) sample $Z$ from $P_{Z}$, (b) sample $X$ from $P_{G}(X|Z)$. Note that the first term on the r.h.s. of (15) is always non-positive, while the l.h.s. does not depend on $Q$. This shows that if conditional distributions $Q$ are not restricted then

where the infimum is achieved for $Q(Z|X)=P_{G}(Z|X)$. However, for any restricted class $\mathcal{Q}$ of conditional distributions $Q(Z|X)$ we only have

where the inequality accounts for the fact that $Q(Z|X)$ might be not flexible enough to match $P(Z|X)$ for all values of $X$.

Relation between AAE, AVB, and VAE

For any distributions $P_{X}$ and $P_{G}$:

Appendix B Proofs

We start by introducing an important lemma relating the two sets over which $W_{c}$ and $W_{c}^{\dagger}$ are optimized.

$\mathcal{P}_{X,Y}\subseteq\mathcal{P}(P_{X},P_{G})$ with identity if $P_{G}(Y|Z=z)$ are Dirac distributions for all $z\in\mathcal{Z}$We conjecture that this is also a necessary condition. The necessity is not used in the remainder of the paper..

Inequality (9) and the first identity in (10) obviously follows from Lemma 6. The tower rule of expectation, and the conditional independence property of $\mathcal{P}_{X,Y,Z}$ implies

Let $\mathcal{X}=\mathcal{R}^{d}$ and assume the conditional distributions $P_{G}(Y|Z=z)$ have mean values $G(z)\in\mathcal{R}^{d}$ and marginal variances $\sigma_{1}^{2},\dots,\sigma_{d}^{2}\geq 0$ for all $z\in\mathcal{Z}$, where $G\colon\mathcal{Z}\to\mathcal{X}$. Take $c(x,y)=\|x-y\|^{2}_{2}$. Then

Proof is similar to the one of Theorem 1. See Section B.2. ∎

First inequality follows from (9). For the identity we proceed similarly to the proof of Theorem 1 and write

Together with (17) and the fact that $\mathcal{P}_{X,Z}=\mathcal{P}(X\sim P_{X},Z\sim P_{Z})$ this concludes the proof.

B.3 Proof of Proposition 2

Corollary 7 shows that $G^{\dagger}$ does not depend on the variance $\sigma^{2}$. When $\sigma^{2}=0$ the distribution $P_{G}(Y|Z)$ turns into Dirac. In this case we combine Theorem 1 and Corollary 7 to conclude that $G^{\dagger}$ also minimizes $W_{c}(P_{X},P_{G}^{0})$. Next we prove that $G^{*}_{\sigma}$ generally depends on $\sigma^{2}$.

We will need the following simple result, which is basically saying that the variance of a sum of two independent random variables is a sum of the variances:

Under conditions of Proposition 2, assume $Y\sim P_{G}^{\sigma}$. Then

Next we prove the remaining implication of Proposition 2. Namely, that when $\sigma^{2}>0$ the function $G^{*}_{\sigma}$ minimizing $W_{c}(P_{X},P_{G}^{\sigma})$ depends on $\sigma^{2}$. The proof is based on the following example: $\mathcal{X}=\mathcal{Z}=\mathcal{R}$, $P_{X}=\mathcal{N}(0,1)$, $P_{Z}=\mathcal{N}(0,1)$, and $0<\sigma^{2}<1$. Note that by setting $G(z)=c\cdot z$ for any $c>0$ we ensure that $P_{G}^{\sigma}$ is the Gaussian distribution, because a convolution of two Gaussians is also Gaussian. In particular if we take $G^{*}(z)=\sqrt{1-\sigma^{2}}\cdot z$ Lemma 8 implies that $P_{G^{*}}^{\sigma}$ is the standard normal Gaussian $\mathcal{N}(0,1)$. In other words, we obtain the global minimum $W_{c}(P_{X},P_{G^{*}}^{\sigma})=0$ and $G^{*}$ clearly depends on $\sigma^{2}$.

B.4 Proof of Proposition 3

We just give a sketch of the proof: consider discrete distributions $P_{X}$ supported on two points $\{x_{0},x_{1}\}$, and $P_{Z}$ supported on $\{0,1\}$ and let $y_{0}=G(0)$, $y_{1}=G(1)$ ($y_{0}\neq y_{1}$). Given an optimal $f^{*}$, one can modify locally it around $y_{0}$ without changing its Lipschitz constant such that the obtained $f_{\epsilon}$ is an $\epsilon$-approximation of $f^{*}$ whose gradients at $y_{0}$ and $y_{1}$ point in directions arbitrarily different from those of $f^{*}$. For smooth functions, by moving $y_{0}$ and $y_{1}$ away from the segment $[x_{0},x_{1}]$ but close to each other $\|y_{0}-y_{1}\|\leq K\epsilon$, the gradients of $f^{*}$ will point in directions roughly opposite but the constraint on the Hessian will force the gradients of $f_{\mathcal{F},0}$ at $y_{0}$ and $y_{1}$ to be very close. Finally, putting $y_{0},y_{1}$ on the segment $[x_{0},x_{1}]$, one can get an $f_{\mathcal{F}}^{*}$ whose gradients at $y_{0}$ and $y_{1}$ are exactly opposite, while taking $f_{\mathcal{F},\epsilon}(y)=f_{\mathcal{F}}^{*}(y+\epsilon)$, we can swap the direction at one of the points while changing the criterion by less than $\epsilon$.