Fisher GAN

Introduction

Generative Adversarial Networks (GANs) have recently become a prominent method to learn high-dimensional probability distributions. The basic framework consists of a generator neural network which learns to generate samples which approximate the distribution, while the discriminator measures the distance between the real data distribution, and this learned distribution that is referred to as fake distribution. The generator uses the gradients from the discriminator to minimize the distance with the real data distribution. The distance between these distributions was the object of study in , and highlighted the impact of the distance choice on the stability of the optimization. The original GAN formulation optimizes the Jensen-Shannon divergence, while later work generalized this to optimize f-divergences , KL , the Least Squares objective . Closely related to our work, Wasserstein GAN (WGAN) uses the earth mover distance, for which the discriminator function class needs to be constrained to be Lipschitz. To impose this Lipschitz constraint, WGAN proposes to use weight clipping, i.e. a data independent constraint, but this comes at the cost of reducing the capacity of the critic and high sensitivity to the choice of the clipping hyper-parameter. A recent development Improved Wasserstein GAN (WGAN-GP) introduced a data dependent constraint namely a gradient penalty to enforce the Lipschitz constraint on the critic, which does not compromise the capacity of the critic but comes at a high computational cost.

We build in this work on the Integral probability Metrics (IPM) framework for learning GAN of . Intuitively the IPM defines a critic function $f$, that maximally discriminates between the real and fake distributions. We propose a theoretically sound and time efficient data dependent constraint on the critic of Wasserstein GAN, that allows a stable training of GAN and does not compromise the capacity of the critic. Where WGAN-GP uses a penalty on the gradients of the critic, Fisher GAN imposes a constraint on the second order moments of the critic. This extension to the IPM framework is inspired by the Fisher Discriminant Analysis method.

We introduce in Section 2 the Fisher IPM, a scaling invariant distance between distributions. Fisher IPM introduces a data dependent constraint on the second order moments of the critic that discriminates between the two distributions. Such a constraint ensures the boundedness of the metric and the critic. We show in Section 2.2 that Fisher IPM when approximated with neural networks, corresponds to a discrepancy between whitened mean feature embeddings of the distributions. In other words a mean feature discrepancy that is measured with a Mahalanobis distance in the space computed by the neural network.

We show in Section 3 that Fisher IPM corresponds to the Chi-squared distance ($\chi_{2}$) when the critic has unlimited capacity (the critic belongs to a universal hypothesis function class). Moreover we prove in Theorem 2 that even when the critic is parametrized by a neural network, it approximates the $\chi_{2}$ distance with a factor which is a inner product between optimal and neural network critic. We finally derive generalization bounds of the learned critic from samples from the two distributions, assessing the statistical error and its convergence to the Chi-squared distance from finite sample size.

We use Fisher IPM as a GAN objective Code is available at https://github.com/tomsercu/FisherGAN and formulate an algorithm that combines desirable properties (Table 1): a stable and meaningful loss between distributions for GAN as in Wasserstein GAN , at a low computational cost similar to simple weight clipping, while not compromising the capacity of the critic via a data dependent constraint but at a much lower computational cost than . Fisher GAN achieves strong semi-supervised learning results without need of batch normalization in the critic.

Learning GANs with Fisher IPM

It is easy to see that $d_{\mathscr{F}}$ defines a pseudo-metric over $\mathscr{P}(\pazocal{X})$. Note specifically that if $\mathscr{F}$ is not bounded, $\sup_{f}$ will scale $f$ to be arbitrarily large. By choosing $\mathscr{F}$ appropriately , various distances between probability measures can be defined.

While a standard IPM (Equation (1)) maximizes the discrepancy between the means of a function under two different distributions, Fisher IPM looks for critic $f$ that achieves a tradeoff between maximizing the discrepancy between the means under the two distributions (between class variance), and reducing the pooled second order moment (an upper bound on the intra-class variance).

Standardized discrepancies have a long history in statistics and the so-called two-samples hypothesis testing. For example the classic two samples Student’s $t-$ test defines the student statistics as the ratio between means discrepancy and the sum of standard deviations. It is now well established that learning generative models has its roots in the two-samples hypothesis testing problem . Non parametric two samples testing and model criticism from the kernel literature lead to the so called maximum kernel mean discrepancy (MMD) . The MMD cost function and the mean matching IPM for a general function space has been recently used for training GAN .

Second formulation: Constrained form. Since the distance is scaling invariant, $d_{\mathscr{F}}$ can be written equivalently in the following constrained form:

where $\hat{\Omega}(f_{p},g_{\theta})=\frac{1}{2N}\sum_{i=1}^{N}f^{2}_{p}(x_{i})+\frac{1}{2M}\sum_{j=1}^{M}f^{2}_{p}(g_{\theta}(z_{j}))$. For simplicity we will have $M=N$.

2 Fisher IPM with Neural Networks

We will specifically study the case where $\mathscr{F}$ is a finite dimensional Hilbert space induced by a neural network $\Phi_{\omega}$ (see Figure 1 for an illustration). In this case, an IPM with data-independent constraint will be equivalent to mean matching . We will now show that Fisher IPM will give rise to a whitened mean matching interpretation, or equivalently to mean matching with a Mahalanobis distance.

Rayleigh Quotient. Consider the function space $\mathscr{F}_{v,\omega}$, defined as follows

$\Phi_{\omega}$ is typically parametrized with a multi-layer neural network. We define the mean and covariance (Gramian) feature embedding of a distribution as in McGan :

Fisher IPM as defined in Equation (2) on $\mathscr{F}_{v,\omega}$ can be written as follows:

where we added a regularization term ($\gamma>0$) to avoid singularity of the covariances. Note that if $\Phi_{\omega}$ was implemented with homogeneous non linearities such as RELU, if we swap $(v,\omega)$ with $(cv,c^{\prime}\omega)$ for any constants $c,c^{\prime}>0$, the distance $d_{\mathscr{F}_{v,\omega}}$ remains unchanged, hence the scaling invariance.

Constrained Form. Since the Rayleigh Quotient is not amenable to optimization, we will consider Fisher IPM as a constrained optimization problem. By virtue of the scaling invariance and the constrained form of the Fisher IPM given in Equation (3), $d_{\mathscr{F}_{v,\omega}}$ can be written equivalently as:

hence we see that fisher IPM corresponds to the worst case distance between whitened means. Since the means are white, we don’t need to impose further constraints on $\omega$ as in . Another interpretation of the Fisher IPM stems from the fact that:

Learning GAN with Fisher IPM. Hence we see that learning GAN with Fisher IPM:

corresponds to a min-max game between a feature space and a generator. The feature space tries to maximize the Mahalanobis distance between the feature means embeddings of real and fake distributions. The generator tries to minimize the mean embedding distance.

Theory

We note here that LSGAN at full capacity corresponds to a Chi-Squared divergence, with the main difference that LSGAN has different objectives for the generator and the discriminator (bilevel optimizaton), and hence does not optimize a single objective that is a distance between distributions. The Chi-squared divergence can also be achieved in the $f$-gan framework from . We discuss the advantages of the Fisher formulation in Appendix C.

Equivalently we have following relative approximation error:

Fisher GAN Algorithm using ALM

For any choice of the parametric function class $\mathscr{F}_{p}$ (for example $\mathscr{F}_{v,\omega}$), note the constraint in Equation (4) by $\hat{\Omega}(f_{p},g_{\theta})=\frac{1}{2N}\sum_{i=1}^{N}f^{2}_{p}(x_{i})+\frac{1}{2N}\sum_{j=1}^{N}f^{2}_{p}(g_{\theta}(z_{j})).$ Define the Augmented Lagrangian corresponding to Fisher GAN objective and constraint given in Equation (4):

where $\lambda$ is the Lagrange multiplier and $\rho>0$ is the quadratic penalty weight. We alternate between optimizing the critic and the generator. Similarly to we impose the constraint when training the critic only. Given $\theta$, for training the critic we solve $\max_{p}\min_{\lambda}\pazocal{L}_{F}(p,\theta,\lambda).$ Then given the critic parameters $p$ we optimize the generator weights $\theta$ to minimize the objective $\min_{\theta}\hat{\mathscr{E}}(f_{p},g_{\theta}).$ We give in Algorithm 1, an algorithm for Fisher GAN, note that we use ADAM for optimizing the parameters of the critic and the generator. We use SGD for the Lagrange multiplier with learning rate $\rho$ following practices in Augmented Lagrangian .

Experiments

We experimentally validate the proposed Fisher GAN. We claim three main results: (1) stable training with a meaningful and stable loss going down as training progresses and correlating with sample quality, similar to . (2) very fast convergence to good sample quality as measured by inception score. (3) competitive semi-supervised learning performance, on par with literature baselines, without requiring normalization of the critic.

We report results on three benchmark datasets: CIFAR-10 , LSUN and CelebA . We parametrize the generator $g_{\theta}$ and critic $f$ with convolutional neural networks following the model design from DCGAN . For $64\times 64$ images (LSUN, CelebA) we use the model architecture in Appendix F.2, for CIFAR-10 we train at a $32\times 32$ resolution using architecture in F.3 for experiments regarding sample quality (inception score), while for semi-supervised learning we use a better regularized discriminator similar to the Openai and ALI architectures, as given in F.4.We used Adam as optimizer for all our experiments, hyper-parameters given in Appendix F.

Qualitative: Loss stability and sample quality. Figure 3 shows samples and plots during training. For LSUN we use a higher number of D updates ($n_{c}=5$) , since we see similarly to WGAN that the loss shows large fluctuations with lower $n_{c}$ values. For CIFAR-10 and CelebA we use reduced $n_{c}=2$ with no negative impact on loss stability. CIFAR-10 here was trained without any label information. We show both train and validation loss on LSUN and CIFAR-10 showing, as can be expected, no overfitting on the large LSUN dataset and some overfitting on the small CIFAR-10 dataset. To back up our claim that Fisher GAN provides stable training, we trained both a Fisher Gan and WGAN where the batch normalization in the critic $f$ was removed (Figure 4).

Quantitative analysis: Inception Score and Speed. It is agreed upon that evaluating generative models is hard . We follow the literature in using “inception score” as a metric for the quality of CIFAR-10 samples. Figure 5 shows the inception score as a function of number of $g_{\theta}$ updates and wallclock time. All timings are obtained by running on a single K40 GPU on the same cluster. We see from Figure 5, that Fisher GAN both produces better inception scores, and has a clear speed advantage over WGAN-GP.

Quantitative analysis: SSL. One of the main premises of unsupervised learning, is to learn features on a large corpus of unlabeled data in an unsupervised fashion, which are then transferable to other tasks. This provides a proper framework to measure the performance of our algorithm. This leads us to quantify the performance of Fisher GAN by semi-supervised learning (SSL) experiments on CIFAR-10. We do joint supervised and unsupervised training on CIFAR-10, by adding a cross-entropy term to the IPM objective, in conditional and unconditional generation.

Conditional Generation with CE Regularization. We also trained conditional generator models, conditioning the generator on $y$ by concatenating the input noise with a 1-of-K embedding of the label: we now have $g_{\theta}(z,y)$. We parametrize the critic in $\mathscr{F}_{v,\omega}$ and modify the critic objective as above. We also add a cross-entropy term for the generator to minimize during its training step: $\pazocal{L}_{G}=\hat{\mathscr{E}}+\lambda_{G}\sum_{z\sim p(z),y\sim p(y)}CE(g_{\theta}(z,y),y;S,\Phi_{\omega})$. For generator updates we still need to sample only a single minibatch since we use the minibatch of samples from $g_{\theta}(z,y)$ to compute both the IPM loss $\mathscr{\hat{E}}$ and CE. The labels are sampled according to the prior $y\sim p(y)$, which defaults to the discrete uniform prior when there is no class imbalance. We found $\lambda_{D}=\lambda_{G}=0.1$ to be optimal.

New Parametrization of the Critic: “$K+1$ SSL”. One specific successful formulation of SSL in the standard GAN framework was provided in , where the discriminator classifies samples into $K+1$ categories: the $K$ correct clases, and $K+1$ for fake samples. Intuitively this puts the real classes in competition with the fake class. In order to implement this idea in the Fisher framework, we define a new function class of the critic that puts in competition the $K$ class directions of the classifier $S_{y}$, and another “K+1” direction $v$ that indicates fake samples. Hence we propose the following parametrization for the critic: $f(x)=\sum_{y=1}^{K}p(y|x)\left\langle{S_{y}},{\Phi_{\omega}(x)}\right\rangle-\left\langle{v},{\Phi_{\omega}(x)}\right\rangle$, where $p(y|x)=\rm{Softmax}(\left\langle{S},{\Phi_{\omega}(x)}\right\rangle)_{y}$ which is also optimized with Cross-Entropy. Note that this critic does not fall under the interpretation with whitened means from Section 2.2, but does fall under the general Fisher IPM framework from Section 2.1. We can use this critic with both conditional and unconditional generation in the same way as described above. In this setting we found $\lambda_{D}=1.5,\,\lambda_{G}=0.1$ to be optimal.

Layerwise normalization on the critic. For most GAN formulations following DCGAN design principles, batch normalization (BN) in the critic is an essential ingredient. From our semi-supervised learning experiments however, it appears that batch normalization gives substantially worse performance than layer normalization (LN) or even no layerwise normalization. We attribute this to the implicit whitening Fisher GAN provides.

Table 3 shows the SSL results on CIFAR-10. We show that Fisher GAN has competitive results, on par with state of the art literature baselines. When comparing to WGAN with weight clipping, it becomes clear that we recover the lost SSL performance. Results with the $K+1$ critic are better across the board, proving consistently the advantage of our proposed $K+1$ formulation. Conditional generation does not provide gains in the setting with layer normalization or without normalization.

Conclusion

We have defined Fisher GAN, which provide a stable and fast way of training GANs. The Fisher GAN is based on a scale invariant IPM, by constraining the second order moments of the critic. We provide an interpretation as whitened (Mahalanobis) mean feature matching and $\chi_{2}$ distance. We show graceful theoretical and empirical advantages of our proposed Fisher GAN.

The authors thank Steven J. Rennie for many helpful discussions and Martin Arjovsky for helpful clarifications and pointers.

References

Appendix A WGAN-GP versus Fisher GAN

be the Jacobian matrix of the $\Phi_{\omega}(.)$. It is easy to see that

is the so called metric tensor in information geometry (See for instance and references there in). The gradient penalty for WGAN of can be derived from a Rayleigh quotient principle as well, written in the constraint form:

Using the special parametrization we can write:

is the expected Riemannian metric tensor . Hence we obtain:

Hence Gradient penalty can be seen as well as mean matching in the metric defined by the expected metric tensor $\mathscr{M}_{\omega}$.

Improved WGAN IPM can be written as follows :

Both Improved WGAN are doing mean matching using different Mahalanobis distances! While improved WGAN uses an expected metric tensor $\mathscr{M}_{\omega}$ to compute this distance, Fisher IPM uses a simple pooled covariance $\Sigma_{\omega}$ to compute this metric. It is clear that Fisher GAN has a computational advantage!

Appendix B Chi-squared distance and Pearson Divergence

Appendix C Fisher GAN and φ𝜑\varphi-divergence Based GANs

Since $f$-gan also introduces a GAN formulation which recovers the Chi-squared divergence, we compare our approaches.

Let us recall here the definition of $\varphi$-divergence:

As shown in and in , for any function space $\mathscr{F}$ we get the lower bound:

For the particular case $\varphi(t)=(t-1)^{2}$ and $\varphi^{*}(t)=\frac{1}{4}t^{2}+t$ we have the Pearson $\chi_{2}$ divergence:

Hence to optimize the same cost function of Fisher GAN in the $\varphi$-GAN framework we have to consider:

Fisher GAN gives an inequality for the symmetric Chi-squared and the $\varphi$-GAN gives a lower variational bound. i.e compare for $\varphi$-GAN:

while equivalent at the optimum those two formulations for the symmetric Chi-squared given in Equations (10), and (11) have different theoretical and practical properties. On the theory side:

While the formulation in (10) is a $\varphi$ divergence, the formulation given by the Fisher criterium in (11) is an IPM with a data dependent constraint. This is a surprising result because $\varphi$-divergences and IPM exhibit different properties and the only known non trivial $\varphi$ divergence that is also an IPM with data independent function class is the total variation distance . When we allow the function class to be dependent on the distributions, the symmetric Chi-squared divergence (in fact general Chi-squared also) can be cast as an IPM! Hence in the context of GAN training we inherit the known stability of IPM based training for GANs.

Theorem 2 for the Fisher criterium gives us an approximation error when we change the function from the space of measurable functions to a hypothesis class. It is not clear how tight the lower bound in the $\varphi$-divergence will be as we relax the function class.

Once we parametrize the critic $f$ as a neural network with linear output activation, i.e. $f(x)=\left\langle{v},{\Phi_{\omega}(x)}\right\rangle$, we see that the optimization is unconstrained for the $\varphi$-divergence formulation (10) and the weights updates can explode and have an unstable behavior. On the other hand in the Fisher formulation (11) the data dependent constraint that is imposed slowly through the lagrange multiplier, enforces a variance control that prevents the critic from blowing up and causing instabilities in the training. Note that in the Fisher case we have three players: the critic, the generator and the lagrange multiplier. The lagrange multiplier grows slowly to enforce the constraint and to approach the Chi-squared distance as training converges. Note that the $\varphi$-divergence formulation (10) can be seen as a Fisher GAN with fixed lagrange multiplier $\lambda=\frac{1}{2}$ that is indeed unstable in theory and in our experiments.

Note that if the Neyman divergence is of interest, it can also be obtained as the following Fisher criterium:

Using a neural network $f(x)=\left\langle{v},{\Phi_{\omega}(x)}\right\rangle$, the Neyman divergence can be achieved with linear output activation and a data dependent constraint:

To obtain the same divergence as a $\varphi$-divergence we need $\varphi(u)=\frac{(1-u)^{2}}{u}$, and $\varphi^{*}(u)=2-2\sqrt{1-u},~{}(u<1).$ Moreover exponential activation functions are used in , which most likely renders this formulation also unstable for GAN training.

Appendix D Proofs

Consider the space of measurable functions,

which gives us the expression of $\lambda_{*}$:

Define the following distance between two distributions:

We refer to this distance as the $\chi_{2}$ distance between two distributions. It is easy to see that :

and the optimal critic $f_{\chi}$ has the following expression:

Define the means difference functional $\mathscr{E}$:

For a symmetric function class $\mathscr{H}$, the Fisher IPM has the following expression:

note that since $\mathscr{H}$ is symmetric all quantities are positive after taking the sup (if $\mathscr{H}$ was not symmetric one can take the absolute values, and similar results hold with absolute values.)

Appendix E Theorem 3: Generalization Bounds

Let $\mathscr{H}$ be a function space of real valued functions on $\pazocal{X}$. We assume that $\mathscr{H}$ is bounded, there exists $\nu>0$, such that $\left\lVert{f}\right\rVert_{\infty}\leq\nu$. Since the second moments are bounded we can relax this assumption using Chebyshev’s inequality, we have:

hence we have boundedness with high probability. Define the expected mean discrepancy $\mathscr{E}(.)$ and the second order norm $\Omega(.)$:

For $\tau>0$. The following generalization bound on the estimation of the Chi-squared distance, with probability $1-12e^{-\tau}$:

$\hat{\lambda}$ is the Lagrange multiplier, $c_{1},c_{2},c_{3},c_{4}$ are numerical constants, and $\mathscr{R}_{M,N}$ is the rademacher complexity:

is the effective dimension $(d(\gamma)<<m)$. Hence we see that typically $\varepsilon_{n}=O(\frac{1}{\sqrt{n}})$.

Recall some definitions of the Fisher IPM:

We assume that a saddle point for this problem exists and it is feasible. We assume also that $\hat{\lambda}$ is positive and bounded.

The fisher IPM empirical estimate is given by:

Bounding the Approximation Error. By Theorem 2 we know that:

Hence we are left with bounding the statistical error using empirical processes theory. Assume $\mathscr{H}$ is a space of bounded functions i.e $\left\lVert{f}\right\rVert_{\infty}\leq\nu$.

Bounding the Statistical Error. Note that we have: (i) $\hat{\pazocal{L}}(f^{*},\hat{\lambda})\leq\hat{\pazocal{L}}(\hat{f},\hat{\lambda})$ and (ii) $\Omega(f^{*})=1$.

Let $S=\{x_{1}\dots x_{N},y_{1}\dots y_{M}\}$. Define the following quantities:

Let $Z(S)=\sup_{f\in\mathscr{F}}\mathscr{E}(f)-\hat{\mathscr{E}}(f)$, Assume that $\left\lVert{f}\right\rVert_{\infty}\leq\nu$, for all $f\in\mathscr{F}$.

The same result holds for : $Z(S)=\sup_{f\in\mathscr{F}}\hat{\mathscr{E}}(f)-\mathscr{E}(f).$

We have with probability $1-e^{-\tau}$ for all $\delta\in(0,1)$:

Let $\phi$ be a contraction, that is $\lvert{\phi(x)-\phi(y)}\rvert\leq L\lvert{x-y}\rvert$. Then, for every class $\mathscr{F}$,

$\phi\circ\mathscr{F}=\{\phi\circ f,f\in\mathscr{F}\}$.

Note that applying the contraction Lemma for $\phi(x)=x^{2}$ (with lipchitz constant $2\nu$ on $[-\nu,\nu]$) we have:

Now using that for any $\alpha>0$: $2\sqrt{uv}\leq\alpha u+\frac{v}{\alpha}$ we have for $\alpha=\frac{1}{2}$: $\sqrt{\frac{2\tau\nu^{4}}{n}}\leq\frac{\nu^{2}}{2}+\frac{4\tau\nu^{2}}{n}.$ For some universal constants, $c_{1},c_{2}$, let:

we have therefore with probability $1-2e^{-\tau}$:

note that Typically $\eta_{n}=O(\frac{1}{\sqrt{n}})$. The same inequality holds with the same probability:

Note that we have now the following inclusion using Equation (21):

Hence we obtain a data dependent bound in Equations (19),(20) with a union bound with probability $1-6e^{-\tau}$.

Bounding $Z_{2}(S)$. Note that concentration inequalities don’t apply to $Z_{2}(S)$ since the cost function and the function class are data dependent. We need to turn the constraint to a data independent constraint i.e does not depend on the training set. For $f,\hat{\Omega}(f)\leq 1$, by Equation (22) we have with probability $1-2e^{-\tau}$:

we have therefore the following inclusion with probability $1-2e^{-\tau}$:

Now reapplying the inclusion using Equation (21), we get the following bound on the local rademacher complexity with probability $1-2e^{-\tau}$:

Hence with probability $1-6e^{-\tau}$ we have:

Putting all together. We have with probability at least $1-12e^{-\tau}$, for universal constants $c_{1},c_{2},c_{3},c_{4}$

Note that typically $\varepsilon_{n}=O(\frac{1}{\sqrt{n}})$. Hence it follows that:

Example: Bounding local complexity for a simple linear function class.

Hence we obtain the following bound on the local rademacher complexity:

Note that without the local constraint the effective dimension $d(\gamma)$ (typically $d(\gamma)<<m$) is replaced by the ambient dimension $m$.∎

Appendix F Hyper-parameters and Architectures of Discriminator and Generators

As noted in Figure 5 and in above paragraph, we used intialization from a normal distribution with stdev=0.02 for the inception score experiments for both Fisher GAN and WGAN-GP. For transparency, and to show that our architecture and initialization benefits both Fisher GAN and WGAN-GP, we provide plots of different combinations below (Figure 6). Architecture-wise, F64 refers to the architecture described in Appendix F.3 with 64 feature maps after the first convolutional layer. F128 is the architecture from the WGAN-GP codebase , which has double the number of feature maps (128 fmaps) and does not have the two extra layers in G and D (D layers 2-7, G layers 9-14). The result reported in the WGAN-GP paper corresponds to WGAN-GP F128 He init. For WGAN (Figure 7) the 64-fmap architecture gives some initial instability but catches up to the same level as the 128-fmap architecture.

F.2 LSUN and CelebA.

F.3 CIFAR-10: Sample Quality and Inceptions Scores Experiments

F.4 CIFAR-10: SSL Experiments

Appendix G Sample implementation in PyTorch

This minimalistic sample code is based on https://github.com/martinarjovsky/WassersteinGAN at commit d92c503.

Adding Cross-Entropy objective and class-conditioned generator.

First note the essential change in the critic’s forward pass definition:

G.2 Full diff from reference

Note that from the arXiv LaTeX source, the file diff.txt could be used in combination with git apply.