Reducing Noise in GAN Training with Variance Reduced Extragradient

Introduction

Many empirical risk minimization algorithms rely on gradient-based optimization methods. These iterative methods handle large-scale training datasets by computing gradient estimates on a subset of it, a mini-batch, instead of using all the samples at each step, the full batch, resulting in a method called stochastic gradient descent (SGD, Robbins and Monro (1951); Bottou (2010)).

SGD methods are known to efficiently minimize single objective loss functions, such as cross-entropy for classification or squared loss for regression. Some algorithms go beyond such training objective and define multiple agents with different or competing objectives. The associated optimization paradigm requires a multi-objective joint minimization. An example of such a class of algorithms are the generative adversarial networks (GANs, Goodfellow et al., 2014), which aim at finding a Nash equilibrium of a two-player minimax game, where the players are deep neural networks (DNNs).

As of their success on supervised tasks, SGD based algorithms have been adopted for GAN training as well. Recently, Gidel et al. (2019a) proposed to use an optimization technique coming from the variational inequality literature called extragradient (Korpelevich, 1976) with provable convergence guarantees to optimize games (see § 2). However, convergence failures, poor performance (sometimes referred to as “mode collapse”), or hyperparameter susceptibility are more commonly reported compared to classical supervised DNN optimization.

We question naive adoption of such methods for game optimization so as to address the reported training instabilities. We argue that as of the two player setting, noise impedes drastically more the training compared to single objective one. More precisely, we point out that the noise due to the stochasticity may break the convergence of the extragradient method, by considering a simplistic stochastic bilinear game for which it provably does not converge.

The theoretical aspect we present in this paper is further supported empirically, since using larger mini-batch sizes for GAN training has been shown to considerably improve the quality of the samples produced by the resulting generative model: Brock et al. (2019) report a relative improvement of $46\%$ of the Inception Score metric (see § 4) on ImageNet if the batch size is increased $8$–fold. This notable improvement raises the question if noise reduction optimization methods can be extended to game settings. In turn, this would allow for a principled training method with the practical benefit of omitting to empirically establish this multiplicative factor for the batch size.

In this paper, we investigate the interplay between noise and multi-objective problems in the context of GAN training. Our contributions can be summarized as follows: (i) we show in a motivating example how the noise can make stochastic extragradient fail (see § 2.2). (ii) we propose a new method “stochastic variance reduced extragradient” (SVRE) that combines variance reduction and extrapolation (see Alg. 1 and § 3.2) and show experimentally that it effectively reduces the noise. (iii) we prove the convergence of SVRE under local strong convexity assumptions, improving over the known rates of competitive methods for a large class of games (see § 3.2 for our convergence result and Table 1 for comparison with standard methods). (iv) we test SVRE empirically to train GANs on several standard datasets, and observe that it can improve SOTA deep models in the late stage of their optimization (see § 4).

GANs as a Game and Noise in Games

The models in a GAN are a generator $G$, that maps an embedding space to the signal space, and should eventually map a fixed noise distribution to the training data distribution, and a discriminator $D$ whose purpose is to allow the training of the generator by classifying genuine samples against generated ones. At each iteration of the algorithm, the discriminator $D$ is updated to improve its “real vs. generated” classification performance, and the generator $G$ to degrade it.

From a game theory point of view, GAN training is a differentiable two-player game where the generator $G_{{\bm{\theta}}}$ and the discriminator $D_{{\bm{\varphi}}}$ aim at minimizing their own cost function ${\mathcal{L}}^{G}$ and ${\mathcal{L}}^{D}$, resp.:

When ${\mathcal{L}}^{D}=-{\mathcal{L}}^{G}=:{\mathcal{L}}$ this game is called a zero-sum game and (2P-G) is a minimax problem:

Note how ${\bm{\theta}}_{t}$ and ${\bm{\varphi}}_{t}$ are updated with a gradient from a different point, the extrapolated one. In the context of a zero-sum game, for any convex-concave function ${\mathcal{L}}$ and any closed convex sets $\Theta$ and $\Phi$, the extragradient method converges (Harker and Pang, 1990, Thm. 12.1.11).

2 Stochasticity Breaks Extragradient

As the (EG) converges for some examples for which gradient methods do not, it is reasonable to expect that so does its stochastic counterpart (at least to a neighborhood). However, the resulting noise in the gradient estimate may interact in a problematic way with the oscillations due to the adversarial component of the gameGidel et al. (2019b) formalize the notion of “adversarial component” of a game, which yields a rotational dynamics in gradients methods (oscillations in parameters), as illustrated by the gradient field of Fig. 1 (right).. We depict this phenomenon in Fig. 1, where we show the direction of the noisy gradient on single objective minimization example and contrast it with a multi-objective one.

We present a simplistic example where the extragradient method converges linearly (Gidel et al., 2019a, Corollary 1) using the full gradient but diverges geometrically when using stochastic estimates of it. Note that standard gradient methods, both batch and stochastic, diverge on this example.

In particular, we show that: (i) if we use standard stochastic estimates of the gradients of ${\mathcal{L}}$ with a simple finite sum formulation, then the iterates ${\bm{\omega}}_{t}:=({\bm{\theta}}_{t},{\bm{\varphi}}_{t})$ produced by the stochastic extragradient method (SEG) diverge geometrically, and on the other hand (ii) the full-batch extragradient method does converge to the Nash equilibrium ${\bm{\omega}}^{*}$ of this game (Harker and Pang, 1990, Thm. 12.1.11).

Proof sketch. All detailed proofs can be found in § C of the appendix. We consider the following stochastic optimization problem (with $d=n$):

Sampling a mini-batch without replacement $I\subset\{1,\ldots,n\}$, we denote ${\bm{A}}_{I}:=\sum_{i\in I}{\bm{A}}_{i}$. The extragradient update rule can be written as:

where $I$ and $J$ are the mini-batches sampled for the update and the extrapolation step, respectively. Let us write $N_{t}:=\|{\bm{\theta}}_{t}\|^{2}+\|{\bm{\varphi}}_{t}\|^{2}$. Noticing that $[{\bm{A}}_{I}{\bm{\theta}}]_{i}=[{\bm{\theta}}]_{i}$ if $i\in I$ and otherwise, we have,

This result may seem contradictory with the standard result on SEG (Juditsky et al., 2011) saying that the average of the iterates computed by SEG does converge to the Nash equilibrium of the game. However, an important assumption made by Juditsky et al. is that the iterates are projected onto a compact set and that estimator of the gradient has finite variance. These assumptions break in this example since the variance of the estimator is proportional to the norm of the (unbounded) parameters. Note that constraining the optimization problem (24) to bounded domains $\Theta$ and $\Phi$, would make the finite variance assumption from Juditsky et al. (2011) holds. Consequently, the averaged iterate $\bar{\bm{\omega}}_{t}:=\frac{1}{t}\sum_{s=0}^{t-1}{\bm{\omega}}_{s}$ would converge to ${\bm{\omega}}^{*}$. In § A.1, we explain why in a non-convex setting, the convergence of the last iterate is preferable.

Reducing Noise in Games with Variance Reduced Extragradient

One way to reduce the noise in the estimation of the gradient is to use mini-batches of samples instead of one sample. However, mini-batch stochastic extragradient fails to converge on (24) if the mini-batch size is smaller than half of the dataset size (see § C.1). In order to get an estimator of the gradient with a vanishing variance, the optimization literature proposed to take advantage of the finite-sum formulation that often appears in machine learning (Schmidt et al., 2017, and references therein).

Let us assume that the objective in (2P-G) can be decomposed as a finite sum such thatThe “noise dataset” in a GAN is not finite though; see § D.1 for details on how to cope with this in practice.

Johnson and Zhang (2013) propose the “stochastic variance reduced gradient” (SVRG) as an unbiased estimator of the gradient with a smaller variance than the vanilla mini-batch estimate. The idea is to occasionally take a snapshot ${\bm{\omega}}^{\mathcal{S}}$ of the current model’s parameters, and store the full batch gradient ${\bm{\mu}}^{\mathcal{S}}$ at this point. Computing the full batch gradient ${\bm{\mu}}^{\mathcal{S}}$ at ${\bm{\omega}}^{\mathcal{S}}$ is an expensive operation but not prohibitive if done infrequently (for instance once every dataset pass).

Assuming that we have stored ${\bm{\omega}}^{\mathcal{S}}$ and ${\bm{\mu}}^{\mathcal{S}}:=({\bm{\mu}}_{\bm{\theta}}^{\mathcal{S}},{\bm{\mu}}_{\bm{\varphi}}^{\mathcal{S}})$, the SVRG estimates of the gradients are:

Originally, SVRG was introduced as an epoch based algorithm with a fixed epoch size: in Alg. 1, one epoch is an inner loop of size $N$ (Line 6). However, Hofmann et al. (2015) proposed instead to sample the size of each epoch from a geometric distribution, enabling them to analyze SVRG the same way as SAGA under a unified framework called $q$-memorization algorithm. We generalize their framework to handle the extrapolation step (EG) and provide a convergence proof for such $q$-memorization algorithms for games in § C.2.

One advantage of Hofmann et al. (2015)’s framework is also that the sampling of the epoch size does not depend on the condition number of the problem, whereas the original proof for SVRG had to consider an epoch size larger than the condition number (see Leblond et al. (2018, Corollary 16) for a detailed discussion on the convergence rate for SVRG). Thus, this new version of SVRG with a random epoch size becomes adaptive to the local strong convexity since none of its hyper-parameters depend on the strong convexity constant.

However, because of some new technical aspects when working with monotone operators, Palaniappan and Bach (2016)’s proofs (both for SAGA and SVRG) require a step-size (and epoch length for SVRG) that depends on the strong monotonicity constant making these algorithms not adaptive to local strong monotonicity. This motivates the proposed SVRE algorithm, which may be adaptive to local strong monotonicity, and is thus more appropriate for non-convex optimization.

2 SVRE: Stochastic Variance Reduced Extragradient

We describe our proposed algorithm called stochastic variance reduced extragradient (SVRE) in Alg. 1. In an analogous manner to how Palaniappan and Bach (2016) combined SVRG with the gradient method, SVRE combines SVRG estimates of the gradient (6) with the extragradient method (EG).

With SVRE we are able to improve the convergence rates for variance reduction for a large class of stochastic games (see Table 1 and Thm. 2), and we show in § 3.3 that it is the only method which empirically converges on the simple example of § 2.2.

We now describe the theoretical setup for the convergence result. A standard assumption in convex optimization is the assumption of strong convexity of the function. However, in a game, the operator,

associated with the updates is no longer the gradient of a single function. To make an analogous assumption for games the optimization literature considers the notion of strong monotonicity.

This definition is a generalization of strong convexity for operators: if $f$ is $\mu$-strongly convex, then $\nabla f$ is a $\mu$-monotone operator. Another assumption is the $\gamma$ regularity assumption,

Note that an operator is always $(0,0)$-regular. This assumption originally introduced by Tseng (1995) has been recently used (Azizian et al., 2019) to improve the convergence rate of extragradient. For instance for a full rank bilinear matrix problem $\gamma$ is its smallest singular value. More generally, in the case $\gamma_{\bm{\theta}}=\gamma_{\bm{\varphi}}$, the regularity constant is a lower bound on the minimal singular value of the Jacobian of $F$ (Azizian et al., 2019).

One of our main assumptions is the cocoercivity assumption, which implies the Lipchitzness of the operator in the unconstrained case. We use the cocoercivity constant because it provides a tighter bound for general strongly monotone and Lipschitz games (see discussion following Theorem 2).

For our main result, we make strong convexity, cocoercivity and regularity assumptions.

We now present our convergence result for SVRE with non-uniform sampling (to make our constants comparable to those of Palaniappan and Bach (2016)), but note that we have used uniform sampling in all our experiments (for simplicity).

3 Motivating example

The example (24) for $\epsilon=0$ seems to be challenging in the stochastic setting since all the standard methods and even the stochastic extragradient method fails to find its Nash equilibrium (note that this example is not strongly monotone). We set $n=d=100$, and draw $[{\bm{A}}_{i}]_{kl}=\delta_{kli}\text{ and }[{\bm{b}}_{i}]_{k},[{\bm{c}}_{i}]_{k}\sim\mathcal{N}(0,1/d)\,,\;1\leq k,l\leq d$, where $\delta_{kli}=1$ if $k=l=i$ and otherwise. Our optimization problem is:

We compare variants of the following algorithms (with uniform sampling and average our results over 5 different seeds): (i) AltSGD: the standard method to train GANs–stochastic gradient with alternating updates of each player. (ii) SVRE: Alg. 1. The AVG prefix correspond to the uniform average of the iterates, $\bar{\bm{\omega}}:=\frac{1}{t}\sum_{s=0}^{t-1}{\bm{\omega}}_{s}$. We observe in Fig. 4 that AVG-SVRE converges sublinearly (whereas AVG-AltSGD fails to converge).

This motivates a new variant of SVRE based on the idea that even if the averaged iterate converges, we do not compute the gradient at that point and thus we do not benefit from the fact that this iterate is closer to the optimums (see § A.1). Thus the idea is to occasionally restart the algorithm, i.e., consider the averaged iterate as the new starting point of our algorithm and compute the gradient at that point. Restart goes well with SVRE as we already occasionally stop the inner loop to recompute ${\bm{\mu}}^{\mathcal{S}}$, at which point we decide (with a probability $p$ to be fixed) whether or not to restart the algorithm by taking the snapshot at point $\bar{\bm{\omega}}_{t}$ instead of ${\bm{\omega}}_{t}$. This variant of SVRE is described in Alg. 3 in § E and the variant combining VRAd in § D.1.

In Fig. 4 we observe that the only method that converges is SVRE and its variants. We do not provide convergence guarantees for Alg. 3 and leave its analysis for future work. However, it is interesting that, to our knowledge, this algorithm is the only stochastic algorithm (excluding batch extragradient as it is not stochastic) that converge for (24). Note that we tried all the algorithms presented in Fig. 3 from Gidel et al. (2019a) on this unconstrained problem and that all of them diverge.

GAN Experiments

In this section, we investigate the empirical performance of SVRE for GAN training. Note, however, that our theoretical analysis does not hold for games with non-convex objectives such as GANs.

Datasets. We used the following datasets: (i) MNIST(Lecun and Cortes, ), (ii) CIFAR-10(Krizhevsky, 2009, §3), (iii) SVHN(Netzer et al., 2011), and (iv) ImageNetILSVRC 2012 (Russakovsky et al., 2015), using $28\!\times\!28$, $3\!\times\!32\!\times\!32$, $3\!\times\!32\!\times\!32$, and $3\!\times\!64\!\times\!64$ resolution, respectively.

Metrics. We used the Inception score (IS, Salimans et al., 2016) and the Fréchet Inception distance (FID, Heusel et al., 2017) as performance metrics for image synthesis. To gain insights if SVRE indeed reduces the variance of the gradient estimates, we used the second moment estimate–SME (uncentered variance), computed with an exponentially moving average. See § F.1 for details.

DNN architectures. For experiments on MNIST, we used the DCGAN architectures (Radford et al., 2016), described in § F.2.1. For real-world datasets, we used two architectures (see § F.2 for details and § F.2.2 for motivation): (i) SAGAN (Zhang et al., 2018), and (ii) ResNet, replicating the setup of Miyato et al. (2018), described in detail in § F.2.3 and F.2.4, respectively. For clarity, we refer the former as shallow, and the latter as deep architectures.

We conduct experiments using the following optimization methods for GANs: (i) BatchE:full–batch extragradient, (ii) SG:stochastic gradient (alternating GAN), and (iii) SE:stochastic extragradient, and (iv) SVRE:stochastic variance reduced extragradient. These can be combined with adaptive learning rate methods such as Adam or with parameter averaging, hereafter denoted as –A and AVG–, respectively. In § D.1, we present a variant of Adam adapted to variance reduced algorithms, that is referred to as –VRAd. When using the SE–A baseline and deep architectures, the convergence rapidly fails at some point of training (cf. § G.3). This motivates experiments where we start from a stored checkpoint taken before the baseline diverged, and continue training with SVRE. We denote these experiments with WS–SVRE (warm-start SVRE).

1 Results

Comparison on MNIST. The MNIST common benchmark allowed for comparison with full-batch extragradient, as it is feasible to compute. Fig. 3 depicts the IS metric while using either a stochastic, full-batch or variance reduced version of extragradient (see details of SVRE-GAN in § D.2). We always combine the stochastic baseline (SE) with Adam, as proposed by Gidel et al. (2019a). In terms of number of parameter updates, SVRE performs similarly to BatchE–A (see Fig. 4(a), § G). Note that the latter requires significantly more computation: Fig. 3(a) depicts the IS metric using the number of mini-batch computations as x-axis (a surrogate for the wall-clock time, see below). We observe that, as SE–A has slower per-iteration convergence rate, SVRE converges faster on this dataset. At the end of training, all methods reach similar performances (IS is above $8.5$, see Table 9, § G).

Computational cost. The relative cost of one pass over the dataset for SVRE versus vanilla SGD is a factor of $5$: the full batch gradient is computed (on average) after one pass over the dataset, giving a slowdown of $2$; the factor $5$ takes into account the extra stochastic gradient computations for the variance reduction, as well as the extrapolation step overhead. However, as SVRE provides less noisy gradient, it may converge faster per iteration, compensating the extra per-update cost. Note that many computations can be done in parallel. In Fig. 3(a), the x-axis uses an implementation-independent surrogate for wall-clock time that counts the number of mini-batch gradient computations. Note that some training methods for GANs require multiple discriminator updates per generator update, and we observed that to stabilize our baseline when using the deep architectures it was required to use $1{:}5$ update ratio of $G{:}D$ (cf. § G.3), whereas for SVRE we used ratio of $1{:}1$ (Tab. 2 lists the results). Second moment estimate and Adam. Fig. 3(b) depicts the averaged second-moment estimate for parameters of the Generator, where we observe that SVRE effectively reduces it over the iterations. The reduction of these values may be the reason why Adam combined with SVRE performs poorly (as these values appear in the denominator, see § D.1). To our knowledge, SVRE is the first optimization method with a constant step size that has worked empirically for GANs on non-trivial datasets.

Comparison on real-world datasets. In Fig. 3(c), we compare SVRE with the SE–A baseline on SVHN, using shallow architectures. We observe that although SE–A in some experiments obtains better performances in the early iterations, SVRE allows for obtaining improved final performances. Tab. 2 summarizes the results on CIFAR-10 and SVHN with deep architectures. We observe that, with deeper architectures, SE–A is notably more unstable, as training collapsed in $100$% of the experiments. To obtain satisfying results for SE–A, we used various techniques such as a schedule of the learning rate and different update ratios (see § G.3). On the other hand, SVRE did not collapse in any of the experiments but took longer time to converge compared to SE–A. Interestingly, although WS–SVRE starts from an iterate point after which the baseline diverges, it continues to improve the obtained FID score and does not diverge. See § G for additional experiments.

Related work

Surprisingly, there exist only a few works on variance reduction methods for monotone operators, namely from Palaniappan and Bach (2016) and Davis (2016). The latter requires a co-coercivity assumption on the operator and thus only convex optimization is considered. Our work provides a new way to use variance reduction for monotone operators, using the extragradient method (Korpelevich, 1976). Recently, Iusem et al. (2017) proposed an extragradient method with variance reduction for an infinite sum of operators. The authors use mini-batches of growing size in order to reduce the variance of their algorithm and to converge with a constant step-size. However, this approach is prohibitively expensive in our application. Moreover, Iusem et al. are not using the SAGA/SVRG style of updates exploiting the finite sum formulation, leading to sublinear convergence rate, while our method benefits from a linear convergence rate exploiting the finite sum assumption.

Daskalakis et al. (2018) proposed a method called Optimistic-Adam inspired by game theory. This method is closely related to extragradient, with slightly different update scheme. More recently, Gidel et al. (2019a) proposed to use extragradient to train GANs, introducing a method called ExtraAdam. This method outperformed Optimistic-Adam when trained on CIFAR-10. Our work is also an attempt to find principled ways to train GANs. Considering that the game aspect is better handled by the extragradient method, we focus on the optimization issues arising from the noise in the training procedure, a disregarded potential issue in GAN training.

In the context of deep learning, despite some very interesting theoretical results on non-convex minimization (Reddi et al., 2016; Allen-Zhu and Hazan, 2016), the effectiveness of variance reduced methods is still an open question, and a recent technical report by Defazio and Bottou (2018) provides negative empirical results on the variance reduction aspect. In addition, two recent large scale studies showed that increased batch size has: (i) only marginal impact on single objective training (Shallue et al., 2018) and (ii) a surprisingly large performance improvement on GAN training (Brock et al., 2019). In our work, we are able to show positive results for variance reduction in a real-world deep learning setting. This unexpected difference seems to confirm the remarkable discrepancy, that remains poorly understood, between multi-objective optimization and standard minimization.

Discussion

Motivated by a simple bilinear game optimization problem where stochasticity provably breaks the convergence of previous stochastic methods, we proposed the novel SVRE algorithm that combines SVRG with the extragradient method for optimizing games. On the theory side, SVRE improves upon the previous best results for strongly-convex games, whereas empirically, it is the only method that converges for our stochastic bilinear game counter-example.

We empirically observed that SVRE for GAN training obtained convergence speed similar to Batch-Extragradient on MNIST, while the latter is computationally infeasible for large datasets. For shallow architectures, SVRE matched or improved over baselines on all four datasets. Our experiments with deeper architectures show that SVRE is notably more stable with respect to hyperparameter choice. Moreover, while its stochastic counterpart diverged in all our experiments, SVRE did not. However, we observed that SVRE took more iterations to converge when using deeper architectures, though notably, we were using constant step-sizes, unlike the baselines which required Adam. As adaptive step-sizes often provide significant improvements, developing such an appropriate version for SVRE is a promising direction for future work. In the meantime, the stability of SVRE suggests a practical use case for GANs as warm-starting it just before the baseline diverges, and running it for further improvements, as demonstrated with the WS–SVRE method in our experiments.

Acknowledgements

This research was partially supported by the Canada CIFAR AI Chair Program, the Canada Excellence Research Chair in “Data Science for Realtime Decision-making”, by the NSERC Discovery Grant RGPIN-2017-06936, by the Hasler Foundation through the MEMUDE project, and by a Google Focused Research Award. Authors would like to thank Compute Canada for providing the GPUs used for this research. TC would like to thank Sebastian Stich and Martin Jaggi, and GG and TC would like to thank Hugo Berard for helpful discussions.

References

Appendix A Noise in games

In light of Theorem 1, the behavior of the iterates on the unconstrained version of (24) ($\epsilon=0$):

where $\Theta$ and $\Phi$ are compact and convex sets, is the following: they will diverge until they reach the boundary of $\Theta$ and $\Phi$ and then they will start to turn around the Nash equilibrium of (12) lying on these boundaries. Using convexity properties, we can then show that the averaged iterates will converge to the Nash equilibrium of the problem. However, with an arbitrary large domain, this convergence rate may be arbitrary slow (since it depends on the diameter of the domain).

Moreover, this behavior might be even more problematic in a non-convex framework because even if by chance we initialize close to the Nash equilibrium, we would get away from it and we cannot rely on convexity to expect the average of the iterates to converge.

Consequently, we would like optimization algorithms generating iterates that stay close to the Nash equilibrium.

Appendix B Definitions and Lemmas

Another important property used is the Lipschitzness of an operator.

A function $({\bm{\theta}},{\bm{\varphi}})\mapsto{\mathcal{L}}({\bm{\theta}},{\bm{\varphi}})$ is said convex-concave if ${\mathcal{L}}(\cdot,{\bm{\varphi}})$ is convex for all ${\bm{\varphi}}\in\Phi$ and ${\mathcal{L}}({\bm{\theta}},\cdot)$ is concave for all ${\bm{\theta}}\in\Theta$. An ${\mathcal{L}}$ is said to be $\mu$-strongly convex concave if $({\bm{\theta}},{\bm{\varphi}})\mapsto{\mathcal{L}}({\bm{\theta}},{\bm{\varphi}})-\frac{\mu}{2}\|{\bm{\theta}}\|_{2}^{2}+\frac{\mu}{2}\|{\bm{\varphi}}\|_{2}^{2}$ is convex concave.

A $L$-Lipschitz and $\mu$-strongly monotone operator is $L^{2}/\mu$-cocoercive

By applying lipschitzness and strong monotonicity,

We rewrite $F$ as the sum of the gradient of convex Lipschitz function $F_{grad}$ and a $L$-Lipschitz and $\mu$-strongly monotone operator $F_{mon}$:

where for the second inequality we used that a $(L+\mu)$-Lipschitz convex function is $(L+\mu)$-cocoercive and Proposition 2. ∎

Appendix C Proof of Theorems

We consider the following stochastic optimization problem,

where $[{\bm{A}}_{i}]_{kl}=1$ if $k=l=i$ and otherwise. Note that $({\bm{A}}_{i})^{\top}={\bm{A}}_{i}$ for $1\leq i\leq n$. Let us consider the extragradient method where to compute an unbiased estimator of the gradients at $({\bm{\theta}},{\bm{\varphi}})$ we sample $i\in\{1,\ldots,n\}$ and use $[{\bm{A}}_{i}{\bm{\theta}},\,{\bm{A}}_{i}{\bm{\varphi}}]$ as estimator of the vector flow.

In this proof we note, ${\bm{A}}_{I}:=\sum_{i\in I}{\bm{A}}_{i}$ and ${\bm{\theta}}^{(I)}$ the vector such that $[{\bm{\theta}}^{(I)}]_{i}=[{\bm{\theta}}]_{i}$ if $i\in I$ and otherwise. Note that ${\bm{A}}_{I}{\bm{\theta}}={\bm{\theta}}^{(I)}$ and that ${\bm{A}}_{I}{\bm{A}}_{J}={\bm{A}}_{I\cap J}$.

Thus the extragradient update rule can be noted as

where $I$ is the mini-batch sampled (without replacement) for the update and $J$ the mini-batch sampled (without replacement) for the extrapolation.

We can thus notice that, when $I\cap J=\emptyset$, we have

Conditioning on ${\bm{\theta}}_{t}$ and ${\bm{\varphi}}_{t}$, we get that

Plugging these expectations in (29), we get that,

C.2 Proof of Theorem 2

We will prove a slightly more general result than Theorem 2. We will work in the context of monotone operator. Let us consider the general extrapolation update rule,

where ${\bm{g}}_{t}$ depends on ${\bm{\omega}}_{t}$ and ${\bm{g}}_{t+1/2}$ depends on ${\bm{\omega}}_{t+1/2}$. For instance, ${\bm{g}}_{t}$ can either be $F({\bm{\omega}}_{t})$, $F_{i_{t}}({\bm{\omega}}_{t})$ or the SVRG estimate defined in (46).

This update rule generalizes (EG) for 2-player games (2P-G) and ExtraSVRG (Alg. 2).

Let us first state a lemma standard in convex analysis (see for instance (Boyd and Vandenberghe, 2004)),

Let ${\bm{\omega}}\in\Omega$ and ${\bm{\omega}}^{+}:=P_{\Omega}({\bm{\omega}}+{\boldsymbol{u}})$ then for all ${\bm{\omega}}^{\prime}\in\Omega$ we have,

Then since ${\bm{\omega}}^{+}$ is the projection onto the convex set $\Omega$ of ${\bm{\omega}}+{\boldsymbol{u}}$ we have that $({\bm{\omega}}^{+}-({\bm{\omega}}+{\boldsymbol{u}}))^{\top}({\bm{\omega}}^{+}-{\bm{\omega}}^{\prime})\leq 0\,,\;\forall\,{\bm{\omega}}^{\prime}\in\Omega$, leading to the result of the Lemma. ∎

If $F$ is $(\mu_{\bm{\theta}},\mu_{\bm{\varphi}})$-strongly monotone for any ${\bm{\omega}},{\bm{\omega}}^{\prime},{\bm{\omega}}^{\prime\prime}\in\Omega$ we have,

where we noted ${\bm{\omega}}:=({\bm{\theta}},{\bm{\varphi}})$.

By $(\mu_{\bm{\theta}},\mu_{\bm{\varphi}})$-strong monotonicity,

and then we use the inequality $2\|{\bm{a}}^{\prime}-{\bm{a}}^{\prime\prime}\|_{2}^{2}\geq\|{\bm{a}}-{\bm{a}}^{\prime\prime}\|_{2}^{2}-2\|{\bm{a}}^{\prime}-{\bm{a}}\|_{2}^{2}$ to get the result claimed. ∎

Using this update rule we can thus deduce the following lemma, the derivation of this lemma is very similar from the derivation of Harker and Pang (1990, Lemma 12.1.10).

Considering the update rule (34), we have for any ${\bm{\omega}}\in\Omega$ and any $t\geq 0$,

By applying Lem. 1 for $({\bm{\omega}},{\boldsymbol{u}},{\bm{\omega}}^{+},{\bm{\omega}}^{\prime})=({\bm{\omega}}_{t},-\eta_{t}{\bm{g}}_{t+1/2},{\bm{\omega}}_{t+1},{\bm{\omega}})$ and $({\bm{\omega}},{\boldsymbol{u}},{\bm{\omega}}^{+},{\bm{\omega}}^{\prime})=({\bm{\omega}}_{t},-\eta_{t}{\bm{g}}_{t},{\bm{\omega}}_{t+1/2},{\bm{\omega}}_{t+1})$, we get,

Then, we can use Young’s inequality $-2a^{\top}b\leq\|a\|_{2}^{2}+\|b\|_{2}^{2}$ to get,

Note that if we would have set ${\bm{g}}_{t}=\bm{0}$ and ${\bm{g}}_{t+1/2}$ any estimate of the gradient at ${\bm{\omega}}_{t}$ we recover the standard lemma for gradient method.

Let us consider unbiased estimates of the gradient,

We will consider a class of algorithm called uniform memorization algorithms first introduced by (Hofmann et al., 2015). This class of algorithms describes a large subset of variance reduced algorithms taking advantage of the finite sum formulation such as SAGA (Defazio et al., 2014), SVRG (Johnson and Zhang, 2013) or $q$-SAGA and $\mathcal{N}$-SAGA (Hofmann et al., 2015). In this work, we will use a slightly more general definition of such algorithm in order to be able to handle extrapolation steps:

A uniform $q$-memorization algorithm evolves iterates $({\bm{\omega}}_{t})$ according to (34), with ${\bm{g}}_{t}$ defined in (46) and selecting in each iteration $t$ a random index set $J_{t}$ of memory locations to update according to,

such that any $k$ has the same probability $q/n$ to be updated, i.e., $P\{k\}=\sum_{J_{t},k\in J_{t}}P(J_{t})=q/n$, $\forall k\in\{1,\ldots,n\}.$

In the case of SVRG, either $J_{t}=\emptyset$ or $J_{t}=\{1,\ldots,n\}$ (when we update the snapshot).

For any $t\geq 0$, if we consider a $q$-memorization algorithm we have

We use an extended version of Young’s inequality: $\|\sum_{i=1}^{k}{\bm{a}}_{i}\|^{2}\leq k\sum_{i=1}^{k}\|{\bm{a}}_{i}\|^{2}$,

Notice that since $i_{t}$ and $j_{t}$ are independently sampled from the same distribution we have

By assuming that each $F_{i}$ is $L_{i}$-Lipschitz we get,

where $\bar{L}^{2}:=\frac{1}{n^{2}}\sum_{i=1}^{n}\frac{L_{i}^{2}}{\pi_{j}}$. Note that ${\bm{\omega}}_{t}$ and ${\bm{\omega}}_{t+1/2}$ do not depend on $j_{t}$ (which is the index sampled for the update step), that is not the case for $i$ (the index for the extrapolation step) since ${\bm{\omega}}_{t+1/2}$ is the result of the extrapolation. ∎

We will use the definition of $q$-uniform memorization algorithms (saying that ${\bm{\alpha}}_{i}$ is updated at time $t+1$ with probability $q/n$). We call this event "$i$ updated",

Using all these lemmas we can prove our theorem.

In this proof we will consider a constant step-size $\eta_{t}=(\eta_{\bm{\theta}},\eta_{\phi})$. For simplicity of notations we will consider the notation,

where for the second inequality we used Young’s inequality and the Lipchitzness of $F_{j}$ and for the last one we used the co-coercivity of $F_{j}$:

By using the strong convexity of $F$ and Young’s inequality we have that

We finally use the projection-type error bound $\|F_{i}({\bm{\omega}}_{t})-F_{i}({\bm{\omega}}^{*})\|^{2}\geq\gamma_{i}^{2}\|{\bm{\omega}}_{t}-{\bm{\omega}}^{*}\|^{2}$ the same way as (Azizian et al., 2019) to get,

where $\bar{\gamma}^{2}:=\frac{1}{n}\sum_{k=1}^{n}\frac{\gamma_{i}^{2}}{n\pi_{i}}$. We can thus conclude the proof using the strong convexity of $F$,

Appendix D Details on the SVRE–GAN Algorithm

Variance reduction is usually performed on finite sum dataset. However, the noise dataset in GANs (sampling from the noise variable $z$ for the generator $G$) is in practice considered as an infinite dataset. We considered several ways to cope with this:

Infinitely taking new samples from a predefined latent distribution $p_{g}$. In this case, from a theoretical point of view, in terms of using finite sum formulation, there is no convergence guarantee for SVRE even in the strongly convex case. Moreover, the estimators (65) and (66) are biased estimator of the gradient (as ${\bm{\mu}}_{D}$ and ${\bm{\mu}}_{G}$ do not estimate the full expectation but a finite sum).

Sampling a different noise dataset at each epoch, i.e. considering a different finite sum at each epoch. In that case, we are performing a variance reduction of this finite sum over the epoch.

Fix a finite sum noise dataset for the entire training.

In practice, we did not notice any notable difference between the three alternatives.

Particular choices such as the optimization method (e.g. Adam (Kingma and Ba, 2015)), learning rates, and normalization, have been established in practice as almost prerequisite for convergenceFor instance, Daskalakis et al. (2018); Gidel et al. (2019a) plugged Adam into their principled method to get better results., in contrast to supervised classification problems where they have been shown to only provide a marginal value (Wilson et al., 2017). To our knowledge, SVRE is the only method that works with a constant step size for GANs on non-trivial datasets. This combined with the fact that recent works empirically tune the first moment controlling hyperparameter to ($\beta_{1}$, see below) and the variance reduction (VR) one ($\beta_{2}$, see below) to a non-zero value, sheds light on the reason behind the success of Adam on GANs.

However, combining SVRE with adaptive step size scheme on GANs remains an open problem. We first briefly describe the update rule of Adam, and then we propose a new adaptation of it that is more suitable for VR methods, which we refer to as variance reduced Adam (VRAd).

Adam stores an exponentially decaying average of both past gradients $m_{t}$ and squared gradients $v_{t}$, for each parameter of the model:

where $\beta_{1},\beta_{2}\in$, $m_{0}=0$, $v_{0}=0$, and $t=1,\dots T$ denotes the iteration. $m_{t}$ and $v_{t}$ are respectively the estimates of the first and the second moments of the stochastic gradient. To compensate the bias toward due to initialization, Kingma and Ba (2015) propose to use bias-corrected estimates of these first two moments:

The Adam update rule can be described as:

Adam can be understood as an approximate gradient method with a diagonal step size of $\eta_{Adam}:=\frac{\eta}{\sqrt{{\boldsymbol{v}}_{t}}+\epsilon}$. Since VR methods aim to provide a vanishing ${\boldsymbol{v}}_{t}$, they lead to a too large step-size $\eta_{Adam}$ of $\frac{\eta}{{\epsilon}}$. This could indicate that the update rule of Adam may not be a well-suited method to combine with VR methods.

This motivates the introduction of a new Adam-inspired variant of adaptive step sizes that maintain a reasonable size even when ${\boldsymbol{v}}_{t}$ vanishes,

This adaptive variant of Adam is motivated by the step size $\eta^{*}=\eta\frac{{\bm{m}}_{t}^{2}}{{\boldsymbol{v}}_{t}}$ derived by Schaul et al. (2013). (VRAd) is simply the square-root of $\eta^{*}$ in order to stick with Adam’s scaling of ${\boldsymbol{v}}_{t}$.

D.2 SVRE-GAN

In order to cope with the issues introduced by the stochastic game formulation of the GAN models, we proposed the SVRE algorithm Alg. 1 which combines SVRG and extragradient method. We refer to the method of applying SVRE to train GANs as the SVRE-GAN method, and we describe it in detail in Alg. 2 (generalizing it with mini-batching, but using uniform probabilities). Assuming that we have $\mathcal{D}[n_{d}]$ and $\mathcal{Z}[n_{z}]$, respectively two mini-batches of size $B$ of the true dataset and the noise dataset, we compute $\nabla_{D}{\mathcal{L}}^{D}(G,D,\mathcal{D}[n_{d}],\mathcal{Z}[n_{z}])$ and $\nabla_{G}{\mathcal{L}}^{G}(G,D,\mathcal{Z}[n_{z}])$ the respective mini-batches gradient of the discriminator and the generator:

where $\mathcal{Z}_{i}$ and $\mathcal{D}_{j}$ are respectively the $i^{th}$ example of the noise dataset and the $j^{th}$ of the true dataset. Note that $n_{z}$ and $n_{d}$ are lists and thus that we allow repetitions in the summations over $n_{z}$ and $n_{d}$. The variance reduced gradient of the SVRG method are thus given by:

where $G^{\mathcal{S}}$ and $D^{\mathcal{S}}$ are the snapshots and ${\bm{\mu}}_{D}$ and ${\bm{\mu}}_{G}$ their respective gradients.

Note that the double sum in Line $4$ can be written as two sums because of the separability of the expectations in typical GAN objectives. Thus the time complexity for calculating $\mu^{D}$ is still $O(n)$ and not $O(n^{2})$ which would be prohibitively expensive.

Appendix E Restarted SVRE

Alg. 3 describes the restarted version of SVRE presented in § 3.3. With a probability $p$ (fixed) before the computation of ${\bm{\mu}}_{{\bm{\varphi}}}^{\mathcal{S}}$ and ${\bm{\mu}}_{{\bm{\theta}}}^{\mathcal{S}}$, we decide whether to restart SVRE (by using the averaged iterate as the new starting point–Alg. 3, Line $6$–$\bar{\bm{\omega}}_{t}$) or computing the batch snapshot at a point ${\bm{\omega}}_{t}$.

Appendix F Details on the implementation

For our experiments, we used the PyTorchhttps://pytorch.org/ deep learning framework, whereas for computing the FID and IS metrics, we used the provided implementations in Tensorflowhttps://www.tensorflow.org/.

We provide more details about the metrics enumerated in § 4. Both FID and IS use: (i) the Inception v3 network (Szegedy et al., 2015) that has been trained on the ImageNet dataset consisting of ${\sim}1$ million RGB images of $1000$ classes, $C=1000$. (ii) a sample of $m$ generated images $x\sim p_{g}$, where usually $m=50000$.

It aims at estimating (i) if the samples look realistic i.e., $p(y|x)$ should have low entropy, and (ii) if the samples are diverse (from different ImageNet classes) i.e., $p(y)$ should have high entropy. As these are combined using the Kullback–Leibler divergence, the higher the score is, the better the performance. Note that the range of IS scores at convergence varies across datasets, as the Inception network is pretrained on the ImageNet classes. For example, we obtain low IS values on the SVHN dataset as a large fraction of classes are numbers, which typically do not appear in the ImageNet dataset. Since MNIST has greyscale images, we used a classifier trained on this dataset and used $m=5000$. For the rest of the datasets, we used the original implementationhttps://github.com/openai/improved-gan/ of IS in TensorFlow, and $m=50000$.

F.1.2 Fréchet Inception Distance

Contrary to IS, FID aims at comparing the synthetic samples $x\sim p_{g}$ with those of the training dataset $x\sim p_{d}$ in a feature space. The samples are embedded using the first several layers of the Inception network. Assuming $p_{g}$ and $p_{d}$ are multivariate normal distributions, it then estimates the means ${\bm{m}}_{g}$ and ${\bm{m}}_{d}$ and covariances $C_{g}$ and $C_{d}$, respectively for $p_{g}$ and $p_{d}$ in that feature space. Finally, FID is computed as:

where $d^{2}$ denotes the Fréchet Distance. Note that as this metric is a distance, the lower it is, the better the performance. We used the original implementation of FIDhttps://github.com/bioinf-jku/TTUR in Tensorflow, along with the provided statistics of the datasets.

F.1.3 Second Moment Estimate

To evaluate SVRE effectively, we used the second moment estimate (SME, uncentered variance, see § D.1) of the gradient estimate throughout the iterations $t=1\dots T$ per parameter, computed as: $v_{t}=\gamma v_{t-1}+(1-\gamma)g_{t}^{2}$, where $g_{t}$ denotes the gradient estimate for the parameter and iteration $t$, and $\gamma=0.9$. For SVRE, $g_{t}$ is $d_{{\bm{\varphi}}}$ and $d_{{\bm{\theta}}}$ (see Eq. 65 and 66) for $G$ and $D$, respectively. We initialize $g_{0}=0$ and we use bias-corrected estimates: $\hat{v}=\frac{v_{t}}{1-\gamma^{t}}$. As the second moment estimate is computed per each parameter of the model, we depict the average of these values for the parameters of $G$ and $D$ separately.

In this work, as we aim at assessing if SVRE effectively reduces the variance of the gradient updates, we use SME in our analysis as it is computationally inexpensive and fast to compute.

F.1.4 Entropy & Total Variation on MNIST

For the experiments on MNIST illustrated in Fig. 4(a) & 3(b) in § 4, we plot in § G the entropy (E) of the generated samples’ class distribution, as well as the total variation (TV) between the class distribution of the generated samples and a uniform one (both computed using a pretrained network that classifies its $10$ classes).

F.2 Architectures & Hyperparameters

We describe the models we used in the empirical evaluation of SVRE by listing the layers they consist of, as adopted in GAN works, e.g. (Miyato et al., 2018). With “conv.” we denote a convolutional layer and “transposed conv” a transposed convolution layer (Radford et al., 2016). The models use Batch Normalization (Ioffe and Szegedy, 2015) and Spectral Normalization layers (Miyato et al., 2018).

F.2.1 Architectures for experiments on MNIST

For experiments on the MNIST dataset, we used the DCGAN architectures (Radford et al., 2016), listed in Table 3, and the parameters of the models are initialized using PyTorch default initialization. We used mini-batch sizes of $50$ samples, whereas for full dataset passes we used mini-batches of $500$ samples as this reduces the wall-clock time for its computation. For experiments on this dataset, we used the non saturating GAN loss as proposed (Goodfellow et al., 2014):

where $p_{d}$ and $p_{z}$ denote the data and the latent distributions (the latter to be predefined).

For both the baseline and the SVRE variants we tried the following step sizes $\eta=[1\times 10^{-2},$ $1\times 10^{-3},1\times 10^{-4}]$. We observe that SVRE can be used with larger step sizes. In Table 9, we used $\eta=1\times 10^{-4}$ and $\eta=1\times 10^{-2}$ for SE–A and SVRE(–VRAd), respectively.

F.2.2 Choice of architectures on real-world datasets

We replicate the experimental setup described for CIFAR-10 and SVHN in (Miyato et al., 2018), described also below in § F.2.4. We observe that this experimental setup is highly sensitive to the choice of the hyperparameters (see our results in § G.3), making it more difficult to compare the optimization methods for a fixed hyperparameter choice. In particular, apart from the different combinations of learning rates for $G$ and $D$, for the baseline this also included experimenting with: $\beta_{1}$ (see (58)), a multiplicative factor of exponential learning rate decay scheduling $\gamma$, as well as different ratio of updating $G$ and $D$ per iteration. These observations, combined with that we had limited computational resources, motivated us to use shallower architectures, which we describe below in § F.2.3, and which use an inductive bias of so-called Self–Attention layers (Zhang et al., 2018). As a reference, our SAGAN and ResNet architectures for CIFAR-10 have approximately $35$ and $85$ layers, respectively–in total for G and D, including the non linearity and the normalization layers. For clarity, although the deeper and the shallower architectures differ as they are based on ResNet and SAGAN, we refer these as deep (see § F.2.3) and shallow (see § F.2.4), respectively.

F.2.3 Shallower SAGAN architectures

We used the SAGAN architectures (Zhang et al., 2018), as the techniques of self-attention introduced in SAGAN were used to obtain the state-of-art GAN results on ImageNet (Brock et al., 2019). In summary, these architectures: (i) allow for attention-driven, long-range dependency modeling, (ii) use spectral normalization (Miyato et al., 2018) on both $G$ and $D$ (efficiently computed with the power iteration method); and (iii) use different learning rates for $G$ and $D$, as advocated in (Heusel et al., 2017). The foremost is obtained by combining weights, or alternatively attention vectors, with the convolutions across layers, so as to allow modeling textures that are consistent globally–for the generator, or enforcing geometric constraints on the global image structure–for the discriminator.

We used the architectures listed in Table 5 for CIFAR-10 and SVHN datasets, and the architectures described in Table 6 for the experiments on ImageNet. The models’ parameters are initialized using the default initialization of PyTorch.

For experiments with SAGAN, we used the hinge version of the adversarial non-saturating loss (Lim and Ye, 2017; Zhang et al., 2018):

where consistent with the notation above, $p_{d}$ and $p_{z}$ denote the data and the latent distributions.

For the SE–A baseline we obtained best performances when $\eta_{G}=1\times 10^{-4}$ and $\eta_{D}=4\times 10^{-4}$, for G and D, respectively. Similarly as noted for MNIST, using SVRE allows for using larger order of the step size on the rest of the datasets, whereas SE–A with increased step size ($\eta_{G}=1\times 10^{-3}$ and $\eta_{D}=4\times 10^{-3}$ failed to converge. In Table 2, $\eta_{G}=1\times 10^{-3}$, $\eta_{D}=4\times 10^{-3}$, and $\eta_{G}=5\times 10^{-3}$, $\eta_{D}=8\times 10^{-3},\beta_{1}=0.3$ for SVRE and SVRE–VRAd, respectively. We did not use momentum for the vanilla SVRE experiments.

We experimented with ResNet (He et al., 2015) architectures on CIFAR-10 and SVHN, using the architectures listed in Table 8, that replicate the setup described in (Miyato et al., 2018) on CIFAR-10. For experiments with ResNet, we used the hinge version of the adversarial non-saturating loss, Eq. 71 and 72. For this architectures, we refer the reader to § G.3 for details on the hyperparameters, where we list the hyperparameters along with the obtained results.

The results in Table 2 on MNIST are obtained using $5$ runs with different seeds, and the shown performances are the averaged values. Each experiment was run for $100K$ iterations. The corresponding scores with the standard deviations are as follows: (i) IS: $8.62{\pm}.02$, $8.58{\pm}.08$, $8.56{\pm}.11$; (ii) FID: $0.17{\pm}.03$, $0.15{\pm}.01$, $0.18{\pm}.02$; for SE–A, SVRE, and SVRE–VRAd, respectively. On this dataset, we obtain similar final performances if run for many iterations, however SVRE converges faster (see Fig. 3). Fig. 5 illustrates additional metrics of the experiments shown in Fig. 3.

G.2 Results with shallow architectures

Fig. 6 depicts the results on ImageNet using the shallow architectures described in Table 6, § F.2.3. Table 9 summarizes the results obtained on SVHN, CIFAR-10 and ImageNet with these architectures. Fig. 7 depicts the SME metric (see § F.1.3) for the the SE–A baseline and SVRE shown in Fig. 3(c), on SVHN.

G.3 Results with deeper architectures

We observe that GAN training is more challenging when using deeper architectures and some empirical observations differ in the two settings. For example, our stochastic baseline is drastically more unstable and often does not start to converge, whereas SVRE is notably stable, but slower compared to when using shallower architectures. In this section, all our discussions focus on deep architectures (see § F.2.4).

For our stochastic baselines, irrespective whether we use the extragradient or gradient method, we observe that the convergence is notably more unstable (see Fig. 8) when using the deep architectures described in § F.2.4. More precisely, either the training fails to converge or it diverges at later iterations. When updating G and D equal number of times i.e. using $1:1$ update ratio, using SE–A on CIFAR-10 we obtained best FID score of $24.91$ using $\eta_{G}=2\times 10^{-4}$, $\eta_{D}=4\times 10^{-4},\beta_{1}=0$, while experimenting with several combinations of $\eta_{G},\eta_{D},\beta_{1}$. Using exponential learning rate decay with a multiplicative factor of $0.99$, improved the best FID score to $20.70$, obtained for the experiment with $\eta_{G}=2\times 10^{-4}$, $\eta_{D}=2\times 10^{-4},\beta_{1}=0$. Finally, using $1:5$ update ratio, with $\eta_{G}=2\times 10^{-4}$, $\eta_{D}=2\times 10^{-4},\beta_{1}=0$ provided best FID of $18.65$ for the baseline. Figures 7(a) and 7(b) depict the hyper-parameter sensitivity of SE–A and SG–A, respectively. The latter denotes the alternating GAN training with Adam, that is most commonly used for GAN training.

We observe that SVRE is more stable in terms of hyperparameter selection, as it always starts to converge and does not diverge at later iterations. Relative to experiments with shallower architectures, we observe that with deeper architectures SVRE takes longer to converge than its baseline for this architecture. With constant step size of $\eta_{G}=1\times 10^{-3}$, $\eta_{D}=4\times 10^{-3}$ we obtain FID score of $23.56$ on CIFAR-10. Note that this result outperforms the baseline when using no additional tricks (which themselves require additional hyperparameter tuning). Fig. 9 depicts the FID scores obtained when training with SVRE on the SVHN dataset, for two different hyperparameter settings, using four different seeds for each. From this set of experiments, we observe that contrary to the baseline that either did not converge or diverged in all our experiments, SVRE always converges. However, we observe different performances for different seeds. This suggests that more exhaustive empirical hyperparameter search that aims to find an empirical setup that works best for SVRE or further combining SVRE with adaptive step size techniques are both promising research directions (see our discussion below). Fig. 10 depicts our WS–SVRE experiment, where we start from a stored checkpoint for which we obtained best FID score for the SE–A baseline, and we continue the training with SVRE. It is interesting that besides that the baseline diverged after the stored checkpoint, SVRE further reduced the FID score. Moreover, we observe that using different update ratios does not impact much the performance, what on the other hand was necessary to make the baseline algorithm converge.

Fig. 11 depicts the second moment estimate (see § F.1.3) for the experiments with deep architectures. We observe that: (i) the estimated SME quantity is more bounded and changes more smoothly for SVRE (as we do not observe large oscillations of it as it is the case for SE–A); as well as that (ii) divergence of the SE–A baseline correlates with large oscillations of SME, in this case, observed for the Discriminator. Regarding the latter, there exist larger in magnitude oscillations of SME (note that the exponential moving average hyperparameter for computing SME is $\gamma=0.9$, see § F.1.3).

In summary, we observe the following most important advantages of SVRE when using deep architectures: (i) consistency of convergence, and improved stability; as well as (ii) reduced number of hyperparameters. Apart from the practical benefit for applications, the former could allow for a more fair comparison of GAN variants. The latter refers to the fact that SVRE omits the tuning of the sensitive (for the stochastic baseline) $\beta_{1}$ hyperparameter (see (58)), as well as $r$ and $\gamma$–as training converges for SVRE without using different update ratio and step size schedule, respectively. It is important to note that the stochastic baseline does not converge when using constant step size (i.e. when SGD is used instead of Adam). In our experiments we compared SVRE that uses constant step size, with Adam, making the comparison unfair toward SVRE. Hence, our results indicate that SVRE can be further combined with adaptive step size schemes, so as to obtain both stable GAN performances and fast convergence when using these architectures. Nonetheless, the fact that the baseline either does not start to converge or it diverges later makes SVRE and WS–SVRE a promising approach for practitioners using these deep architectures, whereas, for shallower ones, SVRE speeds up the convergence and often provides better final performances.