Multi-Generator Generative Adversarial Nets

Introduction

Generative Adversarial Nets (GANs) (Goodfellow et al., 2014) are a recent novel class of deep generative models that are successfully applied to a large variety of applications such as image, video generation, image inpainting, semantic segmentation, image-to-image translation, and text-to-image synthesis, to name a few (Goodfellow, 2016). From the game theory metaphor, the model consists of a discriminator and a generator playing a two-player minimax game, wherein the generator aims to generate samples that resemble those in the training data whilst the discriminator tries to distinguish between the two as narrated in (Goodfellow et al., 2014). Training GAN, however, is challenging as it can be easily trapped into the mode collapsing problem where the generator only concentrates on producing samples lying on a few modes instead of the whole data space (Goodfellow, 2016).

Many GAN variants have been recently proposed to address this problem. They can be grouped into two main categories: training either a single generator or many generators. Methods in the former include modifying the discriminator’s objective (Salimans et al., 2016; Metz et al., 2016), modifying the generator’s objective (Warde-Farley & Bengio, 2016), or employing additional discriminators to yield more useful gradient signals for the generators (Nguyen et al., 2017; Durugkar et al., 2016). The common theme in these variants is that generators are shown, at equilibrium, to be able to recover the data distribution, but convergence remains elusive in practice. Most experiments are conducted on toy datasets or on narrow-domain datasets such as LSUN (Yu et al., 2015) or CelebA (Liu et al., 2015). To our knowledge, only Warde-Farley & Bengio (2016) and Nguyen et al. (2017) perform quantitative evaluation of models trained on much more diverse datasets such as STL-10 (Coates et al., 2011) and ImageNet (Russakovsky et al., 2015).

Given current limitations in the training of single-generator GANs, some very recent attempts have been made following the multi-generator approach. Tolstikhin et al. (2017) apply boosting techniques to train a mixture of generators by sequentially training and adding new generators to the mixture. However, sequentially training many generators is computational expensive. Moreover, this approach is built on the implicit assumption that a single-generator GAN can generate very good images of some modes, so reweighing the training data and incrementally training new generators will result in a mixture that covers the whole data space. This assumption is not true in practice since current single-generator GANs trained on diverse datasets such as ImageNet tend to generate images of unrecognizable objects. Arora et al. (2017) train a mixture of generators and discriminators, and optimize the minimax game with the reward function being the weighted average reward function between any pair of generator and discriminator. This model is computationally expensive and lacks a mechanism to enforce the divergence among generators. Ghosh et al. (2017) train many generators by using a multi-class discriminator that, in addition to detecting whether a data sample is fake, predicts which generator produces the sample. The objective function in this model punishes generators for generating samples that are detected as fake but does not directly encourage generators to specialize in generating different types of data.

We propose in this paper a novel approach to train a mixture of generators. Unlike aforementioned multi-generator GANs, our proposed model simultaneously trains a set of generators with the objective that the mixture of their induced distributions would approximate the data distribution, whilst encouraging them to specialize in different data modes. The result is a novel adversarial architecture formulated as a minimax game among three parties: a classifier, a discriminator, and a set of generators. Generators create samples that are intended to come from the same distribution as the training data, whilst the discriminator determines whether samples are true data or generated by generators, and the classifier specifies which generator a sample comes from. We term our proposed model as Mixture Generative Adversarial Nets (MGAN). We provide analysis that our model is optimized towards minimizing the Jensen-Shannon Divergence (JSD) between the mixture of distributions induced by the generators and the data distribution while maximizing the JSD among generators.

Empirically, our proposed model can be trained efficiently by utilizing parameter sharing among generators, and between the classifier and the discriminator. In addition, simultaneously training many generators while enforcing JSD among generators helps each of them focus on some modes of the data space and learn better. Trained on CIFAR-10, each generator learned to specialize in generating samples from a different class such as horse, car, ship, dog, bird or airplane. Overall, the models trained on the CIFAR-10, STL-10 and ImageNet datasets successfully generated diverse, recognizable objects and achieved state-of-the-art Inception scores (Salimans et al., 2016). The model trained on the CIFAR-10 even outperformed GANs trained in a semi-supervised fashion (Salimans et al., 2016; Odena et al., 2016).

In short, our main contributions are: (i) a novel adversarial model to efficiently train a mixture of generators while enforcing the JSD among the generators; (ii) a theoretical analysis that our objective function is optimized towards minimizing the JSD between the mixture of all generators’ distributions and the real data distribution, while maximizing the JSD among generators; and (iii) a comprehensive evaluation on the performance of our method on both synthetic and real-world large-scale datasets of diverse natural scenes.

Generative Adversarial Nets

Given the discriminator $D$ and generator $G$, both parameterized via neural networks, training GAN can be formulated as the following minimax objective function:

where $\mathbf{x}$ is drawn from data distribution $P_{data}$, $\mathbf{z}$ is drawn from a prior distribution $P_{\mathbf{z}}$. The mapping $G\left(\mathbf{z}\right)$ induces a generator distribution $P_{model}$ in data space. GAN alternatively optimizes $D$ and $G$ using stochastic gradient-based learning. As a result, the optimization order in 1 can be reversed, causing the minimax formulation to become maximin. $G$ is therefore incentivized to map every $\mathbf{z}$ to a single $\mathbf{x}$ that is most likely to be classified as true data, leading to mode collapsing problem. Another commonly asserted cause of generating less diverse samples in GAN is that, at the optimal point of $D$, minimizing $G$ is equivalent to minimizing the JSD between the data and model distributions, which has been empirically proven to prefer to generate samples around only a few modes whilst ignoring other modes (Huszár, 2015; Theis et al., 2015).

Proposed Mixture GANs

We now present our main contribution of a novel approach that can effectively tackle mode collapse in GAN. Our idea is to use a mixture of many distributions rather than a single one as in the standard GAN, to approximate the data distribution, and simultaneously we enlarge the divergence of those distributions so that they cover different data modes.

Fig. 1 illustrates the general architecture of our proposed MGAN, where all components are parameterized by neural networks. $G_{k}$ (s) tie their parameters together except the input layer, whilst $C$ and $D$ share parameters except the output layer. This parameter sharing scheme enables the networks to leverage their common information such as features at low-level layers that are close to the data layer, hence helps to train model effectively. In addition, it also minimizes the number of parameters and adds minimal complexity to the standard GAN, thus the whole process is still very efficient.

It can be seen that the solution $C_{k}^{*}$ is a general case of $D^{*}$ when $D$ classifies samples from two distributions with equal weight of $\nicefrac{{1}}{{2}}$. We refer the proofs for $D^{*}$ to Prop. 1 in (Goodfellow et al., 2014), and our proof for $C_{k}^{*}$ to Appendix B in this manuscript. ∎

At the equilibrium point of the minimax problem in Eq. (2), the optimal $G^{*},D^{*}$, and $C^{*}$ satisfy

Since the last two terms in Eq. (4) are constant, that concludes our proof. ∎

Please refer to our proof in Appendix B of this manuscript. ∎

Related Work

Recent attempts to address the mode collapse by modifying the discriminator include minibatch discrimination (Salimans et al., 2016), Unrolled GAN (Metz et al., 2016) and Denoising Feature Matching (DFM) (Warde-Farley & Bengio, 2016). The idea of minibatch discrimination is to allow the discriminator to detect samples that are noticeably similar to other generated samples. Although this method can generate visually appealing samples, it is computationally expensive, thus normally used in the last hidden layer of discriminator. Unrolled GAN improves the learning by unrolling computational graph to include additional optimization steps of the discriminator. It could effectively reduce the mode collapsing problem, but the unrolling step is expensive, rendering it unscalable up to large-scale datasets. DFM augments the objective function of generator with one of a Denoising AutoEncoder (DAE) that minimizes the reconstruction error of activations at the penultimate layer of the discriminator. The idea is that gradient signals from DAE can guide the generator towards producing samples whose activations are close to the manifold of real data activations. DFM is surprisingly effective at avoiding mode collapse, but the involvement of a deep DAE adds considerable computational cost to the model.

An alternative approach is to train additional discriminators. D2GAN (Nguyen et al., 2017) employs two discriminators to minimize both Kullback-Leibler (KL) and reverse KL divergences, thus placing a fair distribution across the data modes. This method can avoid the mode collapsing problem to a certain extent, but still could not outperform DFM. Another work uses many discriminators to boost the learning of generator (Durugkar et al., 2016). The authors state that this method is robust to mode collapse, but did not provide experimental results to support that claim.

Another direction is to train multiple generators. The so-called MIX+GAN (Arora et al., 2017) is related to our model in the use of mixture but the idea is very different. Based on min-max theorem (Neumann, 1928), the MIX+GAN trains a mixture of multiple generators and discriminators with different parameters to play mixed strategies in a min-max game. The total reward of this game is computed by weighted averaging rewards over all pairs of generator and discriminator. The lack of parameter sharing renders this method computationally expensive to train. Moreover, there is no mechanism to enforce the divergence among generators as in ours.

Some attempts have been made to train a mixture of GANs in a similar spirit with boosting algorithms. Wang et al. (2016) propose an additive procedure to incrementally train new GANs on a subset of the training data that are badly modeled by previous generators. As the discriminator is expected to classify samples from this subset as real with high confidence, i.e. $D\left(\mathbf{x}\right)$ is high, the subset can be chosen to include $\mathbf{x}$ where $D\left(\mathbf{x}\right)$ is larger than a predefined threshold. Tolstikhin et al. (2017), however, show that this heuristic fails to address the mode collapsing problem. Thus they propose AdaGAN to introduce a robust reweighing scheme to prepare training data for the next GAN. AdaGAN and boosting-inspired GANs in general are based on the assumption that a single-generator GAN can learn to generate impressive images of some modes such as dogs or cats but fails to cover other modes such as giraffe. Therefore, removing images of dogs or cats from the training data and train a next GAN can create a better mixture. This assumption is not true in practice as current single-generator GANs trained on diverse data sets such as ImageNet (Russakovsky et al., 2015) tend to generate images of unrecognizable objects.

The most closely related to ours is MAD-GAN (Ghosh et al., 2017) which trains many generators and uses a multi-class classifier as the discriminator. In this work, two strategies are proposed to address the mode collapse: (i) augmenting generator’s objective function with a user-defined similarity based function to encourage different generators to generate diverse samples, and (ii) modifying discriminator’s objective functions to push different generators towards different identifiable modes by separating samples of each generator. Our approach is different in that, rather than modifying the discriminator, we use an additional classifier that discriminates samples produced by each generator from those by others under multi-class classification setting. This nicely results in an optimization problem that maximizes the JSD among generators, thus naturally enforcing them to generate diverse samples and effectively avoiding mode collapse.

Experiments

In this section, we conduct experiments on both synthetic data and real-world large-scale datasets. The aim of using synthetic data is to visualize, examine and evaluate the learning behaviors of our proposed MGAN, whilst using real-world datasets to quantitatively demonstrate its efficacy and scalability of addressing the mode collapse in a much larger and wider data space. For fair comparison, we use experimental settings that are identical to previous work, and hence we quote the results from the latest state-of-the-art GAN-based models to compare with ours.

We use TensorFlow (Abadi et al., 2016) to implement our model and will release the code after publication. For all experiments, we use: (i) shared parameters among generators in all layers except for the weights from the input to the first hidden layer; (ii) shared parameters between discriminator and classifier in all layers except for the weights from the penultimate layer to the output; (iii) Adam optimizer (Kingma & Ba, 2014) with learning rate of 0.0002 and the first-order momentum of 0.5; (iv) minibatch size of 64 samples for training discriminators; (v) ReLU activations (Nair & Hinton, 2010) for generators; (vi) Leaky ReLU (Maas et al., 2013) with slope of 0.2 for discriminator and classifier; and (vii) weights randomly initialized from Gaussian distribution $\mathcal{N}(0,0.02\boldsymbol{I})$ and zero biases. We refer to Appendix C for detailed model architectures and additional experimental results.

In the first experiment, following (Nguyen et al., 2017) we reuse the experimental design proposed in (Metz et al., 2016) to investigate how well our MGAN can explore and capture multiple data modes. The training data is sampled from a 2D mixture of 8 isotropic Gaussian distributions with a covariance matrix of 0.02$\boldsymbol{I}$ and means arranged in a circle of zero centroid and radius of 2.0. Our purpose of using such small variance is to create low density regions and separate the modes.

We employ 8 generators, each with a simple architecture of an input layer with 256 noise units drawn from isotropic multivariate Gaussian distribution $\mathcal{N}\left(0,\boldsymbol{I}\right)$, and two fully connected hidden layers with 128 ReLU units each. For the discriminator and classifier, one hidden layer with 128 ReLU units is used. The diversity hyperparameter $\beta$ is set to 0.125.

Fig. 2c shows the evolution of 512 samples generated by our model and baselines through time. It can be seen that the regular GAN generates data collapsing into a single mode hovering around the valid modes of data distribution, thus reflecting the mode collapse in GAN as expected. At the same time, UnrolledGAN (Metz et al., 2016), D2GAN (Nguyen et al., 2017) and our MGAN distribute data around all 8 mixture components, and hence demonstrating the abilities to successfully learn multimodal data in this case. Our proposed model, however, converges much faster than the other two since it successfully explores and neatly covers all modes at the early step 15K, whilst two baselines produce samples cycling around till the last steps. At the end, our MGAN captures data modes more precisely than UnrolledGAN and D2GAN since, in each mode, the UnrolledGAN generates data that concentrate only on several points around the mode’s centroid, thus seems to produce fewer samples than ours whose samples fairly spread out the entire mode, but not exceed the boundary whilst the D2GAN still generates many points scattered between two adjacent modes.

Next we further quantitatively compare the quality of generated data. Since we know the true distribution $P_{data}$ in this case, we employ two measures, namely symmetric Kullback-Leibler (KL) divergence and Wasserstein distance. These measures compute the distance between the normalized histograms of 10,000 points generated from the model to true $P_{data}$. Figs. 2a and 2b again clearly demonstrate the superiority of our approach over GAN, UnrolledGAN and D2GAN w.r.t both distances (lower is better); notably the Wasserstein distances from ours and D2GAN’s to the true distribution almost reduce to zero, and at the same time, our symmetric KL metric is significantly better than that of D2GAN. These figures also show the stability of our MGAN (black curves) and D2GAN (red curves) during training as they are much less fluctuating compared with GAN (green curves) and UnrolledGAN (blue curves).

Lastly, we perform experiments with different numbers of generators. The MGAN models with 2, 3, 4 and 10 generators all successfully explore 8 modes but the models with more generators generate fewer points scattered between adjacent modes. We also examine the behavior of the diversity coefficient $\beta$ by training the 4-generator model with different values of $\beta$. Without the JSD force ($\beta=0$), generated samples cluster around one mode. When $\beta=0.25$, the JSD force is weak and generated data cluster near 4 different modes. When $\beta=0.75$ or $1.0$, the JSD force is too strong and causes the generators to collapse, generating 4 increasingly tight clusters. When $\beta=0.5$, generators successfully cover all of the 8 modes. Please refer to Appendix C.1 for experimental details.

2 Real-world Datasets

Next we train our proposed method on real-world databases from natural scenes to investigate its performance and scalability on much more challenging large-scale image data.

We use 3 widely-adopted datasets: CIFAR-10 (Krizhevsky & Hinton, 2009), STL-10 (Coates et al., 2011) and ImageNet (Russakovsky et al., 2015). CIFAR-10 contains 50,000 32$\times$32 training images of 10 classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck. STL-10, subsampled from ImageNet, is a more diverse dataset than CIFAR-10, containing about 100,000 96$\times$96 images. ImageNet (2012 release) presents the largest and most diverse consisting of over 1.2 million images from 1,000 classes. In order to facilitate fair comparison with the baselines in (Warde-Farley & Bengio, 2016; Nguyen et al., 2017), we follow the procedure of (Krizhevsky et al., 2012) to resize the STL-10 and ImageNet images down to 48$\times$48 and 32$\times$32, respectively.

Evaluation protocols.

Model architectures.

Our generator and discriminator architectures closely follow the DCGAN’s design (Radford et al., 2015). The only difference is we apply batch normalization (Ioffe & Szegedy, 2015) to all layers in the networks except for the output layer. Regarding the classifier, we empirically find that our proposed MGAN achieves the best performance (i.e., fast convergence rate and high inception score) when the classifier shares parameters of all layers with the discriminator except for the output layer. The reason is that this parameter sharing scheme would allow the classifier and discriminator to leverage their common features and representations learned at every layer, thus helps to improve and speed up the training progress. When the parameters are not tied, the model learns slowly and eventually yields lower performance.

During training we observe that the percentage of active neurons chronically declined (see Appendix C.2). One possible cause is that the batch normalization center (offset) is gradually shifted to the negative range, thus deactivating up to 45% of ReLU units of the generator networks. Our ad-hoc solution for this problem is to fix the offset at zero for all layers in the generator networks. The rationale is that for each feature map, the ReLU gates will open for about 50% highest inputs in a minibatch across all locations and generators, and close for the rest.

We also experiment with other activation functions of generator networks. First we use Leaky ReLU and obtain similar results with using ReLU. Then we use MaxOut units (Goodfellow et al., 2013) and achieves good Inception scores but generates unrecognizable samples. Finally, we try SeLU (Klambauer et al., 2017) but fail to train our model.

Hyperparameters.

Inception results.

We now report the Inception scores obtained by our MGAN and baselines in Tab. 1. It is worthy to note that only models trained in a completely unsupervised manner without label information are included for fair comparison; and DCGAN’s and D2GAN’s results on STL-10 are available only for the models trained on 32$\times$32 resolution. Overall, our proposed model outperforms the baselines by large margins and achieves state-of-the-art performance on all datasets. Moreover, we would highlight that our MGAN obtains a score of 8.33 on CIFAR-10 that is even better than those of models trained with labels such as 8.09 of Improved GAN (Salimans et al., 2016) and 8.25 of AC-GAN (Odena et al., 2016). In addition, we train our model on the original 96$\times$96 resolution of STL-10 and achieve a score of 9.79$\pm$0.08. This suggests the MGAN can be successfully trained on higher resolution images and achieve the higher Inception score.

Image generation.

Next we present samples randomly generated by our proposed model trained on the 3 datasets for qualitative assessment. Fig. 3a shows CIFAR-10 32$\times$32 images containing a wide range of objects in such as airplanes, cars, trucks, ships, birds, horses or dogs. Similarly, STL-10 48$\times$48 generated images in Fig. 3b include cars, ships, airplanes and many types of animals, but with wider range of different themes such as sky, underwater, mountain and forest. Images generated for ImageNet 32$\times$32 are diverse with some recognizable objects such as lady, old man, birds, human eye, living room, hat, slippers, to name a few. Fig. 4a shows several cherry-picked STL-10 96$\times$96 images, which demonstrate that the MGAN is capable of generating visually appealing images with complicated details. However, many samples are still incomplete and unrealistic as shown in Fig. 4b, leaving plenty of room for improvement.

Finally, we investigate samples generated by each generator as well as the evolution of these samples through numbers of training epochs. Fig. 5 shows images generated by each of the 10 generators in our MGAN trained on CIFAR-10 at epoch 20, 50, and 250 of training. Samples in each row correspond to a different generator. Generators start to specialize in generating different types of objects as early as epoch 20 and become more and more consistent: generator 2 and 3 in flying objects (birds and airplanes), generator 4 in full pictures of cats and dogs, generator 5 in portraits of cats and dogs, generator 8 in ships, generator 9 in car and trucks, and generator 10 in horses. Generator 6 seems to generate images of frog or animals in a bush. Generator 7, however, collapses in epoch 250. One possible explanation for this behavior is that images of different object classes tend to have different themes. Lastly, Wang et al. (2016) noticed one of the causes for non-convergence in GANs is that the generators and discriminators constantly vary; the generators at two consecutive epochs of training generate significantly different images. This experiment demonstrates the effect of the JSD force in preventing generators from moving around the data space.

Conclusion

We have presented a novel adversarial model to address the mode collapse in GANs. Our idea is to approximate data distribution using a mixture of multiple distributions wherein each distribution captures a subset of data modes separately from those of others. To achieve this goal, we propose a minimax game of one discriminator, one classifier and many generators to formulate an optimization problem that minimizes the JSD between $P_{data}$ and $P_{model}$, i.e., a mixture of distributions induced by the generators, whilst maximizes JSD among such generator distributions. This helps our model generate diverse images to better cover data modes, thus effectively avoids mode collapse. We term our proposed model Mixture Generative Adversarial Network (MGAN).

The MGAN can be efficiently trained by sharing parameters between its discriminator and classifier, and among its generators, thus our model is scalable to be evaluated on real-world large-scale datasets. Comprehensive experiments on synthetic 2D data, CIFAR-10, STL-10 and ImageNet databases demonstrate the following capabilities of our model: (i) achieving state-of-the-art Inception scores; (ii) generating diverse and appealing recognizable objects at different resolutions; and (iv) specializing in capturing different types of objects by the generators.

References

Appendix A Appendix: Framework

Appendix B Appendix: Proofs for Section 3.1

Replacing the optimal $C^{*}$ and $D^{*}$ into Eq. (2), we can reformulate the objective function for the generator as follows:

Theorem 3 (Thm. 3 restated).

If the data distribution has the form: $p_{data}\left(\mathbf{x}\right)=\sum_{k=1}^{K}\pi_{k}q_{k}\left(\mathbf{x}\right)$ where the mixture components $q_{k}\left(\mathbf{x}\right)$(s) are well-separated, the minimax problem in Eq. (2) or the optimization problem in Eq. (3) has the following solution:

We first recap the optimization problem for finding the optimal $G^{*}$:

The $i$-th expectation in Eq. (9) can be derived as follows:

Therefore, we obtain the following inequality:

and the equality occurs if for almost every $\mathbf{x}$ except for those in a zero measure set, we have:

and we peak the minimum if $p_{G_{k}}=q_{k},\,\forall k$ since this solution satisfies both

and the conditions depicted in Eq. (10). That concludes our proof.∎

Appendix C Appendix: Additional Experiments

The true data is sampled from a 2D mixture of 8 Gaussian distributions with a covariance matrix 0.02$\boldsymbol{I}$ and means arranged in a circle of zero centroid and radius 2.0. We use a simple architecture of 8 generators with two fully connected hidden layers and a classifier and a discriminator with one shared hidden layer. All hidden layers contain the same number of 128 ReLU units. The input layer of generators contains 256 noise units sampled from isotropic multivariate Gaussian distribution $\mathcal{N}\left(0,\boldsymbol{I}\right)$. We do not use batch normalization in any layer. We refer to Tab. 2 for more specifications of the network and hyperparameters. “Shared” is short for parameter sharing among generators or between the classifier and the discriminator. Feature maps of 8/1 in the last layer for $C$ and $D$ means that two separate fully connected layers are applied to the penultimate layer, one for $C$ that outputs $8$ logits and another for $D$ that outputs $1$ logit.

Fig. 6 shows samples produced by MGANs with different numbers of generators trained on synthetic data for 25,000 epochs. The model with 1 generator behaves similarly to the standard GAN as expected. The models with 2, 3 and 4 generators all successfully cover 8 modes, but the ones with more generators draw fewer points scattered between adjacent modes. Finally, the model with 10 generators also covers 8 modes wherein 2 generators share one mode and one generator hovering around another mode.

The effect of β𝛽\beta on generated samples.

To examine the behavior of the diversity coefficient $\beta$, Fig. 7 compares samples produced by our MGAN with 4 generators after 25,000 epochs of training with different values of $\beta$. Without the JSD force ($\beta=0$), generated samples cluster around one mode. When $\beta=0.25$, generated data clusters near 4 different modes. When $\beta=0.75$ or $1.0$, the JSD force is too strong and causes the generators to collapse, generating 4 increasingly tight clusters. When $\beta=0.5$, generators successfully cover all of the 8 modes.

C.2 Real-world datasets

During training we observe that the percentage of active neurons, which we define as ReLU units with positive activation for at least 10% of samples in the minibatch, chronically declined. Fig. 8a shows the percentage of active neurons in generators trained on CIFAR-10 declined consistently to 55% in layer 2 and 60% in layer 3. Therefore, the quality of generated images, after reaching the peak level, started declining. One possible cause is that the batch normalization center (offset) is gradually shifted to the negative range as shown in the histogram in Fig. 8b. We also observe the same problem in DCGAN. Our ad-hoc solution for this problem, i.e., we fix the offset at zero for all layers in the generator networks. The rationale is that for each feature map, the ReLU gates will open for about 50% highest inputs in a minibatch across all locations and generators, and close for the rest. Therefore, batch normalization can keep ReLU units alive even when most of their inputs are otherwise negative, and introduces a form of competition that encourages generators to “specialize” in different features. This measure significantly improves performance but does not totally solve the dying ReLUs problem. We find that late in the training, the input to generators’ ReLU units became more and more right-skewed, causing the ReLU gates to open less and less often.

Experiment settings.

For the experiments on three large-scale natural scene datasets (CIFAR-10, STL-10, ImageNet), we closely followed the network architecture and training procedure of DCGAN. The specifications of our models trained on CIFAR-10, STL-10 48$\times$48, STL-10 96$\times$96 and ImageNet datasets are described in Tabs. (3, 4, 5, 6), respectively. “BN” is short for batch normalization and “BN center” is short for whether to learn batch normalization’s center or set it at zero. “Shared” is short for parameter sharing among generators or between the classifier and the discriminator. Feature maps of 10/1 in the last layer for $C$ and $D$ means that two separate fully connected layers are applied to the penultimate layer, one for $C$ that outputs $10$ logits and another for $D$ that outputs $1$ logit. Finally, Figs. (9, 10, 11, 12, 13) respectively are the enlarged version of Figs. (3a, 3b, 3c, 4a, 4b) in the main manuscript.