Dropout-GAN: Learning from a Dynamic Ensemble of Discriminators

Introduction

Generative adversarial networks , or GAN, is a framework that integrates adversarial training in the generative modeling process. According to its original proposal , the framework is composed of two models - one generator and one discriminator - that train together by playing a minimax game. While the generator tries to fool the discriminator by producing fake samples that look realistic, the discriminator tries to distinguish between real and fake samples better over time, making it harder to be fooled by the generator.

However, one of the main problems with GAN is mode collapse , where the generator is able to fool the discriminator by only producing data coming from the same data mode, i.e., connected components of the data manifold. This leads to a poor generator that is only able to produce samples within a narrow scope of the data space, resulting in the generation of only similarly looking samples. Hence, at the end of training, the generator comes short regarding learning the full data distribution, and, instead, is only able to learn a small segment of it. This is the main issue we try to tackle in this work.

In a disparate line of work, dropout was introduced by and it has been proven to be a very useful and widely used technique in neural networks to prevent overfitting . In practice, it simply consists of omitting or dropping out, the output of some randomly chosen neurons with a probability $d$ or dropout rate. The intuition behind this process is to ensure that neurons are not entirely dependent on a specific set of other neurons to produce their outputs. Instead, with dropout, each neuron relies on the population behavior of several other neurons, promoting generalization in the network. Hence, the overall network becomes more flexible and less prone to overfitting.

The main idea of this work consists of applying the same dropout principles to generative multi-adversarial networks. This is accomplished by taking advantage of multiple adversarial training, where the generator’s output is dependent on the feedback given by a specific set of discriminators. By applying dropout on the feedback of each discriminator, we force the generator to not rely on a specific discriminator or discriminator ensemble to learn how to produce realistic samples. Thus, the generator guides its learning from the varied feedback given by a dynamic ensemble of discriminators that changes at every batch.

In our use case, one can then see mode collapse as a consequence of overfitting to the feedback of a single discriminator, or even a static ensemble of discriminators. Hence, by dynamically changing the adversarial ensemble at every batch, the generator is stimulated to induce variety in its output to increase the chances of fooling the different possible discriminators that may remain in the ensemble at the end. Our main contributions can be stated as follows:

We propose a novel and generic framework, named Dropout-GAN (Section 3), that trains a single generator against a dynamically changing ensemble of discriminators.

We provide useful discussions and insights regarding the benefits of multiple adversarial training in GAN, namely the increase training stability (Section 4).

We test our method on several datasets and multiple metrics, showing that it succeeds in reducing mode collapse by promoting sample diversity within epochs (Sections 5 and 6).

We show that proposed approach of applying adversarial dropout also improves several other GAN approaches on several metrics and datasets of different size and nature (Section 7), confirming the extensibility of our framework.

Generative Adversarial Networks

As originally proposed , the standard GAN framework consists of two different models: a generator ($G$), that tries to capture the real data distribution to generate fake samples that look realistic, and a discriminator ($D$), that tries to do a better job at distinguishing real and fake samples. $G$ maps a latent space to the data space by receiving noise as input and applying transformations to it to generate unseen samples, while $D$ maps a given sample to a probability $p$ of it coming from the real data distribution.

In the ideal setting, given enough iterations, $G$ would eventually start producing samples that look so realistic that $D$ would not be able to distinguish between real and fake samples anymore. Hence, $D$ would assign $p=0.5$ to all samples, reaching a full state of confusion. However, due to training instability, this equilibrium is hard to reach in practice. The two models play the following minimax game:

where $p_{z}(z)$ represents the noise distribution used to sample $G$’s input and $G(z)$ represents its output, which can be considered as a fake sample originated from mapping the modified input noise to the data space. On the other hand, $p_{r}(x)$ represents the real data distribution and $D(x)$ represents the output of $D$, i.e., the probability $p$ of sample $x$ being a real sample from the training set.

In order to maximize Eq. 1, $D$’s goal is then to maximize the probability of correctly classifying a sample as real or fake by getting better at distinguishing such cases by assigning $p$ close to 1 to real images and $p$ close to 0 to generated images. By contrast, to minimize Eq. 1, $G$ tries to minimize the probability of its generated samples being considered as fake by $D$, through fooling $D$ into assigning them a $p$ value close to 1.

However, in practice $\log(1-D(G(z)))$ might saturate due vanishing gradient problems in the beginning of training caused by $D$ being able to easily distinguish between real and fake samples. As a workaround, the authors propose to maximize $\log(D(G(z)))$ instead, making it no longer a minimax game. Nevertheless, $G$ still continues to exploit $D$’s weaknesses in distinguishing real and fake samples by using $D$’s feedback to update its parameters and slightly change its output to more likely trick $D$ in the next iterations.

Dropout-GAN

We propose to integrate adversarial feedback dropout in generative multi-adversarial networks, forcing $G$ to appease and learn from a dynamic ensemble of discriminators. This ultimately encourages $G$ to produce samples from a variety of modes, since it now needs to fool the different possible discriminators that may remain in the ensemble. Variations in the ensemble are achieved by dropping out the feedback of each $D$ with a certain probability $d$ at the end of every batch. This means that $G$ will only consider the loss of the remaining discriminators in the ensemble while updating its parameters at each iteration. Figure 1 illustrates the proposed framework.

Our initial modification to the value function $V$ of the minimax game is presented in equation (2), where $\delta_{k}$ is a Bernoulli variable ($\delta_{k}\sim Bern(1-d)$) and $\small\{D_{k}\small\}$ is the set of $K$ total discriminators. The gradients calculated from the loss of a given discriminator $D_{k}$, are only used for the calculation of $G$’s final gradient updates when $\delta_{k}=1$, with $P(\delta_{k}=1)=1-d$. Otherwise, this information is discarded:

There is, however, the possibility of all discriminators being dropped out from the set, leaving $G$ without any guidance on how to further update its parameters. In this case, we randomly pick one discriminator D_{j}\in\big{\{}D_{k}\big{\}} and follow the original objective function presented in equation (1), using solely the gradient updates related to $D_{j}$’s loss to update $G$. Hence, taking into consideration this special case, our final value function, $F$, is set as follows:

It is important to note that each discriminator trains independently, i.e., is not aware of the existence of the other discriminators, since no changes were made on their individual gradient updates. This implies that even if dropped out, each $D$ updates its parameters at the end of every batch. The detailed algorithm of the proposed solution can be found in Algorithm 1.

Implementation Details

In this section, we provide a detailed study of the effects of using a different number of discriminators together with different dropout rates. Moreover, we further provide insights into the consequence of splitting the batch among the different discriminators on the generator’s training. The proposed framework was implemented using Tensorflow .

Training instability has been noticeably reported as one of GAN biggest problems. Here, we show that this problem can be eased by using multiple adversaries. This is also stated in previous works , however, without much detailed evidence. Furthermore, on top of increasing training stability, using multiple discriminators enables the usage of the original $G$ loss, since there is now an increased chance that $G$ receives positive feedback from at least one $D$ and is able to guide its learning successfully .

To analyze the training procedure, we correlate the degree of training instability with the gradient updates that are being used by $G$ to update its parameters at the end of each batch. The intuition is that if such updates are big, the parameters of the model will change drastically at each iteration. This is intuitively an alarming sign that the training is not being efficient, especially if it still occurs after several epochs of training, since $G$ is repeatedly greatly updating its output, instead of performing slight, mild changes in a controlled fashion.

We found that when using multiple discriminators such gradients would converge to zero as training progressed, while, on the contrary, they remained high (in terms of its absolute value) when using solely one discriminator. On the other hand, we also noticed that as the number of discriminators increases, the point at which $G$’s gradients start to converge also increases, suggesting that using more discriminators can delay the learning process. However, this is expected since $G$ now receives more (and possibly contradictory) feedback regarding its generated samples, needing more time to utilize such information wisely.

2 Batch Partitioning

The main purpose of splitting the batch among the different discriminators is to encourage each to specialize in different data modes. This is achieved by training them with a different subset of samples of the same size within each batch. This applies to both the fake samples produce by $G$ and real samples retrieved from the training set. Such partitioning also allows data parallelism, diminishing the overhead caused by using more discriminators in the framework.

To further investigate the success in forcing the different discriminators to focus on different data modes, we argue that $G$’s capacity of fooling the ensemble should decrease in such situation. This is indeed confirmed in our experiments, with $G$’s loss being higher when the batches are split, especially later on in training where each $D$ had enough time to focus on a single or a small subset of data modes. Thus, one can then associate the higher $G$ loss with the generated samples now having to comply with a higher number of realistic features to be able to fool the dynamic ensemble of discriminators, with a subset of such features being used by each $D$ to characterize a given sample as real or fake.

We increase the overall batch size to enable each $D$ to be trained on the same original number of samples at each batch. On the other hand, $G$ might still have access to all samples at each batch, since it uses the feedback from the remaining discriminators to update its parameters at the end. However, having weaker discriminators by training each one of them with fewer samples than $G$ is not necessarily bad since they are more likely to give positive feedback to $G$ . This is a result of their possible confused state that can better aid $G$ in producing realistic samples than if it would continuously receive negative feedback, especially in the long run.

3 Dropout Rate

Dropping out the loss of a given $D$ with a probability $d$ before updating $G$’s parameters is what induces variability in our framework. This forces $G$ not to only need to fool one or even a static set of discriminators, but, instead, to fool a dynamic ensemble of adversaries that changes at every batch. Hence, performing this type of dropout can also be seen as a form of regularization, since it aims to promote more generalizability on the fake samples produced by G.

Depending on the number of discriminators used, using a small probability $d$ of dropout might only lead to small changes in the ensemble of adversaries, making the feedback seen by $G$ nearly constant throughout every batch. On the other hand, using a large dropout probability might lead to too much variance in the ensemble, making it difficult for $G$ to learn properly due to the variability of the visible set.

Evidence of the correlation between the dropout rate and the quality of the generated samples is further given in Sections 5 and 6. Similarly to what was discussed in the original Dropout paper , we found that using $d=0.2$ and $d=0.5$ often led to better results, both in a qualitative and quantitative manner. Nevertheless, we also found that using any dropout rate ($0<d\leq 1$) consistently performed better across the different datasets than when using a static ensemble of adversaries ($d=0$).

Experimental Results

We tested the effects of the different parameter settings on three different datasets: MNIST , CIFAR-10 , and CelebA . We compared all possible combinations by using the different number of discriminators across the set $\small\{1,2,5,10\small\}$ with each different dropout rate in $\small\{0.0,0.2,0.5,0.8,1.0\small\}$. We used the DCGAN inspired architecture used in GMAN , with $G$ consisting of 4 convolutional layers of decreasing neuron size, e.g., 128, 64, 32, 1 (for MNIST) or 3 (for CIFAR-10 and CelebA), and each $D$ having 3 convolutional layers of increasing number of neurons, e.g., 32, 64, 128, and a fully connected layer at the end. We refer to GMAN for more information regarding the training settings. Important to note that, even though all discriminators share the same architectures, their weights are initialized differently. Results are reported below for each dataset.

MNIST is composed of 10 different classes of handwritten digits varying from 0 to 9, with the generated samples of Dropout-GAN being shown in Figure 2.

It is visible that the quality and variation of the produced samples increase while using dropout rate values of 0.2 and 0.5 across all different sized discriminator sets. On the other hand, the quality of the produced numbers deteriorates considerably while using high dropout rates, i.e., 0.8 and 1, or no dropout rate at all. However, the quality gets slightly better when using more discriminators on such extreme end dropout rates, since $G$ might still get enough feedback to be able to learn at the end of each batch.

2 CIFAR-10

To further validate our solution, we used the CIFAR-10 dataset also composed of 10 classes, consisting of different transportation vehicles and animals. Results are presented in Figure 3. Once again, we observe worst sample quality when using high or nonexistent dropout values. Moreover, there are also clear traits of mode collapsing while using no dropout rate throughout all numbers of discriminators in the set. Sharper and more diverse samples are obtained while using 0.2 or 0.5 dropout rate and a bigger number of discriminators in the set.

3 CelebA

We lastly tested our approach in the cropped version of CelebA, containing faces of real-world celebrities. Results are given in Figure 4. One can see that using no dropout rate leads to similar looking faces, especially when using 2 and 5 discriminators. Once more, faces produced with mid-ranged dropout values with bigger discriminator ensembles present more variety and sample quality than the rest.

Parameter Evaluation

Since the results shown above rely heavily on subjective judgment, we now evaluate the effects of using a different number of discriminators and dropout rates on each dataset in a quantitative way. Note that the presented results are not state-of-the-art since exploring several architectural settings is not the focus of this work. Instead, by using different architectures on different datasets, our focus is to compare the effect of the different parameter combinations.

We used the Fréchet Inception Distance (FID) to measure the similarity between the fake and real images. The returned distance uses the mean $\mu$ and covariance $cov$ of a multi-variate Gaussian produced from the embeddings of the last pooling layer of the Inception-v3 model for both the real data and the generated data. In the original paper, the authors show that FID is more robust to noise and more correlated to human judgment than Inception Score . Moreover, FID has shown to be sensitive to mode collapse , with the returned distances increasing when samples from certain classes are missing from the generated set.

Minimum FID. Table 1 shows the minimum FID obtained by $G$ for each dataset. Lower values indicate more similarity between the fake and real data. We ran all of our experiments for 40 epochs in total and used the same architecture described previously. To obtain the best FID across all epochs, we generated $10000$ samples from $G$ at the end of each epoch and then proceeded to calculate the FID between the set of the generated samples per epoch and the whole training set.

By analyzing Table 1, we observe that the minimum values of FID for all datasets were mostly obtained when using $d=0.5$. However, by analyzing the local minima obtained while maintaining the same number of discriminators and only varying the dropout rate, it is also noticeable that one can also generally achieve very competitive results while using d\in\big{\{}0.2,0.5,0.8\big{\}}, depending on the number of discriminators and datasets being used. The results also show that applying dropout on multiple discriminators always leads to a better FID rather than maintaining the ensemble of discriminators static, i.e. $d=0$, or singular, i.e., using solely 1 discriminator.

Mean FID. We followed the same procedure and calculated the mean FID across all 40 epochs. Results are presented in Figure 5. This evaluation promotes a broader look at the stage of $G$ at the end of every epoch, reflecting the quality and variety of the generated samples over time. The presented graphs provide a clear vision regarding the advantages of using multiple discriminators instead of solely one, with the FID being better in the first case. Using 5 or 10 discriminators with mid-range dropout rates leads to better FID results across all datasets.

The similar looking performance when using 5 and 10 discriminators can be explained by what was previously mentioned regarding $G$ needing more time to learn from more feedback. Nevertheless, by analysis of Table 1 it is visible that better generated samples are produced when using 10 discriminators on all datasets, even if it takes more training to reach that state. This ultimately means that by having access to more feedback, $G$ is eventually able to produce better, varied samples in a more consistent manner over time.

Cumulative Intra FID. To test the sample diversity within a given epoch, we calculated the FID between the set of generated samples of every epoch. This was accomplished by generating $20000$ samples from $G$ at the end of every epoch and then calculating the FID between the two halves of the generated set. We evaluated the diversity of the generated samples over time by adding all calculated FIDs for each model. Results are shown in Figure 6.

From the analysis of the presented bar graphs, one can see the effect of using a different number of discriminators, with bigger sets of discriminators promoting a wider variety of generated samples within each epoch. This is generally observed across all datasets. Furthermore, it is noticeable the benefits of using mid-range dropout rates to promote sample diversity, especially when using a bigger discriminator set.

Method evaluation

We now proceed to compare our approach of applying adversarial dropout to standard GAN, i.e., Dropout-GAN, with other existing methods in the literature. We followed the toy experiment with a 2D mixture of 8 Gaussian distributions (representing 8 data modes) firstly presented in UnrolledGAN , and further adopted by D2GAN and MGAN . We used the same architecture as D2GAN for a fair comparison. The results are shown in Figure 7, where one can see that Dropout-GAN successfully covers the 8 modes from the real data while having significantly less noisy samples compared to the other discriminator-driven methods. Note that MGAN takes advantage of a multi-generator framework plus an additional classifier network while making use of a different architectural setting. However, due to the simplicity of our approach, we manage to converge to the real data modes faster than the other approach, specifically MGAN, as seen in the early training steps. Moreover, our framework achieves the lowest distance and divergence measures between the real and fake data.

To evaluate the extensibility of our approach, we studied the effects of using adversarial dropout in the following GAN methods: LSGAN, DRAGAN, and standard GAN using both the original (GAN) and modified objective (modGAN). These methods consist of a subset of the methods compared in , since they cover important variations of the original GAN framework where $D$’s output is either a probability (GAN, modGAN, and DRAGAN) or unbounded (LSGAN), while making use of gradient penalty (DRAGAN) or not. We also followed their presented training settings, training models on MNIST, CIFAR-10, and CelebA for 20, 40, and 100 epochs, respectively. We made use of a simpler architectural setting though, similar to the one previously described in Section 5 but with double the number of neurons per convolutional layer.

The best FID scores for each the original and multiple adversarial versions are reported in Table 2. The advantage of using adversarial dropout is significantly visible for each method, lowering the minimum FID obtained considerably for all the tested datasets. For a fair comparison, we used only 2 discriminators when applying adversarial dropout, which makes the overall framework still relatively small with a great benefit in the end results. When simply using an ensemble of discriminators on CIFAR-10, i.e. $d=0$, the proposed dropout variants improve FID by $7.25$, $4.39$, $3.96$, and $9.93$, on GAN, modGAN, LSGAN, and DRAGAN, respectively.

To also test how adversarial dropout behaves on larger datasets, we calculated the Inception Score (IS) to compare the quality in the same set of methods. On top of CIFAR-10, we further used STL-10 , and ImageNet , with the latter two being larger datasets with 100K and 1M images, respectively. We downsized all images to 32x32. We used the same architectures mentioned above. However, we trained each model longer, more specifically 250 epochs for CIFAR-10 and STL-10, and 50 epochs for ImageNet.

The obtained IS are presented in Table 3. Once again, we observe that applying adversarial dropout considerably increases the obtained IS for all tested datasets, without much overhead since only 2 discriminators were used.

A subset of randomly generated samples for each method when using adversarial dropout is presented in Figure 8, where one can see high diversity alongside with high quality, even on the bigger datasets. These results solidify the success of mitigating mode collapse when applying the adversarial dropout to the different methods.

Finally, we directly compare the quality of the generated samples between Dropout-GAN, GMAN , and original GANs with the modified loss, using both 2 and 5 discriminators on CIFAR-10. In their original experiments, used Inception Score as the evaluation metric, with higher values correlating to better generated samples. For a fair direct comparison, we used the same architectures and training procedures as originally used in GMAN’s experiments. Results are presented in Table 4. Dropout-GAN outperforms both methods for all different number of discriminators scenarios on all tested dropout rates.

Related Work

We will now focus on previous work that mitigated mode collapse in GAN. Instead of extending the original framework to multiple adversaries, one can change GAN objective to directly promote sample diversity. WGAN and MMD GAN proposed to optimize distance measurements to stabilize training. On the other hand, EBGAN and Coulomb GANs reformulated the original GAN problem using an energy-based objective to promote sample variability. While Regularized-GAN and MDGAN make use of an autoencoder to penalize missing modes and regularize GAN objective, DFM makes use of autoencoders to perform high-level feature matching. UnrolledGAN changes $G$ objective to satisfy an unrolled optimization of $D$. LSGAN proposes to use a least-squares loss for the $D$ while DRAGAN applies gradient norm penalty on top of original GAN.

Although some work has focused on augmenting the number of generators , or even increasing both the number of generators and discriminators , we turn our focus on methods that solely increases the number of discriminators to prevent mode collapse. D2GAN proposed a single generator dual discriminator architecture where one $D$ rewards samples coming from the true data distribution, while the other rewards samples that are likely to come from $G$. Thus, each $D$ still operates on a different objective function. GMAN proposed a framework where a single $G$ is trained against several discriminators on different levels of difficulty, by either using the mean loss of all discriminators (GMAN-0), picking only the $D$ with the maximum loss in relation to $G$’s output (GMAN-1), or controlled by $G$ through a hyperparameter $\lambda$ (GMAN*). Recently, microbatchGAN assigned a different portion of each minibatch to each discriminator to stimulate sample diversity.

However, all of the described approaches have some sort of constraints, either by restricting each $D$’s architecture to be different, or by using different objective functions for each $D$. We argue that these are limitations from an extensibility point of view, none of which exists in our proposed framework. Moreover, we note that applying Dropout-GAN’s principles of using adversarial dropout to the previously described methods would be a viable step to further promote sample diversity.

Conclusion and Future Work

In this work, we propose to mitigate mode collapse by proposing a new framework, called Dropout-GAN, that enables a single generator to learn from an ensemble of discriminators that dynamically changes at the end of every batch by use of adversarial dropout. We conducted experiments on multiple datasets of different sizes that show that adversarial dropout successfully contributes to a bigger sample variety on multiple GAN approaches. Moreover, it also increases training stability over time by enabling $G$ to receive more quantity and variety of feedback.

In the future, it would be interesting to adjust $G$’s learning rate according to the size of the discriminator set, allowing a more coherent learning speed between $G$ and each $D$, especially when using a large ensemble. Moreover, applying game theory to make the different discriminators dependent, i.e., aware of each other’s feedback, could also be a very interesting path to follow, taking full advantage of using multiple adversarial training.

References