DeLiGAN : Generative Adversarial Networks for Diverse and Limited Data

Introduction

Generative models for images have enjoyed a resurgence in recent years, particularly with the availability of large datasets and advent of deep neural networks . In particular, Generative Adversarial Networks (GANs) and Variational Auto-Encoders (VAE) have shown a lot of promise in this regard. In this paper, we focus on GAN-based approaches.

A typical GAN framework consists of two components, a generator $G$ and a discriminator $D$. The generator $G$ is modelled so that it transforms a random vector $z$ into an image $I$, i.e. $I=G(z)$. $z$ usually arises from an easy-to-sample distribution (e.g. uniform). $G$ is trained to generate images $I$ which are indistinguishable from a sampling of the true distribution, i.e $I\sim p_{data}$, where $p_{data}$ is the true distribution of images. The discriminator $D$ takes an image as input and outputs the probability that the image is from the true data distribution $p_{data}$. In practice, $D$ is trained to output a low probability $p_{D}$ when fed a “fake” (generated) image. $D$ and $G$ are trained adversarially to improve by competing with each other. A proper training regime ensures that at end of training, $G$ generates images which are essentially indistinguishable from real images, i.e. $p_{D}(G(z))=0.5$ .

In recent times, GAN-based approaches have been used to generate impressively realistic house-numbers , faces, bedrooms and a variety of other image categories . Usually, these image categories tend to have extremely complex underlying distributions. This complexity arises from two factors: (1) level of detail (e.g. color photos of objects have more detail than binary handwritten digit images) (2) diversity (e.g. inter and intra-category variability is larger for object categories compared to, say, house numbers). To be viable, generator $G$ needs to have sufficient capacity for tackling these complexity-inducing factors. Typically, such capacity is attained by having deep networks for $G$ . However, training high-capacity generators requires a large amount of training data. Therefore, existing GAN-based approaches are not viable when the amount of training data is limited.

We propose DeLiGAN – a novel GAN-based framework which is especially suited for small-yet-diverse data scenarios (Section 4).

We show that DeLiGAN enables generation of diverse images for a number of different modalities in limited data regimes. In particular, we construct modality-specific models which generate images of handwritten digits (Section 5.3), photo objects (Section 5.4) and hand-drawn sketches (Section 5.5).

To quantitatively characterize the intra-class diversity of generated samples, we also design a modified version of the “inception-score” , a measure which has been found to correlate well with human assessment of generated samples (Section 5.1).

The rest of the paper is organised as follows: We give an overview of the related work in Section 2, review GAN in Section 3 and then go on to describe our model DeLiGAN in Section 4. In Section 5, we discuss experimental results which showcase the capabilities of our model. Towards the end of the paper, we discuss these results and the implications of our design decisions in Section 6. We conclude with some pointers for future work in Section 7.

Related Work

Generative Adversarial Networks (GANs) have recently gained a lot of popularity due to the relative sharpness of samples generated by these models compared to other approaches. The originally proposed baseline approach has been modified to incorporate deep convolutional networks without destabilizing the training scheme and achieving significant qualitative improvements in image quality . Further improvements were made by Salimans et al. by incorporating algorithmic tricks such as mini-batch discrimination which stabilize training and provide better image quality. We incorporate some of these tricks in our work as well.

Our central idea – utilizing a mixture model for latent space – has been suggested in various papers, but mostly in the context of variational inference. For example, Gershman et al. , Jordan et al. and Jaakkola et al. model the approximate posterior of the inferred latent distribution as a mixture model to represent more complicated distributions. More recently, Renzede et al. and Kingma et al. propose ‘normalizing flows’ to transform the latent probability density through a series of invertible mappings to construct a complex distribution. In the context of GANs, no such approaches exist, to the best of our knowledge.

Our approach can be viewed as an attempt to modify the latent space to obtain samples in the high probability regions in the latent space. The notion of latent space modification has been explored in some recent works. For example, Han et al. propose to alternate between training the latent factors and the generator parameters. Arulkumaran et al. formulate an MCMC sampling process to sample from high probability regions of a learned latent space in variational or adversarial autoencoders.

Generative Adversarial Networks (GANs)

Although GANs were introduced in Section 1, we formally describe them below to establish continuity.

A typical GAN framework consists of two components, a generator $G$ and a discriminator $D$. In practice, these two components are usually two neural networks. The generator $G$ is modelled so that it transforms a random vector $z$ into an image $x_{G}$, i.e. $x_{G}=G(z)$. $z$ typically arises from an easy-to-sample distribution, for e.g. $z\sim\mathcal{U}(-1,1)$ where $\mathcal{U}$ denotes a uniform distribution. $G$ is trained to generate images which are indistinguishable from a sampling of the true distribution. In other words, while training $G$, we try to maximise $p_{data}(x_{G})$, the probability that the generated samples belong to the data distribution.

The above equations make explicit the fact that GANs assume a fixed, easy to sample, prior distribution $p_{z}(z)$ and then maximize $p_{data}(x_{G}|z)$ by training the generator network to produce samples from the data distribution.

The discriminator $D$ takes an image $I$ as input and outputs the probability $p_{D}(I)$ that the image is from the true data distribution. Typically, $D$ is trained to output a low probability when fed a “fake” (generated) image. Thus, $D$ is supposed to act as an expert, estimating the probability that the sample is from the true data distribution as opposed to the $G$’s output.

$D$ and $G$ are trained adversarially to improve by competing with each other. This is achieved by alternating between the training phases of $D$ and $G$. $G$ tries to ‘fool’ $D$ into thinking that its outputs are from the the true data distribution by maximizing its score $D(G(z))$. This is achieved by solving the following optimization problem in the generator phase of training:

On the other hand, $D$ tries to minimize the score it assigns to generated samples $G(z)$ by minimising $D(G(z))$ and maximize the score it assigns to the real (training) data $x$ by maximising $D(x)$. Hence, the optimisation problem for $D$ can be formulated as follows:

Hence the combined loss for the GAN can now be written as:

In their work, Goodfellow et al. show that Equation 5 gives us Jensen–Shannon (JS) divergence between the model’s distribution and data generating process. A proper training regime ensures that at the end of training, $G$ generates images which are essentially indistinguishable from real images, i.e. $p_{D}(G(z))=0.5$ and JS divergence achieves its lowest value.

Our model - DeLiGAN

In GAN training, we essentially attempt to learn a mapping from a simple latent distribution $p_{z}$ to the complicated data distribution (Equation 2). This mapping requires a deep generative network which can disentangle the underlying factors of variation in the data distribution and enable diversity in generated samples . In turn, this translates to the requirement of large amounts of data. Therefore, when data is limited yet originates from a diverse image modality, increasing the network depth becomes infeasible. Our solution to this conundrum is the following: Instead of increasing the model depth, we propose to increase the modelling power of the prior distribution. In particular, we propose a reparameterization of the latent space as a Mixture-of-Gaussians model (see Figure 1).

where $g(z|\mu_{i},\Sigma_{i})$ represents the probability of the sample $z$ in the normal distribution, $\mathcal{N}(\mu_{i},\Sigma_{i})$. For reasons which will be apparent shortly (Section 4.1), we assume uniform mixture weights, $\phi_{i}$, i.e.

To obtain a sample from the above distribution, we randomly select one of the $N$ Gaussian components and employ the “reparameterization trick” introduced by Kingma et al. to sample from the chosen Gaussian. We also assume that each Gaussian component has a diagonal covariance matrix. Suppose the $i$-th Gaussian is chosen. Let us denote the diagonal elements of the corresponding covariance matrix as $\sigma_{i}=[\sigma_{i_{1}}\sigma_{i_{2}}\ldots\sigma_{i_{K}}]$ where $K$ is the dimension of the latent space. For the “reparameterization trick”, we represent the sample from the chosen $i$-th Gaussian as a deterministic function of $\mu_{i}$, $\sigma_{i}$ and an auxiliary noise variable $\epsilon$.

Therefore, obtaining a latent space sample translates to sampling $\epsilon\sim\mathcal{N}(0,1)$ and calculating $z$ according to Equation 8. Substituting Equations 7, 8 in RHS of Equation 2, we get:

Let us define $\bm{\mu}=[\mu_{1},\mu_{2},\ldots,\mu_{N}]^{T}$ and $\bm{\sigma}=[\sigma_{1},\sigma_{2},\ldots,\sigma_{N}]^{T}$. Therefore, our new objective is to learn $\bm{\mu}$ and $\bm{\sigma}$ (along with the GAN parameters) to maximise $p_{data}(G(\mu_{i}+\sigma_{i}\epsilon)|\epsilon)$.

Next, we describe the procedure for learning $\bm{\mu}$ and $\bm{\sigma}$.

For each Gaussian component, we first need to initialise its parameters. For $\bm{\mu}_{i},1\leqslant i\leqslant N$, we sample from a simple prior – in our case, a uniform distribution $\mathcal{U}(-1,1)$. For $\sigma_{i}$, we assign a small, fixed non-zero initial value ($0.2$ in our case). Normally, the number of samples we generate from each Gaussian relative to the other Gaussians during training gives us a measure of the ‘weight’ $\bm{\pi}$ for that component. However, $\bm{\pi}$ is not a trainable parameter in our model since we cannot obtain gradients for $\pi_{i}$s. Therefore, as mentioned before, we consider all components to be equally important.

To generate data, we randomly choose one of the $N$ Gaussian components and sample a latent vector $z$ from the chosen Gaussian (Equation 8). $z$ is passed to $G$ to obtain the output data (image). The generated sample $z$ can now be used to train parameters of $D$ or $G$ using the standard GAN training procedure (Equation 5). In addition, $\bm{\mu}$ and $\bm{\sigma}$ are also trained simultaneously along with $G$’s parameters, using gradients arising from $G$’s loss function.

However, we need to consider a subtle issue here involving ${\bm{\sigma}}$. Since $p_{data}(G(z))$ (Equation 9) has local maxima at the $\mu_{i}$s, $G$ tries to decrease the $\sigma_{i}$s in an effort to obtain more samples from the high probability regions. As a result $\sigma_{i}$s can collapse to zero. Hence, we add a ${L_{2}}$ regularizer to the generator cost to prevent this from happening. The original formulation of loss function for $G$ (Equation 3) now becomes:

Note that this procedure can be extended to generate a batch of images for mini-batch training. Indeed, increasing the number of samples per Gaussian increases the accuracy of the gradients used to update $\bm{\mu}$ and $\bm{\sigma}$ since they are averaged out over $p(\epsilon)$ , thereby speeding up training.

Experiments

For our DeLiGAN framework, the choice of $N$, the number of Gaussian components, is made empirically – more complicated data distributions require more Gaussians. Larger values of $N$ potentially help model with relatively increased diversity. However, increasing $N$ also increases memory requirements. Our experiments indicate that increasing $N$ beyond a point has little to no effect on the model capacity since the Gaussian components tend to ‘crowd’ and become redundant. We use a $N$ between $50$ and $100$ for our experiments.

To quantitatively characterize the diversity of generated samples, we also design a modified version of the “inception-score”, a measure which has been found to correlate well with human evaluation . We describe this score next.

We analyze the performance of DeLiGAN models trained on toy data, handwritten digits , photo objects and hand-drawn object sketches and compare with a regular GAN model. Specifically, we use a variant of DCGAN with mini-batch discrimination in the discriminator . We also need to note here that DeLiGAN adds extra parameters over DCGAN. Therefore, we also compare DeLiGAN with baseline models containing an increased number of learnable parameters. We start by describing a series of experiments on toy data.

2 Toy Data

As a baseline GAN model for toy data, we set up a multi-layer perceptron with one hidden layer as $G$ and $D$ (see Figure 2). For the DeLiGAN model, we incorporate the mixture of Gaussian layer as shown in Figure 1. We also compare DeLiGAN with four other baseline models – (i) GAN++ (instead of mixture of Gaussian layer, we add a fully connected layer containing $N$ neurons between the input ($z$) and the generator) (ii) Ensemble-GAN (An ensemble-of-$N$-generators setting for DeLiGAN. During training, we randomly choose one of the generators $G_{i}$ for training and update its parameters along with $\mu_{i},\sigma_{i}$) (iii) $N$x-GAN (We increase number of parameters in the generator network $N$ times by having $N$ times more neurons in the hidden layer) and (iv) MoE-GAN (This is short for Mixture-of-Experts GAN. In this model, we just append a uniform discrete variable via a $N$-dimensional one-hot encoding to the random input $z$).

For the first set of experiments, we design our generator network to output data samples originally belonging to a unimodal 2-D Gaussian data (see Figure 3(g)). Figures 3 (a)-(f) show samples generated by the respective GAN variants for this data. For the unimodal case, all models perform reasonably well in generating samples.

For the second set of experiments, we replace the unimodal distribution with a bi-modal distribution comprising two Gaussians (Figure 3(n)). The results in this case show that DeLiGAN is able to clearly model the two separate distributions whereas the baseline GAN frameworks struggle to model the void in between (Figure 3(h-m)). Although the other variants, containing more parameters, were able to model the two modes, they still struggle to model the local structure in the Gaussians properly. The generations produced by DeLiGAN look the most convincing. Although not obvious from the results, a recurring trend across all the baseline models was the relative difficulty in training due to instabilities. On the other hand, training DeLiGAN was much easier in practice. As we shall soon see, this phenomenon of suboptimal baseline models and better performance by DeLiGAN persists even for more complex data distributions (CIFAR-10, sketches etc.)

3 MNIST

The MNIST dataset contains $60,000$ images of handwritten digits from to $9$ . We conduct experiments on a reduced training set of $500$ images to mimic the low-data scenario. The images are sampled randomly from the dataset, keeping the total number of images per digit constant. For MNIST, the generator network has a fully connected layer followed by $3$ deconvolution layers while the discriminator network has $3$ convolutional layers followed by a mini-batch discrimination layer (see Figure 2).

In Figure 4, we show typical samples generated by both models, arranged in a $7\times 7$ grid. For each model, the last column of digits (outlined in red), contains nearest-neighbor images (from the training set) to the samples present in the last ($7$th) column of the grid. For nearest neighborhood computation, we use $L_{2}$ distance between the images.

The samples produced by our model (Figure 4(b), right) are visibly crisper compared to baseline GAN (Figure 4(a), left). Also, some of the samples produced by the GAN model are almost identical to one other (shown as similarly colored boxes in Figure 4(a)) whereas our model produces more diverse samples. We also observe that some of the samples produced by the baseline GAN model are deformed and don’t resemble any digit. This artifact is much less common in our model. Additionally, in practice, the baseline GAN model frequently diverges during training given the small data regime and the deformation artifact mentioned above becomes predominant, eventually leading to homogeneous non-digit like samples. In contrast, our model remains stable during training and generates samples with better diversity.

4 CIFAR 10

The CIFAR 10 dataset contains $60,000$ $32\times 32$ color images across $10$ object classes. Once again, to mimic the diverse-yet-limited-data scenario, we compare the architectures on a reduced dataset of $2000$ images. The images are drawn randomly from the entire dataset, keeping the number of images per category constant. For the experiments involving CIFAR dataset, we adopt the architecture proposed by Goodfellow et al. . The generator has a fully connected layer followed by $3$ deconvolution layers with batch normalisation after each layer. The discriminator network has $9$ convolutional layers with dropout and weight normalisation, followed by a mini-batch discrimination layer.

Figure 5 shows samples generated by our model and the baseline GAN model. As in the case of MNIST, some of the samples generated by the GAN, shown with similar colored bounding boxes, look nearly identical (Figure 5(a)). Again, we observe that our model produces visibly diverse looking samples and provides more stability. The modified “inception-score” values for the models (Table 1) attest to this observation as well. Note that there exist categories (‘cat’, ‘dog’) with somewhat better diversity scores for GAN. Since images belonging to these categories are similar, these kinds of images would be better represented in the data. As a result, GAN performs better for these categories, whereas DeLiGAN manages to capture even the other under-represented categories. Table 1 also shows the modified inception scores for the GAN++ and MoE-GAN models introduced in the toy experiments. We observe that the performance in this case is actually worse than the baseline GAN model, despite the increased number of parameters. Moreover, adding fully connected layers in the generator in GAN++ also leads to increased instability in training. We hypothesize that the added set of extra parameters worsens the performance given our limited data scenario. In fact, for baseline models such as Ensemble-GAN and $N$x-GAN, the added set of parameters also makes computations prohibitively expensive.

Overall, the CIFAR dataset experiments demonstrate that our model can scale to more complicated real life datasets and still outperform the traditional GANs in low data scenarios.

5 Freehand Sketches

The TU-Berlin dataset , contains $20,000$ hand-drawn sketches evenly distributed among $250$ object categories, which amounts to $80$ images per category. This dataset represents a scenario where the amount of training data is actually limited, unlike previous experiments where the quantity of training data was artificially restricted. For sketches, our network contains $5$ convolutional layers in the discriminator with weight normalization and dropout followed by mini-batch discrimination and $3$ deconvolutional layers, followed by a fully connected layer in the generator. To demonstrate the capability of our model, we perform two sets of experiments.

For the first set of experiments, we select $4$ sketch categories – apple, pear, tomato, candle. These categories have simple global contours, low sketch stroke density and are somewhat similar in appearance. During training, we augment the dataset using the flipped versions of the images. Once again, we compare the generated results of GAN and DeLiGAN. Figure 6 shows the samples generated by DeLiGAN and GAN respectively, trained on the similar looking categories (left side of the dotted line). The samples generated by both the models look visually appealing. Our guess is that since the object categories are very similar, the data distribution can be easily modelled as a continuous distribution in the latent space. Therefore, the latent space doesn’t need a multi-modal representation in this case. This is also borne out by the m-IS diversity scores in Table 2.

For the second set of experiments, we select $5$ diverse looking categories – apple, wine glass, candle, canoe, cup – and compare the generation results for both the models. The corresponding samples are shown in Figure 6 (on the right side of the dotted line). In this case, DeLiGAN samples are visibly better, less hazy, and arise from a more stable training procedure. The samples generated by DeLiGAN also exhibit larger diversity, visibly and according to m-IS scores as well (Table 3).

Discussion

The experiments described above demonstrate the benefits of modelling the latent space as a mixture of learnable Gaussians instead of the conventional unit Gaussian/uniform distribution. One reason for our performance is derived from the fact that mixture models can approximate arbitrarily complex latent distributions, given a sufficiently large number of Gaussian components.

In practice, we also notice that our mixture model approach also helps increase the model stability and is especially useful for diverse, low-data regimes where the latent distribution might not be continuous. Consider the following: The gradients on $\mu_{i}$s push them in the latent space in a direction which increases the discriminator score, $D(G(z))$ as per the gradient update (Equation 11). Thus, samples generated from the updated Gaussian components result in higher probability, $p_{data}(G(z))$.

Hence, as training progresses, we find the $\mu_{i}$s in particular, even if initialised in the lower probability regions, slowly drift towards the regions that lead to samples of high probability, $p_{data}(G(z))$. Hence, fewer points are sampled from the low probability regions. This is illustrated by (i) the locations of samples generated by our model in the toy experiments (Figure 3(d)) (ii) relatively small frequency of bad quality generations (that don’t resemble any digit) for the MNIST experiments (Figure 4). Our model successfully handles the low probability void between the two modes in the data distribution by emulating the void into its own latent distribution. As a result, no samples are produced in these regions. This can also be seen in the MNIST experiments – our model produces very few non-digit like samples compared to the baseline GAN (Figure 4).

In complicated multi-modal settings, the data may be disproportionally distributed among the modes such that some of the modes contain relatively more data points. In this situation, the generator in baseline GAN tends to fit the latent distribution to the mode with maximum data as dictated by the Jensen-Shannon Divergence . This results in low diversity among the generated samples since a section of the data distribution is sometimes overlooked by the generator network. This effect is especially pronounced in low data regimes because the number of modes in the image space increase due to the non-availability of data connecting some of the modes. As a result, the generator tries to fit to a small fraction of the already limited data. This is consistent with our experimental results wherein the diversity and quality of samples produced by baseline GANs deteriorate with decreasing amounts of training data (MNIST – Figure 4, CIFAR – Figure 5) or increasing diversity of the training data (Sketches – Figure 6).

Our design decision of having a trainable mixture model for latent space can be viewed as an algorithmic “plug-in” that can be added to almost any GAN framework including recently proposed models to obtain better performance on diverse data. Finally, it is also important to note that our model is still constrained by the modelling capacity of the underlying GAN framework itself. Hence, as we employ better GAN frameworks on top of our mixture of Gaussians layer, we can expect the model to generate realistic, high-quality samples.

Conclusions and Future Work

In this work, we have shown that reparameterizing the latent space in GANs as a mixture model can lead to a powerful generative model. Via experiments across a diverse set of modalities (digits, hand-drawn object sketches and color photos of objects), we have observed that this seemingly simple modification helps stabilize the model and produce diverse samples even in low data scenarios. Currently, our mixture model setup incorporates some simplifying assumptions (diagonal covariance matrix for each component, equally weighted mixture components) which limit the ability of our model to approximate more complex distributions. These parameters can be incorporated into our learning scheme to better approximate the underlying latent distribution. The source code for training DeLiGAN models and computing modified inception score can be accessed at http://val.cds.iisc.ac.in/deligan/.

Acknowledgements

We would like to thank our anonymous reviewers for their suggestions, NVIDIA for their contribution of Tesla K40 GPU, Qualcomm India for their support to Ravi Kiran Sarvadevabhatla via the Qualcomm Innovation Fellowship and Google Research India for their travel grant support.

References