Gaussian Mixture Generative Adversarial Networks for Diverse Datasets, and the Unsupervised Clustering of Images

Introduction

Generative models have long been an important and active field of research in machine-learning. Such models take as input a training set of data points from an unknown data distribution, and return an estimate of that distribution. By learning to capture the statistical distribution of the training data, this family of models allows one to generate additional data points by sampling from the learned distribution. Well-known families of generative methods include the Naïve Bayes model, Hidden Markov models, Deep Belief Networks, Variational Auto-Encoders [KW13] (VAEs) and Generative Adversarial Networks (GANs) [GPM+14].

Generative Adversarial Networks include a family of methods for learning generative models where the computational approach is based on game theory. The goal of a GAN is to learn a Generator ($G$) capable of generating samples from the data distribution ($p_{\mathcal{X}}$), by converting latent vectors from a lower-dimension latent space ($Z$) to samples in a higher-dimension data space ($\mathcal{X}$). Usually, latent vectors are sampled from $Z$ using the uniform or the normal distribution. In order to train $G$, a Discriminator ($D$) is trained to distinguish real training samples from fake samples generated by $G$. Thus $D$ returns a value $D({{\mathbf{x}}})\in$ which can be interpreted as the probability that the input sample (${{\mathbf{x}}}$) is a real sample from the data distribution. In this configuration, $G$ is trained to obstruct $D$ by generating samples which better resemble the real training samples, while $D$ is continuously trained to tell apart real from fake samples.

Crucially, $G$ has no direct access to real samples from the training set, as it learns solely through its interaction with $D$. If $G$ is able to perfectly match the real data distribution $p_{\mathcal{X}}$, then $D$ will be maximally confused, predicting $0.5$ for all input samples. Such a state is known as a Nash equilibrium, and has been shown in [GPM+14] to be the optimal solution for this learning framework. Both $D$ and $G$ are implemented by deep differentiable networks, typically consisting of multiple convolutional and fully-connected layers. They are alternately trained using the Stochastic Gradient Descent algorithm.

GANs have been extensively used in the domain of computer-vision, where their applications include super resolution from a single image [LTH+16], text-to-image translation [RAY+16], image-to-image translation [ZPIE17, IZZE16, KCK+17], image in-painting [YCY+16] and video completion [MCL15]. Aside from their usages in the computer-vision domain, GANs have been used for other tasks such as semi-supervised learning [KRMW14, Spr15a], music generation [YCY17, DHYY17], text generation [YZWY16] and speech enhancement [PBS17].

In the short period of time since their introduction, many different enhancement methods and training variants have been suggested to improve their performance (see brief review below). Despite these efforts, often a large proportion of the generated samples is, arguably, not satisfactorily realistic. In some cases the generated sample does not resemble any of the real samples from the training set, and human observers find it difficult to classify synthetically generated samples to one of the classes which compose the training set (see illustration in Figure 1).

The problem described above worsens with the increased complexity of the training set, and specifically when the training set is characterized by large inter-class and intra-class diversity. In this work we focus on this problem, aiming to improve the performance of GANs when the training dataset has large inter-class and intra-class diversity.

In an attempt to improve the performance of the original GAN model [GPM+14], many variants and extensions have been proposed in the past few years. Much effort was directed at improving GANs through architectural changes to $G$ and $D$, as in the DCGANs described in [RMC15]. Improved performance was reported in [MLX+16, GAA+17], among others, by modifying the loss function used to train the GAN model. Additional improvement was achieved by introducing supervision into the training setting, as in conditional GANs [MO14, OOS16]. These conditional variants were shown to enhance the quality of the generated sample, while also improving the stability of the notorious training process of these models.

Another branch of related works, which perhaps more closely relates to our work, involves the learning of a meaningfully structured latent space: Info-GAN [CDH+16] decomposes the input noise into an incompressible source and a "latent code", attempting to discover latent factors of variation by maximizing the mutual information between the latent code and the Generator’s output. This latent code can be used to discover object classes in a purely unsupervised fashion, although it is not strictly necessary that the latent code be categorical. Adversarial Auto-Encoders [MSJ+15] employ GANs to perform variational inference by matching the aggregated posterior of the auto-encoder’s hidden latent vector with an arbitrary prior distribution. As a result, the decoder of the adversarial auto-encoder learns a deep generative model that maps the imposed prior to the data distribution. [BKLW15] combined a Variational Auto-Encoder with a Generative Adversarial Network in order to use the learned feature representations in the GAN’s discriminator as basis for the VAE reconstruction objective. As a result, this hybrid model is capable of learning a latent space in which high-level abstract visual features (e.g. wearing glasses) can be modified using simple arithmetic of latent vectors.

Although modifications to the structure of the latent space have been investigated before as described above, the significance of the probability distribution used for sampling latent vectors was rarely investigated. A common practice today is to use a standard normal (e.g. $N(0,I)$) or uniform (e.g. $U$) probability distribution when sampling latent vectors from the latent space. We wish to challenge this common practice, and investigate the beneficial effects of modifying the distribution used to sample latent vectors in accordance with properties of the target dataset.

Specifically, many datasets, especially those of natural images, are quite diverse, with high inter-class and intra-class variability. At the same time, the representations of these datasets usually span high dimensional spaces, which naturally makes them very sparse. Intuitively, this implies that the underlying data distribution, which we try to learn using a GAN, is also sparse, i.e. it mostly consists of low-density areas with relatively few areas of high-density.

Our approach is to incorporate this prior-knowledge into the model, by sampling latent vectors using a multi-modal probability distribution which better matches these characteristics of the data space. It is important to emphasize that this architectural modification is orthogonal to, and can be used in conjunction with, other architectural improvements such as those reviewed above. Supervision can be incorporated into this model by adding correspondence (not necessarily injective) between labels and mixture components.

The rest of this paper is organized as follows: In Section 2 we describe the family of GM-GAN models. In Section 3 we discuss the shortcomings of the popular Inception Score [SGZ+16], and further show that GANs offer a trade-off between sample quality and diversity. We propose an alternative evaluation score which is, arguably, better suited to the task of image synthesis using GANs, and which can quantify the quality-diversity trade-off. In Section 4 we empirically evaluate our proposed model in the task of sample synthesis, when trained with various diverse datasets. We show that GM-GANs outperform baselines and achieve better scores. In Section 5 we describe a method for clustering datasets using GM-GANs, and provide qualitative and quantitative evaluation using various datasets of real images.

Gaussian Mixture GAN

The target function which we usually optimize for, when training a GAN composed of a Generator $G$ and Discriminator $D$, can be written as follows:

Above $p_{\mathcal{X}}$ denotes the distribution of real training samples, and $p_{{\mathcal{Z}}}$ denotes some $d$-dimensional prior distribution which is used as a source of stochasticity for the Generator. The corresponding loss functions of $G$ and $D$ can be written as follows:

Usually, a multivariate uniform distribution (e.g. $U^{d}$), or a multivariate normal distribution (e.g. $N(0,I_{d\times d}))$ is used as $p_{{\mathcal{Z}}}$ when training GANs. In our proposed model, we optimize for the same target function as in 1, but instead of using a unimodal random distribution for the prior $p_{{\mathcal{Z}}}$, we propose to use a multi-modal distribution which can better suit the inherent multi-modality of the real training data distribution, $p_{\mathcal{X}}$. In this work, we propose to use a mixture of Gaussians as a multi-modal prior distribution. Formally, we have:

where $K$ denotes the number of Gaussians in the mixture, $\{\alpha_{k}\}_{k=1}^{K}$ denotes a categorical random variable, and $p_{k}({{\mathbf{z}}})$ denotes the multivariate Normal distribution $N(\mu_{k},\Sigma_{k})$, defined by the mean vector $\mu_{k}$, and the covariance matrix $\Sigma_{k}$. In the absence of prior knowledge we assume a uniform mixture of Gaussians, that is, $\forall k\in[K]$ $\alpha_{k}=\frac{1}{K}$.

The parameters $\mu_{k},\Sigma_{k}$ of each Gaussian in the mixture can be fixed or learned. One may be able to choose these parameters by using prior knowledge, or pick them randomly. Perhaps a more robust solution is to learn the parameters of the Gaussian Mixture along with the parameters of the GAN in an "end-to-end" fashion. This should, intuitively, allow for a more flexible, and perhaps better performing model. We therefore investigated two variants of the new model - one (static) where the the parameters of the Gaussians mixture are fixed throughout the model’s training process, and one (dynamic) where these parameters are allowed to change during the training process in order to potentially converge to a better a solution. These variants are described in detail next:

Dynamic GM-GAN. We extend our basic model in order to allow for the dynamic tuning of parameters for each of the Gaussians in the mixture. We start by initializing the mean vectors and covariance matrices as in the static case, but we include them in the set of learnable parameters that are optimized during the GAN’s training process. This modification allows the Gaussians’ means to wander to new locations, and lets each Gaussian have a unique covariance matrix. This potentially allows the model to converge to a better local optimum, and achieve better performance.

We note that when training either the static or dynamic variants of our model, we optimize for the same loss functions as in (2) and (3). Clearly other loss functions can be used in conjunction with the suggested architectural modifications, as those changes are independent.

We also note that the dynamic variant of our model includes additional $K*(d^{2}+d)$ trainable parameters, as compared to the static model. In cases where $K$ and $d$ are sufficiently large, this can introduce significant computational overhead to the optimization procedure. To mitigate this issue, one can reduce the number of degrees of freedom in $\Sigma_{k}$, e.g. by assuming a diagonal matrix, in which case the number of additional trainable parameters is reduced to $2*K*d$.

The Generator’s purpose in this setting is, given a latent vector ${\mathbf{z}}$ sampled from the $k$’th Gaussian in the mixture, to generate a sample which will be classified by the discriminator as a real sample of class $f(k)$, where $f:[K]\rightarrow[N]$ is a discrete function mapping identity of Gaussians to class labels. When $K=N$, $f$ is bijective and the model is trained to map each Gaussian to a unique class in the data space. When $K>N$ $f$ is surjective, and multiple Gaussians can be mapped to the same class. This can be useful in cases where the training set is characterized by high intra-class diversity and when single classes can be broken down to multiple, visually distinct, sub-classes. When $K<N$ $f$ is injective, and multiple classes can be mapped to the same Gaussian achieving the clustering of class labels.

We modify both loss functions of $G$ and $D$ to accommodate the class labels. The modified loss functions become the following:

where $y({{\mathbf{x}}})$ denotes the class label of sample ${{\mathbf{x}}}$, and $y({{\mathbf{z}}})$ denotes the index of the Gaussian from which the latent vector ${{\mathbf{z}}}$ has been sampled. The training procedure for GM-GANs is fully described in Algorithm 1.

GAN Evaluation Score

We describe next a new scoring method for GANs, which is arguably better suited for the task than the commonly used Inception Score [SGZ+16].

[SGZ+16] proposed a method to evaluate generative models for natural image synthesis, such as VAEs and GANs, using a pre-trained classifier. It is based on the fact that good samples, i.e. images that look like images from the true data distribution, are expected to yield: (i) low entropy $p(y|{{\mathbf{x}}})$, implying high prediction confidence; (ii) high entropy $p(y)$, implying highly varied predictions. Here ${{\mathbf{x}}}$ denotes an image sampled from the Generator, $p(y|{{\mathbf{x}}})$ denotes the inferred class label probability given ${{\mathbf{x}}}$ by the Inception network [SVI+16] pre-trained on the ImageNet dataset, and $p(y)$ denotes the marginal distribution over all images sampled from the Generator.

The Inception Score [SGZ+16] is therefore defined as:

This score has been used extensively over the last few years. However, it has a number of drawbacks which we found to be rather limiting:

The Inception Score is based on the Inception network [SVI+16], which was pre-trained on the ImageNet dataset. This dataset contains $\sim 1.2$ million natural images belonging to 1,000 different classes. As a result the use of the Inception Score is limited to cases where the dataset consists of natural images. For example, we cannot use the Inception Score to evaluate the performance of a GAN trained on the MNIST dataset, which contains gray-scale images of hand-written digits.

Even in cases where the dataset on which we train a GAN consists of natural images, the distribution of these images is likely to be very different from that of ImageNet. In which case, the confidence of the Inception network’s prediction on such images may not correlate well with their actual quality.

The Inception Score only measures the samples’ inter-class diversity, namely, the distribution of these samples across different classes $p(y)$. Another equally important measure, which must be taken into account, is the intra-class diversity of samples, namely, the variance of different samples which all belong to the same class.

The Inception Score combines together a measure of quality and a measure of diversity into a single score. When evaluating the qualities of a GAN using solely this combined score, one cannot asses the true trade-off between the quality and the diversity of generated images. Thus a given Inception Score can be achieved by a GAN which generates very diverse but poor quality images, and also by a GAN which generates similarly looking but high quality images. Different Inception Scores can also be achieved by the same GAN, when sampling latent vectors with different parameters of the source probability distribution (e.g. $\sigma$), as illustrated in Figure 2.

2 Alternative Score: Measuring the Quality-Diversity Trade-off

Our proposed measure is motivated by the expectation that a well trained Generator will map samples of high probability in the latent space to samples of high probability in the target domain, and vice versa. If we measure the quality of a sample ${{\mathbf{x}}}\in\mathcal{X}$ in the target domain by its probability $p_{\mathcal{X}}({{\mathbf{x}}})$, then we can expect samples drawn from dense areas in the latent space (i.e. close to the modals of the distribution) to be mapped to high quality samples in the target domain, and vice versa. Therefore, we can increase the expected quality of generated samples in the target domain by sampling with high probability from dense areas of the latent space, and with low probability from sparse areas of the latent space. While increasing the expected quality of generated samples, this procedure also reduces the sample diversity In our experiments, we were able to control this quality-diversity trade-off by modifying the probability distribution which is used for sampling latent vectors from the latent space $Z$ (see Figs. 4, 6). We further elaborate on this matter in Section 4.3.. This fundamental trade-off between quality and diversity must be quantified if we want to compare the performance different GAN models.

Next we propose a new scoring method for GANs, which allows one to evaluate the trade-off between samples’ quality and diversity. This scoring method also relies on a pre-trained classifier, but unlike the Inception Score, this classifier is trained on the same training set on which the GAN is trained on. This classifier is used to measure both the quality and the diversity of generated samples, as explained below.

To measure the quality of a generated sample ${{\mathbf{x}}}$, we propose to use an intermediate representation of ${{\mathbf{x}}}$ in the pre-trained classifier $c$, and to measure the Euclidean distance from this representation to its nearest-neighbor in the training set. More specifically, if $c_{l}({{\mathbf{x}}})$ denotes the activation levels in the pre-trained classifier’s layer $l$ given sample ${{\mathbf{x}}}$, then the quality score $q({{\mathbf{x}}})$ is defined as:

Above $a$ denotes a constant greater than zero, and $NN({{\mathbf{x}}})$ denotes the nearest-neighbor of ${{\mathbf{x}}}$ in the training set, defined as $NN({{\mathbf{x}}})=\underset{{{\mathbf{x}}}^{\prime}\in X}{\arg\min}||c_{l}({{\mathbf{x}}})-c_{l}({{\mathbf{x}}}^{\prime})||_{2}$. We also define the quality score for a set of samples $X$ as follows:

Diversity Score

To measure the diversity of generated samples, we take into account both the inter-class, and the intra-class diversity. For intra-class diversity we measure the average (negative) MS-SSIM metric [WSB+03] between all pairs of generated images in a given set of generated images $X$:

For intra-class diversity, we use the pre-trained classifier to classify the set of generated images, such that for each sampled image ${{\mathbf{x}}}$, we have a classification prediction in the form of a one-hot vector $c({{\mathbf{x}}})$. We then measure the entropy of the average one-hot classification prediction vector to evaluate the diversity between classes in the samples set:

We combine both the intra-class and the inter-class diversity scores into a single diversity score as follows:

Combined Score

While it is important to look at the quality and diversity scores separately, since they measure two complementary properties of a model, it is sometimes necessary to obtain a single score per model. We therefore define the following combined measure:

The range of the proposed quality, diversity and combined scores is $$, where marks the lowest score, and$1$ marks the highest score. This property makes them easy to comprehend, and convenient to use when comparing the performance of different models.

Experimental Evaluation

In this section we empirically evaluate the benefits of our proposed approach, comparing the performance of GM-GAN with alternative baselines. Specifically, we compare the performance of the unsupervised GM-GAN model to that of the originally proposed GAN [GPM+14], and the performance of our proposed supervised GM-GAN model to that of AC-GAN [OOS16]. In both cases, the baseline models’ latent space probability distribution is standard normal, i.e. $\textbf{z}\sim N(0,I)$. The network architectures and hyper-parameters used for training the GM-GAN models are similar to those used for training the baseline models. For the most part we used the Static GM-GAN with default values $d=100$, $c=0.1$, $\sigma=0.15$, $B_{D}=64$, $b_{G}=128$, $\gamma=0.0002$; $K$ and $iters$ varied in the different experiments. The Dynamic GM-GAN model was only used in Figure 7.

In the following experiments we evaluated the different models on the 6 datasets listed in Table 1. In all cases, the only pre-processing made on the training images is a transformation of pixel-values to the range of $$.

We trained two instances of the GM-GAN model, one supervised using the labels of the samples, and one unsupervised which was not given access to these labels. In both cases, we used $K=9$ Gaussians in the mixture from which latent vectors are sampled. Figure 3 presents samples generated by the baseline models (GAN, AC-GAN) and samples generated by our proposed GM-GAN models (both unsupervised and supervised variants). It is clear that both variants of the GM-GAN generate samples with a higher likelihood, which matches the original distribution more closely as compared to the baseline methods. It is also evident that in this configuration, the diversity of samples generated by the GM-GAN model is lower than that of the classic GAN model. This illustrates the trade-off between quality and diversity, which we explore more thoroughly in Section 4.3. Figure 4 demonstrates the superiority of GM-GAN as compared to classic GAN, when measuring the trade-off between quality and diversity offered by these models (see Section 4.3 for further elaboration on this matter).

An intriguing observation is that the GM-GAN’s Generator is capable, without any supervision, of mapping each Gaussian in the latent space to samples in the data-space which are almost perfectly aligned with a single Gaussian. We also observe this when training unsupervised GM-GAN on the MNIST and Fashion-MNIST datasets. In Section 5 we exploit this phenomenon by training unsupervised clustering models. Finally, we note that the GM-GAN models converge considerably faster than the classical GAN model. Figure 5 shows the (negative) log-likelihood of samples generated from the different models, as a function of the training epoch.

2 Real Datasets, Inception Scores

We next turn to evaluate our proposed models when trained on more complex datasets. We start by using the customary Inception Score [SGZ+16] to evaluate and compare the performance of the difference models, the two GM-GAN models and the baseline models (GAN and AC-GAN). We trained the models on two real datasets with 10 classes each, the CIFAR-10 [KNH] and STL-10 [AC11] datasets. Each variant of the GM-GAN model was trained multiple times, each time using a different number ($K$) of Gaussians in the latent space probability distribution. In addition, each model was trained 10 times using different initial parameter values. We then computed for each model its mean Inception Score and the corresponding standard error. The results for the two unsupervised and two supervised models are presented in Table 2. In all cases, the two GM-GAN models achieve higher scores when compared to the respective baseline model. The biggest improvement is achieved in the supervised case, where the supervised variant of the GM-GAN model outperforms AC-GAN by a large margin. We also found that the number of Gaussians used in the GM-GAN’s latent space probability distribution can improve or impair the performance of the corresponding model, depending on the dataset.

3 Trade-off between Quality and Diversity

As discussed in Section 3, the Inception Score is not sufficient, on its own, to illustrate the trade-off between the quality and the diversity of samples which a certain GAN is capable of generating. In our experiments, we control the quality-diversity trade-off by varying, after the model’s training, the probability distribution which is used to sample latent vectors from the latent space. We do so by multiplying the covariance matrix of each Gaussian by a scaling factor $\sigma$. Specifically, when using the baseline models we sample $\textbf{z}\sim N(0,\sigma*I)$, and when using the GM-GAN models we sample $\textbf{z}|k\sim N(\mu_{k},\sigma*\Sigma_{k})$, $k\sim Categ(\frac{1}{K},...,\frac{1}{K})$. Thus, when $\sigma<1$, latent vectors are sampled with lower variance around the modes of the latent space probability distribution, and therefore the respective samples generated by the Generator are of higher expected quality, but lower expected diversity. The opposite happens when $\sigma>1$, where the respective samples generated by the Generator are of lower expected quality, but higher expected diversity. Figures 4, 6 demonstrate qualitatively the quality-diversity trade-off offered by GM-GANs when trained on the Toy and MNIST datasets.

We evaluated each model by calculating our proposed Quality Score from Eq. (6), and the Combined Diversity Score from Eq. (10), for each $\sigma\in\{0.5,0.6,...,1.9,2.0\}$. Each model was trained 10 times using different initial parameter values. We computed for each model its mean Quality and mean Combined Diversity scores and the corresponding standard errors. The Quality and Diversity Scores of the GM-GAN and baseline models, when trained on the CIFAR-10, STL-10, Fashion-MNIST and MNIST datasets, are presented in Figure 7. In some cases (e.g. supervised training on CIFAR-10 and STL-10) the results show a clear advantage for our proposed model as compared to the baseline, as both the quality and the diversity scores of GM-GAN surpass those of AC-GAN, for all values of $\sigma$. In other cases (e.g. unsupervised training on CIFAR-10 and STL-10), the results show that for the lower-end range of $\sigma$, the baseline model offers higher quality, but dramatically lower diversity samples, as compared to our proposed model. In accordance, when visually examining the samples generated by the two models, we notice that most samples generated by the baseline model belong to a single class, while samples generated by our model are much more diverse and are scattered uniformly between different classes. In all cases, the charts predictably show an ascending Quality Score, and a descending Combined Diversity Score, as $\sigma$ is increased.

Unsupervised Clustering using GM-GANs

Throughout our experiments, we noticed an intriguing phenomenon where the unsupervised variant of GM-GAN tends to map latent vectors sampled from different Gaussians in the latent space to samples of different classes in the data space. Specifically, each Gaussian in the latent space is usually mapped, by the GM-GAN’s Generator, to a single class in the data space. Figures 3, 8 demonstrate this phenomenon on different datasets. The fact that the latent space in our proposed model is sparse, while being composed of multiple Gaussians with little overlap, may be the underlying reason for this phenomenon. Next we exploit this observation to develop a new clustering algorithm, and provide quantitative evaluation of the proposed method.

Our clustering method works as follows: we first train an unsupervised GM-GAN on the dataset, where $K$, the number of Gaussians forming the latent space, is set to equal the number of clusters in the intended partition. Using the trained GM-GAN model, we sample from each Gaussian $k\in[K]$ a set of $M$ latent vectors, from which we generate a set of $M$ synthetic samples $\widetilde{X}_{k}=\left\{\widetilde{{\mathbf{x}}}_{k}^{(i)}\right\}_{i\in[M]}$. We then train a $K$-way multi-class classifier on the unified set of samples from all Gaussians $\bigcup_{k\in[K]}\widetilde{X}_{k}$, where the label of sample $\widetilde{{\mathbf{x}}}\in\widetilde{X}_{k}$ is set to $k$, i.e. the index of the Gaussian from which the corresponding latent vector has been sampled. Finally, we obtain the soft-assignment to clusters of each sample ${\mathbf{x}}$ in the original dataset by using the output of this classifier $c({\mathbf{x}})\in^{K}$ when given ${\mathbf{x}}$ as input. Each element $c({\mathbf{x}})_{k}$ ($k\in[K]$) of this output vector marks the association level of the sample ${\mathbf{x}}$ to the cluster $k$. A hard-assignment to clusters can be trivially calculated from the soft-assignment vector by selecting the cluster $k$ with which the sample is mostly associated, i.e. $\arg\max_{k\in[K]}c({\mathbf{x}})_{k}$. This clustering procedure is formally described in Algorithm 2.

2 Empirical Evaluation

We evaluated the proposed clustering method on three different datasets: MNIST, Fashion-MNIST, and a subset of the Synthetic Traffic Signs Dataset containing 10 selected classes (see Table 1). To evaluate clustering performance we adopt two commonly used metrics: Normalized Mutual Information (NMI), and Clustering Accuracy (ACC). Clustering accuracy measures the accuracy of the hard-assignment to clusters, with respect to the best permutation of the dataset’s ground-truth labels. Normalized Mutual Information measures the mutual information between the ground-truth labels and the predicted labels based on the clustering method. The range of both metrics is $$ where a larger value indicates more precise clustering results. Both metrics are formally defined as follows:

Above $X$ denotes the dataset on which clustering is performed, $y({\mathbf{x}})$ denotes the ground-truth label of sample ${\mathbf{x}}$, $c({\mathbf{x}})$ denotes the cluster assignment of sample ${\mathbf{x}}$, $H$ denotes entropy, $I$ denotes mutual-information, and $S_{N}$ denotes the set of all permutations on $N$ elements (the number of classes in the dataset). The unsupervised clustering scores of our method are presented in Table 3.

When evaluated on the MNIST dataset, our method outperforms other recent alternative methods, and, to the best of our knowledge, achieves state-of-the-art performance. Less impressive performance is achieved on the Fashion-MNIST dataset. The fact that this dataset is characterized by small inter-class diversity may be the underlying reason for this. In such a case, an GM-GAN with merely $K=10$ Gaussians may struggle to model this dataset in such a way where different Gaussians in the latent space are mapped to different classes in the data space. Thus, some Gaussians in the latent space are mapped to multiple classes in the data-space and therefore the resulting performance of our method deteriorates. In such cases improved performance can potentially be achieved by increasing the number of Gaussians forming the latent space; however, in this configuration it would not be possible to quantitatively measure the performance of the resulting dataset partitioning, thus we skip this test.

Summary and Discussion

This work is motivated by the observation that the commonly used GAN architecture may be ill suited to model data in such cases where the training set is characterized by large inter-class and intra-class diversity, a common case with real-world datasets these days. To address this problem we propose a variant of the basic GAN model where the probability distribution over the latent space is a mixture of Gaussians, a multi-modal distribution much like the target data distribution which the GAN is trained to model. Additionally, we propose a supervised variant of this model which is capable of conditional sample synthesis. We note that these modifications can be applied to any GAN model, regardless of the specifics of the loss function and architecture.

In order to compare the different models, we note that the performance of GANs, and perhaps other families of generative models, exhibits a certain trade-off between the quality of their generated samples and the diversity of those samples. Therefore arguably the performance of such models must be evaluated by separately measuring the quality and the diversity of the generated samples, unlike common practice. For this purpose we propose a scoring method which separately takes into account these two factors. The proposed score can be modified, based on the application’s requirement, by adjusting the proportion of each factor when employing the trained model.

In our empirical study, using both synthetic and real-world datasets, we quantitatively showed that GM-GANs outperform baselines, both when evaluated using the commonly used Inception Score [SGZ+16], and when evaluated using our own alternative scoring method. We also demonstrated how the quality-diversity trade-off offered by our models can be controlled, by altering, post-training, the probability distribution of the latent space. This allows one to sample higher-quality, lower-diversity samples or vice versa, according to one’s needs. Finally, we qualitatively demonstrated how the unsupervised variant of GM-GAN tends to map latent vectors sampled from different Gaussians in the latent space to samples of different classes in the data space. We further showed how this phenomenon can be exploited for the task of unsupervised clustering, and backed our method with quantitative evaluation.

It is important to emphasize that the architectural modifications we proposed in this work are orthogonal to, and can be used in conjunction with, other architectural improvements suggested in prior art, such as those reviewed in Section 1. Thus, other variants of GANs can also benefit from adopting the proposed method. For example, one may use a multi-modal prior in conjunction with the popular WGAN-GP model [GAA+17] in order to achieve better training stability as well as higher quality sample generation, or the InfoGAN model [CDH+16] in order to improve the modeling of multi-modal attributes.

The GM-GAN model, along with the proposed scoring method, allow one to control the quality-diversity trade-off and directly choose between drawing higher-quality or higher-diversity samples. This can be useful in cases where these factors have an influence on the application for which the GAN is employed. For example, when a GAN is used to boost the performance of a classifier trained in a semi-supervised learning settings, e.g. [Spr15a, SGZ+16], both the quality and the diversity of the synthetic samples can influence the performance of the target classifier. Thus one may want to carefully choose the right proportions of these two factors when employing the model. Another example is Curriculum Learning [BLCW09, WCA18], a setting in which training samples are gradually revealed from the easiest to the most difficult. Here one can employ our method in order to initially generate high quality and low diversity samples, which are arguably easier, followed by samples of higher diversity and lower quality.

References