High-Fidelity Image Generation With Fewer Labels

Introduction

Deep generative models have received a great deal of attention due to their power to learn complex high-dimensional distributions, such as distributions over natural images (van den Oord et al., 2016b; Dinh et al., 2017; Brock et al., 2019), videos (Kalchbrenner et al., 2017), and audio (van den Oord et al., 2016a). Recent progress was driven by scalable training of large-scale models (Brock et al., 2019; Menick & Kalchbrenner, 2019), architectural modifications (Zhang et al., 2019; Chen et al., 2019a; Karras et al., 2019), and normalization techniques (Miyato et al., 2018).

Currently, high-fidelity natural image generation hinges upon having access to vast quantities of labeled data. The labels induce rich side information into the training process which effectively decomposes the extremely challenging image generation task into semantically meaningful sub-tasks.

However, this dependence on vast quantities of labeled data is at odds with the fact that most data is unlabeled, and labeling itself is often costly and error-prone. Despite the recent progress on unsupervised image generation, the gap between conditional and unsupervised models in terms of sample quality is significant.

In this work, we take a significant step towards closing the gap between conditional and unsupervised generation of high-fidelity images using generative adversarial networks (GANs). We leverage two simple yet powerful concepts:

Self-supervised learning: A semantic feature extractor for the training data can be learned via self-supervision, and the resulting feature representation can then be employed to guide the GAN training process.

Semi-supervised learning: Labels for the entire training set can be inferred from a small subset of labeled training images and the inferred labels can be used as conditional information for GAN training.

propose and study various approaches to reduce or fully omit ground-truth label information for natural image generation tasks,

achieve a new state of the art (SOTA) in unsupervised generation on imagenet, match the SOTA on $128\times 128$ imagenet using only 10% of the labels, and set a new SOTA (measured by FID) using 20% of the labels, and

open-source all the code used for the experiments at github.com/google/compare_gan.

Background and related work

Besides BigGAN (Brock et al., 2019) only a few prior methods have managed to scale GANs to ImageNet, most of them relying on class-conditional generation using labels. One of the earliest attempts are GANs with auxiliary classifier (AC-GANs) (Odena et al., 2017) which feed one-hot encoded label information with the latent code to the generator and equip the discriminator with an auxiliary head predicting the image class in addition to whether the input is real or fake. More recent approaches rely on a label projection layer in the discriminator, essentially resulting in per-class real/fake classification (Miyato & Koyama, 2018), and self-attention in the generator (Zhang et al., 2019). Both methods use modulated batch normalization (De Vries et al., 2017) to provide label information to the generator. On the unsupervised side, Chen et al. (2019b) showed that auxiliary rotation loss added to the discriminator has a stabilizing effect on the training. Finally, appropriate gradient regularization enables scaling MMD-GANs to ImageNet without using labels (Arbel et al., 2018).

Semi-supervised GANs Several recent works leveraged GANs for semi-supervised learning of classifiers. Both Salimans et al. (2016) and Odena (2016) train a discriminator that classifies its input into $K+1$ classes: $K$ image classes for real images, and one class for generated images. Similarly, Springenberg (2016) extends the standard GAN objective to $K$ classes. This approach was also considered by Li et al. (2017) where separate discriminator and classifier models are applied. Other approaches incorporate inference models to predict missing labels (Deng et al., 2017) or harness joint distribution (of labels and data) matching for semi-supervised learning (Gan et al., 2017). Up to our knowledge, improvements in sample quality through partial label information are only reported in Li et al. (2017); Deng et al. (2017); Sricharan et al. (2017), all of which consider only low-resolution data sets from a restricted domain.

Self-supervised learning Self-supervised learning methods employ a label-free auxiliary task to learn a semantic feature representation of the data. This approach was successfully applied to different data modalities, such as images (Doersch et al., 2015; Caron et al., 2018), video (Agrawal et al., 2015; Lee et al., 2017), and robotics (Jang et al., 2018; Pinto & Gupta, 2016). The current state-of-the-art method on imagenet is due to Gidaris et al. (2018) who proposed predicting the rotation angle of rotated training images as an auxiliary task. This simple self-supervision approach yields representations which are useful for downstream image classification tasks. Other forms of self-supervision include predicting relative locations of disjoint image patches of a given image (Doersch et al., 2015; Mundhenk et al., 2018) or estimating the permutation of randomly swapped image patches on a regular grid (Noroozi & Favaro, 2016). A study on self-supervised learning with modern neural architectures is provided in Kolesnikov et al. (2019).

Reducing the appetite for labeled data

In a nutshell, instead of providing hand-annotated ground truth labels for real images to the discriminator, we will provide inferred ones. To obtain these labels we will make use of recent advancements in self- and semi-supervised learning. We propose and study several different methods with different degrees of computational and conceptual complexity. We emphasize that our work focuses on using few labels to improve the quality of the generative model, rather than training a powerful classifier from a few labels as extensively studied in prior work on semi-supervised GANs.

Before introducing these methods in detail, we discuss how label information is used in state-of-the-art GANs. The following exposition assumes familiarity with the basics of the GAN framework (Goodfellow et al., 2014).

We proceed with describing the proposed pre-trained and co-training approaches to infer labels for GAN training in Sections 3.1 and 3.2, respectively.

We first learn a representation of the real training data using a state-of-the-art self-supervised approach (Gidaris et al., 2018; Kolesnikov et al., 2019), perform clustering on this representation, and use the cluster assignments as a replacement for labels. Following Gidaris et al. (2018) we learn the feature extractor $F$ (typically a convolutional neural network) by minimizing the following self-supervision loss

where $\mathcal{R}$ is the set of the $4$ rotation degrees $\{<span class="katex-display"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mo separator="true">,</mo></mrow><annotation encoding="application/x-tex">,</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.3em;vertical-align:-0.1944em;"></span><span class="mpunct">,</span></span></span></span></span>,<span class="katex-display"><span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><mo separator="true">,</mo></mrow><annotation encoding="application/x-tex">,</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.3em;vertical-align:-0.1944em;"></span><span class="mpunct">,</span></span></span></span></span>\}$, $x^{r}$ is the image $x$ rotated by $r$, and $c_{\text{R}}$ is a linear classifier predicting the rotation degree $r$. After learning the feature extractor $F$, we apply mini batch $k$-Means clustering (Sculley, 2010) on the representations of the training images. Finally, given the cluster assignment function $\hat{y}_{\text{CL}}=c_{\text{CL}}(F(x))$ we train the GAN using the hinge loss, alternatively minimizing the discriminator loss $\mathcal{L}_{D}$ and generator loss $\mathcal{L}_{G}$, namely

where $\hat{p}(z,y)=p(z)\hat{p}(y)$ is the prior distribution with $p(z)=\mathcal{N}(0,I)$ and $\hat{p}(y)$ the empirical distribution of the cluster labels $c_{\text{CL}}(F(x))$ over the training set. We call this approach Clustering and illustrate it in Figure 4.

Semi-supervised method

While semi-supervised learning is an active area of research and a large variety of algorithms has been proposed, we follow Zhai et al. (2019) and simply extend the self-supervised approach described in the previous paragraph with a semi-supervised loss. This ensures that the two approaches are comparable in terms of model capacity and computational cost. Assuming we are provided with labels for a subset of the training data, we attempt to learn a good feature representation via self-supervision and simultaneously train a good linear classifier on the so-obtained representation (using the provided labels).Note that an even simpler approach would be to first learn the representation via self-supervision and subsequently the linear classifier, but we observed that learning the representation and classifier simultaneously leads to better results. More formally, we minimize the loss

where $p(z,y)=p(z)p(y)$ with $p(z)=\mathcal{N}(0,I)$ and $p(y)$ uniform categorical. We use the abbreviation S2GAN for this method.

2 Co-training approach

The main drawback of the transfer-based methods is that one needs to train a feature extractor $F$ via self-supervision and learn an inference mechanism for the labels (linear classifier or clustering). In what follows we detail co-training approaches that avoid this two-step procedure and learn to infer label information during GAN training.

where the first term corresponds to standard conditional training on ($k\%$) labeled real images, the second term is the cross-entropy loss (with weight $\lambda>0$) for the auxiliary classifier $c_{\text{CT}}$ on the labeled real images, the third term is an unsupervised discriminator loss where the labels for the unlabeled real images are predicted by $c_{\text{CT}}$, and the last term is the standard conditional discriminator loss on the generated data. We use the abbreviation S2GAN-CO for this method. See Figure 5 for an illustration.

3 Self-supervision during GAN training

respectively, where $\alpha,\beta>0$ are weights to balance the loss terms. This approach is illustrated in Figure 6.

Experimental setup

GANs are notoriously unstable to train and their performance strongly depends on the capacity of the neural architecture, optimization hyperparameters, and appropriate regularization (Lucic et al., 2018; Kurach et al., 2019). We implemented the conditional BigGAN architecture (Brock et al., 2019) which achieves state-of-the-art results on ImageNet.We dissected the model checkpoints released by Brock et al. (2019) to obtain exact counts of trainable parameters and their dimensions, and match them to byte level (cf. Tables 11 and 10 in Appendix B). We want to emphasize that at this point this methodology is bleeding-edge and successful state-of-the-art methods require careful architecture-level tuning. To foster reproducibility we meticulously detail this architecture at tensor-level detail in Appendix B and open-source our code at https://github.com/google/compare_gan. We use exactly the same optimization hyper-parameters as Brock et al. (2019). Specifically, we employ the Adam optimizer with the learning rates $5\cdot 10^{-5}$ for the generator and $2\cdot 10^{-4}$ for the discriminator ($\beta_{1}=0$, $\beta_{2}=0.999$). We train for 250k generator steps with 2 discriminator steps before each generator step. The batch size was fixed to 2048, and we use a latent code $z$ with $120$ dimensions. We employ spectral normalization in both generator and discriminator. In contrast to BigGAN, we do not apply orthogonal regularization as this was observed to only marginally improve sample quality (cf. Table 1 in Brock et al. (2019)) and we do not use the truncation trick.

Datasets We focus primarily on imagenet, the largest and most diverse image data set commonly used to evaluate GANs. imagenet contains $1.3$M training images and $50$k test images, each corresponding to one of 1k object classes. We resize the images to $128\times 128\times 3$ as done in Miyato & Koyama (2018) and Zhang et al. (2019). Partially labeled data sets for the semi-supervised approaches are obtained by randomly selecting $k\%$ of the samples from each class.

Methods We conduct an extensive comparison of methods detailed in Table 1, namely: Unmodified BigGAN, the unsupervised methods Single label, Random label, Clustering, and the semi-supervised methods S2GAN and S2GAN-CO. In all S2GAN-CO experiments we use soft labels, i.e., the soft-max output of $c_{\text{CT}}$ instead of one-hot encoded hard estimates, as we observed in preliminary experiments that this stabilizes training. For S2GAN we use hard labels by default, but investigate the effect of soft labels in separate experiments. For all semi-supervised methods we have access only to $k\%$ of the ground truth labels where $k\in\{5,10,20\}$. As an additional baseline, we retain $k\%$ labeled real images and discard all unlabeled real images, then using the remaining labeled images to train BigGAN (the resulting model is designated by BigGAN-$k\%$). Finally, we explore the effect of self-supervision during GAN training on the unsupervised and semi-supervised methods.

We train every model three times with a different random seed and report the median FID and the median IS. With the exception of the Single label and BigGAN-$k\%$, the standard deviation of the mean across three runs is very low. We therefore defer tables with the mean FID and IS values and standard deviations to Appendix D. All models are trained on 128 cores of a Google TPU v3 Pod with BatchNorm statistics synchronized across cores.

Unsupervised approaches For Clustering we simply used the best available self-supervised rotation model from Kolesnikov et al. (2019). The number of clusters for Clustering is selected from the set $\{50,100,200,500,1000\}$. The other unsupervised approaches do not have hyper-parameters.

Pre-trained and co-training approaches We employ the wide ResNet-50 v2 architecture with widening factor 16 (Zagoruyko & Komodakis, 2016) for the feature extractor $F$ in the pre-trained approaches described in Section 3.1.

Self-supervision during GAN training For all approaches we use the recommend parameter $\alpha=0.2$ from (Chen et al., 2019b) in (5) and do a small sweep for $\beta$ in (4). For the values tried ($\{0.25,0.5,1.0,2\}$) we do not see a large effect and use $\beta=0.5$ for S3GAN. For S3GAN-CO we did not repeat the sweep, and instead used $\beta=1.0$.

Results and discussion

Recall that the main goal of this work is to match (or outperform) the fully supervised BigGAN in an unsupervised fashion, or with a small subset of labeled data. In the following, we discuss the advantages and drawbacks of the analyzed approaches with respect to this goal.

As a baseline, our reimplementation of BigGAN obtains an FID of 8.4 and IS of 75.0, and hence reproduces the result reported by Brock et al. (2019) in terms of FID. We observed some differences in training dynamics, which we discuss in detail in Section 5.4.

The results for unsupervised approaches are summarized in Figure 7 and Table 2. The fully unsupervised Random label and Single label models both achieve a similar FID of $\sim 25$ and IS of $\sim 20$. This is a quite considerable gap compared to BigGAN and indicates that additional supervision is necessary. We note that one of the three Single label models collapsed whereas all three Random label models trained stably for 250k generator iterations.

Pre-training a semantic representation using self-supervision and clustering the training data on this representation as done by Clustering reduces the FID by about $10\%$ and increases IS by about $10\%$. These results were obtained for 50 clusters, all other options led to worse results. While this performance is still considerably worse than that of BigGAN this result is the current SOTA in unsupervised image generation (Chen et al. (2019b) report an FID of 33 for unsupervised generation).

Example images from the clustering are shown in Figures 14, 15, and 16 in the supplementary material. The clustering is clearly meaningful and groups similar objects within the same cluster. Furthermore, the objects generated by Clustering conditionally on a given cluster index reflect the distribution of the training data belonging to the corresponding cluster. On the other hand, we can clearly observe multiple classes being present in the same cluster. This is to be expected when under-clustering to $50$ clusters. Interestingly, clustering to many more clusters (say $500$) yields results similar to Single label.

2 Semi-supervised approaches

Pre-trained The S2GAN model where we use the classifier pre-trained with both a self-supervised and semi-supervised loss (cf. Section 3.1) matches the BigGAN baseline when $20\%$ of the labels are used and incurs a minor increase in FID when $10\%$ and $5\%$ are used (cf. Table 3). We stress that this is despite the fact that the classifier used to infer the labels has a top-1 accuracy of only $50\%$, $63\%$, and $71\%$ for $5\%$, $10\%$, and $20\%$ labeled data, respectively (cf. Table 3), compared to $100\%$ of the original labels. The results are shown in Table 4 and Figure 8, and random samples as well as interpolations can be found in Figures 9–17 in the supplementary material.

The results for our co-trained model S2GAN-CO which trains a linear classifier in semi-supervised fashion on top of the discriminator representation during GAN training (cf. Section 3.2) are shown in Table 4. It can be seen that S2GAN-CO outperforms all fully unsupervised approaches for all considered label percentages. While the gap between S2GAN-CO with $5\%$ labels and Clustering in terms of FID is small, S2GAN-CO has a considerably larger IS. When using $20\%$ labeled training examples S2GAN-CO obtains an FID of 13.9 and an IS of 49.2, which is remarkably close to BigGAN and S2GAN given the simplicity of the S2GAN-CO approach. As the the percentage of labels decreases, the gap between S2GAN and S2GAN-CO increases.

Interestingly, S2GAN-CO does not seem to train less stably than S2GAN approaches even though it is forced to learn the classifier during GAN training. This is particularly remarkable as the BigGAN-$k\%$ approaches, where we only retain the labeled data for training and discard all unlabeled data, are very unstable and collapse after 60k to 120k iterations, for all three random seeds and for both $10\%$ and $20\%$ labeled data.

3 Self-supervision during GAN training

So far we have seen that the pre-trained semi-supervised approach, namely S2GAN, is able to achieve state-of-the-art performance for $20\%$ labeled data. Here we investigate whether self-supervision during GAN training as described in Section 3.3 can lead to further improvements. Table 4 and Figure 8 show the experimental results for S3GAN, namely S2GAN coupled with self-supervision in the discriminator.

Self-supervision leads to a reduction in FID and increase in IS across all considered settings. In particular we can match the state-of-the-art BigGAN with only $10\%$ of the labels and outperform it using $20\%$ labels, both in terms of FID and IS.

For S3GAN the improvements due to self-supervision during GAN training in FID are considerable, around $10\%$ in most of the cases. Tuning the parameter $\beta$ of the discriminator self-supervision loss in (4) did not dramatically increase the benefits of self-supervision during GAN training, at least for the range of values considered. As shown in Tables 2 and 4, self-supervision during GAN training (with default parameters $\alpha,\beta$) also leads to improvements by $5$ to $10\%$ for both S2GAN-CO and Single label. In summary, self-supervision during GAN training with default parameters leads to a stable improvement across all approaches.

4 Other insights

Effect of soft labels A design choice available to practitioners is whether to use hard labels (i.e., the argmax over the logits), or soft labels (softmax over the logits) for S2GAN and S3GAN (recall that we use soft labels by default for S2GAN-CO and S3GAN-CO). Our initial expectation was that soft labels should help when very little labeled data is available, as soft labels carry more information which can potentially be exploited by the projection discriminator. Surprisingly, the results presented in Table 5 show clearly that the opposite is true. Our current hypothesis is that this is due to the way labels are incorporated in the projection discriminator, but we do not have empirical evidence yet.

Optimization dynamics Brock et al. (2019) report the FID and IS of the model just before the collapse, which can be seen as a form of early stopping. In contrast, we manage to stably train the proposed models for 250k generator iterations. In particular, we also observe stable training for our “vanilla” BigGAN implementation. The evolution of the FID and IS as a function of the training steps is shown in Figure 21 in the appendix. At this point we can only speculate about the origin of this difference. We finally note that by tuning the learning rate we obtained slightly different (but still stable) training dynamics in terms of IS, achieving FID $6.9$ and IS $98$ for S3GAN with $20\%$ labels.

Higher resolution and going below 5% labels Training these models at higher resolution becomes computationally harder and it necessitates tuning the learning rate. We trained several S3GAN models at $256\times 256$ resolution and show the resulting samples in Figures 12–13 and interpolations in Figures 19–20. We also conducted S3GAN experiments in which only $2.5\%$ of the labels are used and observed FID of $13.6$ and IS of $46.3$. This indicates that given a small number of samples one can significantly outperform the unsupervised approaches (c.f. Figure 7).

Conclusion and future Work

In this work we investigated several avenues to reduce the appetite for labeled data in state-of-the-art GANs. We showed that recent advances in self and semi-supervised learning can be used to achieve a new state of the art, both for unsupervised and supervised natural image synthesis.

We believe that this is a great first step towards the ultimate goal of few-shot high-fidelity image synthesis. There are several important directions for future work: (i) investigating the applicability of these techniques for even larger and more diverse data sets, and (ii) investigating the impact of other self- and semi-supervised approaches on the model quality. (iii) investigating the impact of self-supervision in other deep generative models. Finally, we would like to emphasize that further progress might be hindered by the engineering challenges related to training large-scale generative adversarial networks. To help alleviate this issue and to foster reproducibility, we have open-sourced all the code used for the experiments.

Acknowledgments

We would like to thank Ting Chen and Neil Houlsby for fruitful discussions on self-supervision and its application to GANs. We would like to thank Lucas Beyer, Alexander Kolesnikov, and Avital Oliver for helpful discussions on self-supervised semi-supervised learning. We would like to thank Karol Kurach and Marcin Michalski their major contributions the Compare GAN library. We would also like to thank the BigGAN team (Andy Brock, Jeff Donahue, and Karen Simonyan) for their insights into training GANs on TPUs. Finally, we are grateful for the support of members of the Google Brain team in Zurich. This work was partially done while Michael Tschannen was at Google Research.

References

Appendix A Additional samples and interpolations

Appendix B Architectural details

The ResNet architecture implemented following Brock et al. (2019) is described in Tables 6 and 8. We use the abbreviations RS for resample, BN for batch normalization, and cBN for conditional BN (Dumoulin et al., 2017; De Vries et al., 2017). In the resample column, we indicate downscale(D)/upscale(U)/none(-) setting and in the spectral norm column shows whether spectral normalization is applied to all weights in the layer. In Table 8, $y$ stands for the labels and $\bm{h}$ is the output from the layer before (i.e., the pre-logit layer). Tables 7 and 9 show ResBlock details. The addition layer merges the shortcut path and the convolution path by adding them. $h$ and $w$ are the input height and width of the ResBlock, $c_{i}$ and $c_{o}$ are the input channels and output channels for a ResBlock. For the last ResBlock in the discriminator without downsampling, we simply drop the shortcut layer from ResBlock. We list all the trainable variables and their shape in Tables 10 and 11.

Appendix C FID and IS training curves

Appendix D FID and IS: Mean and standard deviations