How Generative Adversarial Networks and Their Variants Work: An Overview

Introduction

Recently, in the machine learning field, generative models have become more important and popular because of their applicability in various fields. Their capability to represent complex and high-dimensional data can be utilized in treating images , videos , music generation , natural languages and other academic domains such as medical images and security . Specifically, generative models are highly useful for image to image translation (See Figure 1) which transfers images to another specific domain, image super-resolution , changing some features of an object in an image and predicting the next frames of a video . In addition, generative models can be the solution for various problems in the machine learning field such as semisupervised learning , which tries to address the lack of labeled data, and domain adaptation , which leverages known knowledge for some tasks in other domains where only little information is given.

Generative Adversarial Networks (GANs) were proposed to solve the disadvantages of other generative models. Instead of maximizing the likelihood, GAN introduces the concept of adversarial learning between the generator and the discriminator. The generator and the discriminator act as adversaries with respect to each other to produce real-like samples. The generator is a continuous, differentiable transformation function mapping a prior distribution $p_{z}$ from the latent space $\mathcal{Z}$ into the data space $\mathcal{X}$ that tries to fool the discriminator. The discriminator distinguishes its input whether it comes from the real data distribution or the generator. The basic intuition behind the adversarial learning is that as the generator tries to deceive the discriminator which also evolves against the generator, the generator improves. This adversarial process gives GAN notable advantages over the other generative models.

For these advantages, GAN has been gaining considerable attention, and the desire to use GAN in many fields is growing. In this study, we explain GAN in detail which generates sharper and better real-like samples than the other generative models by adopting two components, the generator and the discriminator. We look into how GAN works theoretically and how GAN has been applied to various applications.

Table LABEL:tab:overview1 shows GAN and GAN variants which will be discussed in Section 2 and 3. In Section 2, we first present a standard objective function of a GAN and describe how its components work. After that, we present various objective functions proposed recently, focusing on their similarities in terms of the feature matching problem. We then explain the architecture of GAN extending the discussion to dominant obstacles caused by optimizing a minimax problem, especially a mode collapse, and how to address those issues.

In Section 3, we discuss how GAN can be exploited to learn the latent space where a compressed and low dimensional representation of data lies. In particular, we emphasize how the GAN extracts the latent space from the data space with autoencoder frameworks. Section 4 provides several extensions of the GAN applied to other domains and various topics as shown in Table LABEL:tab:overview_app. In Section 5, we observe a macroscopic view of GAN, especially why GAN is advantageous over other generative models. Finally, Section 6 concludes the paper.

L1/.style=fill=green,edge=black, line width=1pt, L2/.style=fill=orange,edge=orange,line width=2pt, L3/.style=fill=yellow,edge=yellow,line width=2pt, L4/.style=fill=pink,edge=pink,line width=2pt,

[ caption = An overview of GANs discussed in Section 2 and 3., label = tab:overview1, star, width = doinside = , mincapwidth=]lll Subject Topic Reference

Object f-divergence GAN , f-GAN , LSGAN functions IPM WGAN , WGAN-GP , FISHER GAN , McGAN , MMDGAN

Architecture DCGAN DCGAN Hierarchy StackedGAN , GoGAN , Progressive GAN Auto encoder BEGAN , EBGAN , MAGAN Issues Theoretical analysis Towards principled methods for training GANs Generalization and equilibrium in GAN Mode collapse MRGAN , DRAGAN , MAD-GAN , Unrolled GAN

Latent space Decomposition CGAN , ACGAN , InfoGAN , ss-InfoGAN

Encoder ALI , BiGAN , Adversarial Generator-Encoder Networks VAE VAEGAN , $\alpha$-GAN

[ caption = Categorization of GANs applied for various topics., label = tab:overview_app, star, width = pos = ht, doinside = , mincapwidth=]lll Domain Topic Reference

Image Image translation Pix2pix , PAN , CycleGAN , DiscoGAN

Super resolution SRGAN Object detection SeGAN , Perceptual GAN for small object detection Object transfiguration GeneGAN , GP-GAN Joint image generation Coupled GAN Video generation VGAN , Pose-GAN , MoCoGAN Text to image Stack GAN , TAC-GAN Change facial attributes SD-GAN , SL-GAN , DR-GAN , AGEGAN

Sequential data Music generation C-RNN-GAN , SeqGAN , ORGAN Text generation RankGAN Speech conversion VAW-GAN

Semi-supervised learning SSL-GAN , CatGAN , Triple-GAN Others Domain adaptation DANN , CyCADA Unsupervised pixel-level domain adaptation

Continual learning Deep generative replay

Medical image segmentation DI2IN , SCAN , SegAN Steganography Steganography GAN , Secure steganography GAN

Generative Adversarial Networks

$V(G,D)$ is a binary cross entropy function that is commonly used in binary classification problems . Note that $G$ maps $z$ from $\mathcal{Z}$ into the element of $\mathcal{X}$, whereas $D$ takes an input $x$ and distinguishes whether $x$ is a real sample or a fake sample generated by $G$.

As $D$ wants to classify real or fake samples, $V(G,D)$ is a natural choice for an objective function in aspect of the classification problem. From $D$’s perspective, if a sample comes from real data, $D$ will maximize its output, while if a sample comes from $G$, $D$ will minimize its output; thus, the $\log(1-D(G(z)))$ term appears in Equation 1. Simultaneously, $G$ wants to deceive $D$, so it tries to maximize $D$’s output when a fake sample is presented to $D$. Consequently, $D$ tries to maximize $V(G,D)$ while $G$ tries to minimize $V(G,D)$, thus forming the minimax relationship in Equation 1. Figure 2 shows an illustration of the GAN.

Beyond the theoretical support of GAN, the above paragraph leads us to infer two points. First, from the optimal discriminator in the above, GAN can be connected into the density ratio trick . That is, the density ratio between the data distribution and the generated data distribution as follows:

where $y=0$ and $y=1$ indicate the generated data and the real data, respectively and $p(y=1)=p(y=0)$ is assumed. This means that GAN addresses the intractability of the likelihood by just using the relative behavior of the two distributions , and transferring this information to the generator to produce real-like samples. Second, GAN can be interpreted to measure the discrepancy between the generated data distribution and the real data distribution and then learn to reduce it. The discriminator is used to implicitly measure the discrepancy.

Despite the advantage and theoretical support of GAN, many shortcomings has been found due to the practical issues and inability to implement the assumption in theory including the infinite capacity of the discriminator. There have been many attempts to solve these issues by changing the objective function, the architecture, etc. Holding the fundamental framework of GAN, we assess variants of the object function and the architectures proposed for the development of GAN. We then focus on the crucial failures of GAN and how to address those issues.

f-GAN generalizes the GAN objective function in terms of f-divergence under an arbitrary convex function $f$. As we do not know the distributions exactly, Equation 3 should be estimated through a tractable form such as an expectation form. By using the convex conjugate $f(u)=\sup_{t\in domf^{\star}}(tu-f^{\star}(t))$, Equation 3 can be reformulated as follows:

where $f^{\star}$ is a Fenchel conjugate of a convex function $f$ and $domf^{\star}$ indicates a domain of $f^{\star}$.

The standard GAN uses a sigmoid cross entropy loss for the discriminator to classify whether its input is real or fake. However, if a generated sample is well classified as real by the discriminator, there would be no reason for the generator to be updated even though the generated sample is located far from the real data distribution. A sigmoid cross entropy loss can barely push such generated samples towards real data distribution since its classification role has been achieved.

Motivated by this phenomenon, least-square GAN (LSGAN) replaces a sigmoid cross entropy loss with a least square loss, which directly penalizes fake samples by moving them close to the real data distribution. Compared to Equation 1, LSGAN solves the following problems:

where $a,b$ and $c$ refer to the baseline values for the discriminator.

Equations 7 and 8 use a least square loss, under which the discriminator is forced to have designated values ($a,b$ and $c$) for the real samples and the generated samples, respectively, rather than a probability for the real or fake samples. Thus, in contrary to a sigmoid cross entropy loss, a least square loss not only classifies the real samples and the generated samples but also pushes generated samples closer to the real data distribution. In addition, LSGAN can be connected to an f-divergence framework as shown in Table 1.

1.2 Integral probability metric

In Equation 13, the range of $v$ determines the semantic meanings of the corresponding IPM metrics. From now on, we discuss IPM metric variants such as the Wasserstein metric, maximum mean discrepancy (MMD), and the Fisher metric based on Equation 13.

The benefit of the EM distance over other metrics is that it is a more sensible objective function when learning distributions with the support of a low-dimensional manifold. The article on WGAN shows that EM distance is the weakest convergent metric in that the convergent sequence under the EM distance does not converge under other metrics and it is continuous and differentiable almost everywhere under the Lipschitz condition, which standard feed-forward neural networks satisfy. Thus, EM distance results in a more tolerable measure than do other distances such as KLD and total variance distance regarding convergence of the distance.

As the $\inf$ term in Equation 14 is intractable, it is converted into a tractable equation via Kantorovich-Rubinstein duality with the Lipschitz function class , ; i.e., $f:X\rightarrow R$, satisfying $d_{R}(f(x_{1}),f(x_{2}))\leq 1\times d_{X}(x_{1},x_{2})$, $\forall x_{1},x_{2}\in X$ where $d_{X}$ denotes the distance metric in the domain $X$. A duality of Equation 14 is as follows:

WGAN with gradient penalty (WGAN-GP) points out that the weight clipping for the critic while training WGAN incurs a pathological behavior of the discriminator and suggests adding a penalizing term of the gradient’s norm instead of the weight clipping. It shows that guaranteeing the Lipschitz condition for the critic via weight clipping constraints the critic in a very limited subset of all Lipschitz functions; this biases the critic toward a simple function. The weight clipping also creates a gradient problem as it pushes weights to the extremes of the clipping range. Instead of the weight clipping, adding a gradient penalty term to Equation 15 for the purpose of implementing the Lipschitz condition by directly constraining the gradient of the critic has been suggested .

From Equation 10, we can generalize several IPM metrics under the measure of the inner product. If we constrain $v$ with $p$ norm where $p$ is a nonnegative integer, we can derive Equation 18 as a feature matching problem as follows by adding the $\|v\|_{p}\leq 1$ condition where $\|v\|_{p}=\{\Sigma_{i=1}^{m}v_{i}^{p}\}^{1/p}$. It should be noted that, with conjugate exponent $q$ of $p$ such that $\frac{1}{p}+\frac{1}{q}=1$, the dual norm of norm $p$ satisfies $\|x\|_{q}=sup\{{<v,x>:\|v\|_{p}\leq 1}\}$ by Holder’s inequality . Motivated by this dual norm property , we can derive a $l_{q}$ mean matching problem as follows:

Similar to other IPM metrics, MMD distance is continuous and differentiable almost everywhere in $\theta$. It can also be understood under the IPM framework with function class $\mathcal{F}=\mathcal{H}_{K}$ as discussed above. By introducing an RKHS with kernel $k$, MMD distance has an advantage over other feature matching metrics in that kernel $k$ can represent various feature space by mapping input data $x$ into other feature space. In particular, MMDGAN can also be connected with WGAN when $f_{\phi}$ is composited to a linear kernel with an output dimension of 1 instead of a Gaussian kernel. The moment matching technique using the Gaussian kernel also has an advantage over WGAN in that it can match even an infinite order of moments since exponential form can be represented as an infinite order via Taylor expansion while WGAN can be treated as a first-order moment matching problem as discussed above. However, a great disadvantage of measuring MMD distance is that computational cost grows quadratically as the number of samples grows .

Meanwhile, CramerGAN argues that the Wasserstein distance incurs biased gradients, suggesting the energy distance between two distributions. In fact, it measures energy distance indirectly in the data manifold but with transformation function $h$. However, CramerGAN can be thought of as the distance in the kernel embedded space of MMDGAN, which forces $h$ to be injective by the additional autoencoder reconstruction loss as discussed above.

In addition to standard IPM in Equation 9, Fisher GAN incorporates a data-dependent constraint by the following equations:

Equation 23 is motivated by fisher linear discriminant analysis (FLDA) which not only maximizes the mean difference but also reduces the total with-in class variance of two distributions. Equation 24 follows from the constraining numerator of Equation 23 to be 1. It is also, as are other IPM metrics, interpreted as a mean feature matching problem under the somewhat different constraints. Under the definition of Equation 10, Fisher GAN can be converted into another mean feature matching problem with second order moment constraint. A mean feature matching problem derived from the FLDA concept is as follows:

The f-divergence family, which can be defined as in Equation 3 with a convex function $f$, has restrictions in that as the dimension $d$ of the data space $x\in\mathcal{X}=R^{d}$ increases, the f-divergence is highly difficult to estimate, and the supports of two distributions tends to be unaligned, which leads a divergence value to infinity . Even though Equation 6 derives a variational lower bound of Equation 3 which looks very similar to Equation 9, the tightness of the lower bound to the true divergence is not guaranteed in practice and can incur an incorrect, biased estimation.

Sriperumbudur et al. showed that the only non-trivial intersection between the f-divergence family and the IPM family is total variation distance; therefore, the IPM family does not inherit the disadvantages of f-divergence. They also proved that IPM estimators using finite i.i.d. samples are more consistent in convergence whereas the convergence of f-divergence is highly dependent on data distributions.

Consequently, employing an IPM family to measure distance between two distributions is advantageous over using an f-divergence family because IPM families are not affected by data dimension and consistently converge to the true distance between two distributions. Moreover, they do not diverge even though the supports of two distributions are disjointed. In addition, Fisher GAN is also equivalent to the Chi-squared distance , which can be covered by an f-divergence framework. However, with a data dependent constraint, Chi-squared distance can use IPM family characteristics, so it is more robust to unstable training of the f-divergence estimation.

1.3 Auxiliary object functions

Reconstruction is to make an output image of a neural network to be the same as an original input image of a neural network. The purpose of the reconstruction is to encourage the generator to preserve the contents of the original input image or to adopt auto-encoder architecture for the discriminator . For a reconstruction objective function, mostly the L1 norm of the difference of the original input image and the output image is used.

When the reconstruction objective term is used for the generator, the generator is trained to maintain the contents of the original input image. In particular, this operation is crucial for tasks where semantic and several modes of the image should be maintained, such as image translation (detailed in Section 4.1.1.2) and auto-encoder reconstruction (detailed in Section 3.2). The intuition of using reconstruction loss for the generator is that it guides the generator to restore the original input in a supervised learning manner. Without a reconstruction loss, the generator is to simply fool the discriminator, so there is no reason for the generator to maintain crucial parts of the input. As a supervised learning approach, the generator treats the original input image as label information, so we can reduce the space of possible mappings of the generator in a way we desire.

There are some GAN variants using a reconstruction objective function for the discriminator, which is naturally derived from auto-encoder architecture for the discriminator . These GANs are based on the aspect that views the discriminator as an energy function, and will be detailed in Section 2.2.3.

A cross entropy loss for a classification is widely added for many GAN applications where labeled data exists, especially semi-supervised learning and domain adaptation. Cross entropy loss can be directly applied to the discriminator, which gives the discriminator an additional role of classification . Other approaches adopt classifier explicitly, training the classifier jointly with the generator and the discriminator through a cross entropy loss (detailed in Section 4.3 and 4.4).

2 Architecture

An architecture of the generator and the discriminator is important as it highly influences the training stability and performance of GAN. Various papers adopt several techniques such as batch normalization, stacked architecture, and multiple generators and discriminators to promote adversarial learning. We start with deep convolutional GAN (DCGAN) , which provides a remarkable benchmark architecture for other GAN variants.

DCGAN provides significant contributions to GAN in that its suggested convolution neural network (CNN) architecture greatly stabilizes GAN training. DCGAN suggests an architecture guideline in which the generator is modeled with a transposed CNN , and the discriminator is modeled with a CNN with an output dimension 1. It also proposes other techniques such as batch normalization and types of activation functions for the generator and the discriminator to help stabilize the GAN training. As it solves the instability of training GAN only through architecture, it becomes a baseline for modeling various GANs proposed later. For example, Im et al. uses a recurrent neural network (RNN) to generate images motivated by DCGAN. By accumulating images of each time step output of DCGAN and combining several time step images, it produces higher visual quality images.

2.2 Hierarchical architecture

In this section, we describe GAN variants that stack multiple generator-discriminator pairs. Commonly, these GANs generate samples in multiple stages to generate large-scale and high-quality samples. The generator of each stage is utilized or conditioned to help the next stage generator to better produce samples as shown in Figure 3.

StackedGAN attempts to learn a hierarchical representation by stacking several generator-discriminator pairs. For each layer of a generator stack, there exists the generator which produces level-specific representation, the corresponding discriminator training the generator adversarially at each level and an encoder which generates the semantic features of real samples. Figure 3a shows a flowchart of StackedGAN. Each generator tries to produce a plausible feature representation that can deceive the corresponding discriminator, given previously generated features and the corresponding hierarchically encoded features.

Generating high resolution images is highly challenging since a large scale generated image is easily distinguished by the discriminator, so the generator often fails to be trained. Moreover, there is a memory issue in that we are forced to set a low mini-batch size due to the large size of neural networks. Therefore, some studies adopt hierarchical stacks of multiple generators and discriminators . This strategy divides a large complex generator’s mapping space step by step for each GAN pair, making it easier to learn to generate high resolution images. However, Progressive GAN succeeds in to generating high resolution images in a single GAN, making training faster and more stable.

Progressive GAN generates high resolution images by stacking each layer of the generator and the discriminator incrementally as shown in Figure 3b. It starts training to generate a very low spatial resolution (e.g. 4$\times$4), and progressively doubles the resolution of generated images by adding layers to the generator and the discriminator incrementally. In addition, it proposes various training techniques such as pixel normalization, equalized learning rate and mini-batch standard deviation, all of which help GAN training to become more stable.

2.3 Auto encoder architecture

An auto encoder is a neural network for unsupervised learning, It assigns its input as a target value, so it is trained in a self-supervised manner. The reason for self-reconstruction is to encode a compressed representation or features of the input, which is widely utilized with a decoder. Its usage will be detailed in Section 3.2.

In this section, we describe GAN variants which adopt an auto encoder as the discriminator. These GANs view the discriminator as an energy function, not a probabilistic model that distinguishes its input as real or fake. An energy model assigns a low energy for a sample lying near the data manifold (a high data density region), while assigning a high energy for a contrastive sample lying far away from the data manifold (a low data density region). These variants are mainly Energy Based GAN (EBGAN), Boundary Equilibrium GAN (BEGAN) and Margin Adaptation GAN (MAGAN), all of which frame GAN as an energy model.

Since an auto encoder is utilized for the discriminator, a pixelwise reconstruction loss between an input and an output of the discriminator is naturally adopted for the discriminator’s energy function and is defined as follows:

where $AE:R^{N_{x}}\Rightarrow R^{N_{x}}$ denotes an auto encoder and $R^{N_{x}}$ represents the dimension of an input and an output of an auto encoder. It is noted that $D(v)$ in Equation 29 is the pixelwise L1 loss for an autoencoder which maps an input $v\in R^{N_{x}}$ into a positive real number $R^{+}$.

A discriminator with an energy $D(v)$ is trained to give a low energy for a real $v$ and a high energy for a generated $v$. From this point of view, the generator produces a contrastive sample for the discriminator, so that the discriminator is forced to be regularized near the data manifold. Simultaneously, the generator is trained to generate samples near the data manifold since the discriminator is encouraged to reconstruct only real samples. Table 2 presents the summarized details of BEGAN, EBGAN and MAGAN. $L_{G}$ and $L_{D}$ indicate the generator loss and the discriminator loss, respectively, and $[t]^{+}=\max(0,t)$ represents a maximum value between $t$ and , which act as a hinge.

Energy-based GAN (EBGAN) interprets the discriminator as an energy agent, which assigns low energy to real samples and high energy to generated samples. Through the $[m-L(G(z))]^{+}$ term in an objective function, the discriminator ignores generated samples with higher energy than $m$ so the generator attempts to synthesize samples that have lower energy than $m$ to fool the discriminator, which allows that mechanism to stabilize training. Margin adaptation GAN (MAGAN) takes a similar approach to EBGAN, where the only difference is that MAGAN does not fix the margin $m$. MAGAN shows empirically that the energy of the generated sample fluctuates near the margin $m$ and that phenomena with a fixed margin make it difficult to adapt to the changing dynamics of the discriminator and generator. MAGAN suggests that margin $m$ should be adapted to the expected energy of real data, thus, $m$ is monotonically reduced, so the discriminator reconstructs real samples more efficiently.

In addition, because the total variance distance belongs to an IPM family with the function class $\mathcal{F}=\{f:\|f\|_{\infty}=\sup_{x}|f(x)|\leq 1\}$, it can be shown that EBGAN is equivalent to optimizing the total variance distance by using the fact that the discriminator’s output for generated samples is only available for $0\leq D\leq m$ . Because the total variance is the only intersection between IPM and f-divergence , it inherits some disadvantages for estimating f-divergence as discussed by Arjovsky et al. and Sriperumbudur et al. .

3 Obstacles in Training GAN

3.2 Practical issues

A second problem is related to an iterative update algorithm suggested in Goodfellow et al. . We wish to train $D$ until optimal for fixed $G$, but optimizing $D$ in such a manner is computationally expensive. Naturally, we must train $D$ in certain $k$ steps and that scheme causes confusion as to whether it is solving a minimax problem or a maximin problem, because $D$ and $G$ are updated alternatively by gradient descent in the iterative procedure. Unfortunately, solutions of the minimax and maximin problem are not generally equal as follows:

With a maximin problem, minimizing $G$ lies in the inner loop in the right side of Equation 35. $G$ is now forced to place its probability mass on the most likely point where the fixed nonoptimal $D$ believes it likely to be real rather than fake. After $D$ is updated to reject the generated fake one, $G$ attempts to move the probability mass to the other most likely point for fixed $D$. In practice, real data distribution is normally a multi modal distribution but in such a maximin training procedure, $G$ does not cover all modes of the real data distribution because $G$ considers that picking only one mode is enough to fool $D$. Empirically, $G$ tends to cover only a single mode or a few modes of real data distribution. This undesirable nonconvergent situation is called a mode collapse. A mode collapse occurs when many modes in the real data distribution are not represented in the generated samples, resulting in a lack of diversity in the generated samples. It can be simply considered as $G$ being trained to be a non one-to-one function which produces a single output value for several input values.

Furthermore, the problem of the existence of the perfect discriminator we discussed in the above paragraph can be connected to a mode collapse. First, assume $D$ comes to output almost 1 for all real samples and 0 for all fake samples. Then, because $D$ produces values near 1 for all possible modes, there is no need for $G$ to represent all modes of real data probability. The theoretical and practical issues discussed in this section can be summarized as follows.

Because the supports of distributions lie on low dimensional manifolds, there exists the perfect discriminator whose gradients vanish on every data point. Optimizing the generator may be difficult because it is not provided with any information from the discriminator.

GAN training optimizes the discriminator for the fixed generator and the generator for fixed discriminator simultaneously in one loop, but it sometimes behaves as if solving a maximin problem, not a minimax problem. It critically causes a mode collapse. In addition, the generator and the discriminator optimize the same objective function $V(G,D)$ in opposite directions which is not usual in classical machine learning, and often suffers from oscillations causing excessive training time.

The theoretical convergence proof does not apply in practice because the generator and the discriminator are modeled with deep neural networks, so optimization has to occur in the parameter space rather than in learning the probability density function itself.

3.3 Training techniques to improve GAN training

As demonstrated in Section 2.3.1 and 2.3.2, GAN training is highly unstable and difficult because GAN is required to find a Nash equilibrium of a non-convex minimax game with high dimensional parameters but GAN is typically trained with gradient descent . In this section, we introduce some techniques to improve training of GAN, to make training more stable and produce better results.

Feature matching : This technique substitutes the discriminator’s output in the objective function (Equation 1) with an activation function’s output of an intermediate layer of the discriminator to prevent overfitting from the current discriminator. Feature matching does not aim on the discriminator’s output, rather it guides the generator to see the statistics or features of real training data, in an effort to stabilize training.

Label smoothing : As mentioned previously, $V(G,D)$ is a binary cross entropy loss whose real data label is 1 and its generated data label is 0. However, since a deep neural network classifier tends to output a class probability with extremely high confidence , label smoothing encourages a deep neural network classifier to produce a more soft estimation by assigning label values lower than 1. Importantly, for GAN, label smoothing has to be made for labels of real data, not for labels of fake data, since, if not, the discriminator can act incorrectly .

Spectral normalization : As we see in Section 2.1.2.1 and 2.1.2.2, WGAN and Improved WGAN impose the discriminator to have Lipschitz continuity which constrain the magnitude of function differentiation. Spectral normalization aims to impose a Lipschitz condition for the discriminator in a different manner. Instead of adding a regularizing term or weight clipping, spectral normalization constrains the spectral norm of each layer of the discriminator where the spectral norm is the largest singular value of a given matrix. Since a neural network is a composition of multi layers, spectral normalization normalizes the weight matrices of each layer to make the whole network Lipschitz continuous. In addition, compared to the gradient penalty method proposed in Improved WGAN, spectral normalization is computationally beneficial since gradient penalty regularization directly controls the gradient of the discriminator.

PatchGAN : PatchGAN is not a technique for stabilizing training of GAN. However, PatchGAN greatly helps to generate sharper results in various applications such as image translation . Rather than producing a single output from the discriminator, which is a probability for its input’s authenticity, PatchGAN makes the discriminator produce a grid output. For one element of the discriminator’s output, its receptive field in the input image should be one small local patch in the input image, so the discriminator aims to distinguish each patch in the input image. To achieve this, one can remove the fully connected layer in the last part of the discriminator in the standard GAN. As a matter of fact, PatchGAN is equivalent to adopting multiple discriminators for every patch of the image, making the discriminator help the generator to represent more sharp images locally.

4 Methods to Address Mode Collapse in GAN

Mode collapse which indicates the failure of GAN to represent various types of real samples is the main catastrophic problem of a GAN. From a perspective of the generative model, mode collapse is a critical obstacle for a GAN to be utilized in many applications, since the diversity of generated data needs to be guaranteed to represent the data manifold concretely. Unless multi modes of real data distribution are not represented by the generative model, such a model would be meaningless to use.

In this section, we present several studies that suggest methods to overcome the mode collapse problem. In Section 2.4.1, we demonstrate studies that exploit new objective functions to tackle a mode collapse, and in Section 2.4.2, we introduce studies which propose architecture modifications. Lastly, in Section 2.4.3, we describe mini-batch discrimination which is a notably and practically effective technique for the mode collapse problem.

Unrolled GAN manages mode collapse with a surrogate objective function for the generator, which helps the generator predict the discriminator’s response by unrolling the discriminator update $k$ steps for the current generator update. As we see in the standard GAN , it updates the discriminator first for the fixed generator and then updates the generator for the updated discriminator. Unrolled GAN differs from standard GAN in that it updates the generator based on a $k$ steps updated discriminator given the current generator update, which aims to capture how the discriminator responds to the current generator update. We see that when the generator is updated, it unrolls the discriminator’s update step to consider the discriminator’s $k$ steps future response with respect to the generator’s current update while updating the discriminator in the same manner as the standard GAN. Since the generator is given more information about the discriminator’s response, the generator spreads its probability mass to make it more difficult for the discriminator to react to the generator’s behavior. It can be seen as empowering the generator because only the generator’s update is unrolled, but it seems to be fair in that the discriminator can not be trained to be optimal in practice due to an infeasible computational cost while the generator is theoretically assumed to obtain enough information from the optimal discriminator.

Deep regret analytic GAN (DRAGAN) suggests that a mode collapse occurs due to the existence of a spurious local Nash Equilibrium in the nonconvex problem. DRAGAN addresses this issue by proposing constraining gradients of the discriminator around the real data manifold. It adds a gradient penalizing term which biases the discriminator to have a gradient norm of 1 around the real data manifold. This method attempts to create linear functions by making gradients have a norm of 1. Linear functions near the real data manifolds form a convex function space, which imposes a global unique optimum. Note that this gradient penalty method is also applied to WGAN-GP . They differ in that DRAGAN imposes gradient penalty constraints only to local regions around the real data manifold while Improved WGAN imposes gradient penalty constraints to almost everywhere around the generated data manifold and real data manifold, which leads to higher constraints than those of DRAGAN.

In addition, EBGAN proposes a repelling regularizer loss term to the generator, which encourages feature vectors in a mini-batch to be orthogonalized. This term is utilized with cosine similarities at a representation level of an encoder and forces the generator not to produce samples fallen in a few modes.

4.2 Architecture methods

Multi agent diverse GAN (MAD-GAN) adopts multiple generators for one discriminator to capture the diversity of generated samples as shown in Figure 7a. To induce each generator to move toward different modes, it adopts a cosine similarity value as an additional objective term to make each generator produce dissimilar samples. This technique is inspired from the fact that as images from two different generators become similar, a higher similarity value is produced, thus, by optimizing this objective term, it may make each generator move toward different modes respectively. In addition, because each generator produces different fake samples, the discriminator’s objective function adopts a soft-max cross entropy loss term to distinguish real samples from fake samples generated by multiple generators.

Mode regularized GAN (MRGAN) assumes that mode collapse occurs because the generator is not penalized for missing modes. To address mode collapse, MRGAN adds an encoder which maps the data space $\mathcal{X}$ into the latent space $\mathcal{Z}$. Motivated from the manifold disjoint mentioned in Section 2.3.1, MRGAN first tries to match the generated manifold and real data manifold using an encoder. For manifold matching, the discriminator $D_{M}$ distinguishes real samples $x$ and its reconstruction $G\circ E(x)$, and the generator is trained with $D_{M}(G\circ E(x))$ with a geometric regularizer $d(x,G\circ E(x))$ where $d$ can be any metric in the data space. A geometric regularizer is used to reduce the geometric distance in the data space, to help the generated manifold move to the real data manifold, and allow the generator and an encoder to learn how to reconstruct real samples. For penalizing missing modes, MRGAN adopts another discriminator $D_{D}$ which distinguishes $G(z)$ as fake and $G\circ E(x)$ as real. Since MRGAN matches manifolds in advance with a geometric regularizer, this modes diffusion step can distribute a probability mass even to minor modes of the data space with the help of $G\circ E(x)$. An outline of MRGAN is illustrated in Figure 7b, where $R$ denotes a geometric regularizing term (reconstruction).

4.3 Mini-batch Discrimination

Mini-batch discrimination allows the discriminator to look at multiple examples in a mini-batch to avoid a mode collapse of the generator. The basic idea is that it encourages the discriminator to allow diversity directly in a mini-batch, not considering independent samples in isolation. To make the discriminator deal with not only each example but the correlation between other examples in a mini-batch simultaneously, it models a mini-batch layer in an intermediate layer of the discriminator, which calculates L1-distance based statistics of samples in a mini-batch. By adding such statistics to the discriminator, each example in a mini-batch can be estimated by how far or close to other examples in a mini-batch they are, and this information can be internally utilized by the discriminator, which helps the discriminator reflect samples’ diversity to the output. For the aspect of the generator, it tries to create statistics similar to those of real samples in the discriminator by adversarial learning procedure.

In addition, Progressive GAN proposed a simplified version of mini-batch discrimination, which uses the mean of the standard deviation for each feature (channel) in each spatial location over the mini-batch. This way do not add trainable parameters which projects statistics of the mini-batch while maintaining its effectiveness for a mode collapse. To summarize, mini-batch discrimination reflects samples’ diversity to the discriminator, helping the discriminator determine whether its input batch is real or fake.

Treating the Latent Space

Latent space, also called an embedding space, is the space in which a compressed representation of data lies. If we wish to change or reflect some attributes of an image (for example, a pose, an age, an expression or even an object of an image), modifying images directly in the image space would be highly difficult because the manifolds where the image distributions lie are high-dimensional and complex. Rather, manipulating in the latent space is more tractable because the latent representation expresses specific features of the input image in a compressed manner. In this section, we investigate how GAN handles latent space to represent target attributes and how a variational approach can be combined with the GAN framework.

The input latent vector $z$ of the generator is so highly entangled and unstructured that we do not know which vector point contains the specific representations we want. From this point of view, several papers suggest decomposing the input latent space to an input vector $c$, which contains the meaningful information and standard input latent vector $z$, which can be categorized into a supervised method and an unsupervised method.

Supervised methods require a pair of data and corresponding attributes such as the data’s class label. The attributes are generally used as an additional input vector as explained below.

Conditional GAN (CGAN) imposes a condition of additional information such as a class label to control the data generation process in a supervised manner by adding an information vector $c$ to the generator and discriminator. The generator takes not only a latent vector $z$ but also an additional information vector $c$ and the discriminator takes samples and the information vector $c$ so that it distinguishes fake samples given $c$. By doing so, CGAN can control the number of digits to be generated, which is impossible for standard GAN.

Auxiliary classifier GAN (AC-GAN) takes a somewhat different approach than CGAN. It is trained by minimizing the log-likelihood of class labels with the adversarial loss. The discriminator produces not only the probability that the input samples are from the real dataset but also the probability over the class labels. Figure 8 outlines CGAN and CGAN with a projection discriminator and ACGAN where $CE$ denotes the cross entropy loss for the classification.

In addition, plug and play generative networks (PPGN) are another type of generative model that produce data under a given condition. Unlike the other methods described above, PPGN does not use the labeled attributes while training the generator. Instead, PPGN learns the auxiliary classifier and the generator producing real-like data via adversarial learning independently. Then, PPGN produces the data for the given condition using the classifier and the generator through an MCMC-based sampler. An important characteristic of PPGN is that they can work as plug and play. When a classifier pretrained with the same data but different labels is given to the generator, the generator can synthesize samples under the condition without further training.

1.2 Unsupervised Methods

Different from the supervised methods discussed above, unsupervised methods do not exploit any labeled information. Thus, they require an additional algorithm to disentangle the meaningful features from the latent space.

InfoGAN decomposes an input noise vector into a standard incompressible latent vector $z$ and another latent variable $c$ to capture salient semantic features of real samples. Then, InfoGAN maximizes the amount of mutual information between $c$ and a generated sample $G(z,c)$ to allow $c$ to capture some noticeable features of real data. In other words, the generator takes the concatenated input $(z,c)$ and maximizes the mutual information, $I(c;G(z,c))$ between a given latent code $c$ and the generated samples $G(z,c)$ to learn meaningful feature representations. However, evaluating mutual information $I(c;G(z,c))$ needs to directly estimate the posterior probability $p(c|x)$, which is intractable. InfoGAN, thus, takes a variational approach which replaces a target value $I(c;G(z,c))$ by maximizing a lower bound.

Both CGAN and InfoGAN learn conditional probability $p(x|c)$ given a certain condition vector $c$; however, they are dissimilar regarding how they handle condition vector $c$. In CGAN, additional information $c$ is assumed to be semantically known (such as class labels), so we have to provide $c$ to the generator and the discriminator during the training phase. On the other hand, $c$ is assumed to be unknown in InfoGAN, so we take $c$ by sampling from prior distribution $p(c)$ and control the generating process based on $I(c;G(z,c))$. As a result, the automatically inferred $c$ in InfoGAN has much more freedom to capture certain features of real data than $c$ in CGAN, which is restricted to known information.

Semi-supervised InfoGAN (ss-InfoGAN) takes advantage of both supervised and unsupervised methods. It introduces some label information in a semi-supervised manner by decomposing latent code $c$ in to two parts, $c=c_{ss}\dot{\bigcup}c_{us}$. Similar to InfoGAN, ss-InfoGAN attempts to learn the semantic representations from the unlabeled data by maximizing the mutual information between the generated data and the unsupervised latent code $c_{us}$. In addition, the semi-supervised latent code $c_{ss}$ is trained to contain features that we want by using labeled data. For InfoGAN, we cannot predict which feature will be learned from the training, while ss-InfoGAN uses labeled data to control the learned feature by maximizing two mutual informations; one between $c_{ss}$ and the labeled real data to guide $c_{ss}$ to encode label information $y$, and the other between the generated data and $c_{ss}$. By combining the supervised and unsupervised methods, ss-InfoGAN learns the latent code representation more easily with a small subset of labeled data than the fully unsupervised methods of InfoGAN.

1.3 Examples

Decomposing the latent space into meaningful attributes within the GAN framework has been exploited for various tasks. StackGAN was proposed for text to image generation that synthesizes corresponding images given text descriptions as shown in Figure 9. StackGAN synthesizes images conditioned on text descriptions in a two-stage process: a low-level feature generation (stage 1), and painting details from a given generated image at stage 1 (stage 2). The generators of each stage are trained adversarially given text embedding information $\varphi_{t}$ from the text $t$. Notably, rather than directly concatenating $\varphi_{t}$ to $z$ as CGAN does, StackGAN proposes a conditional augmentation technique which samples conditioned text latent vectors $c$ from the Gaussian distribution $\mathcal{N}(\mu(\varphi_{t}),\sum(\varphi_{t}))$. By sampling the text embedding vector $c$ from $\mathcal{N}(\mu(\varphi_{t}),\sum(\varphi_{t}))$, this technique attempts to augment more training pairs given limited amount of image-text paired data.

Semantically decomposing GAN (SD-GAN) tries to generate a face having different poses by directly decomposing $z$ into identity and pose parts of a face image and then sampling each latent variable separately from the independent latent distributions. Meanwhile, we do not need to restrict the attributes of an image to facial characteristics. Attributes can be not only characteristics of a face but also scenery features such as the weather. Karacan et al. synthesized outdoor images having specific scenery attributes using the CGAN framework, and they also concatenated an attribute latent vector to $z$ for the generator.

2 With an Autoencoder

In this section, we explore efforts combining an autoencoder structure into the GAN framework. An autoencoder structure consists of two parts: an encoder which compresses data $x$ into latent variable $z$: and a decoder, which reconstructs encoded data into the original data $x$. This structure is suitable for stabilizing GAN because it learns the posterior distribution $p(z|x)$ to reconstruct data $x$, which reduces mode collapse caused by the lack of GAN’s inference ability to map data $x$ to $z$. An autoencoder can also help manipulations at the abstract level become possible by learning a latent representation of a complex, high-dimensional data space with an encoder $\mathcal{X}\to\mathcal{Z}$ where $\mathcal{X}$ and $\mathcal{Z}$ denote the data space and the latent space. Learning a latent representation may make it easier to perform complex modifications in the data space through interpolation or conditional concatenation in the latent space. We demonstrate how GAN variants learn in the latent space in Section 3.2.1 and extend our discussion to proposed ideas which combine a VAE framework, another generative model with an autoencoder, with GAN in Section 3.2.2.

Adversarially learned inference (ALI) and bidirectional GAN (BiGAN) learn latent representations within the GAN framework combined with an encoder. As seen in Figure 10a, they learn the joint probability distribution of data $x$ and latent $z$ while GAN learns only the data distribution directly. The discriminator receives samples from the joint space of the data $x$ and the latent variable $z$ and discriminates joint pairs $(G(z),z)$ and $(x,E(x))$ where $G$ and $E$ represent a decoder and an encoder, respectively. By training an encoder and a decoder together, they can learn an inference $\mathcal{X}\to\mathcal{Z}$ while still being able to generate sharp, high-quality samples.

Ulyanov et al. proposed a slightly peculiar method in which adversarial learning is run between the generator $G$ and the encoder $E$ instead of the discriminator. They showed that adversarial learning in the latent space using the generator and the encoder theoretically results in the perfect generator. Upon their theorems, the generator minimizes the divergence between $E(G(z))$ and a prior $z$ in the latent space while the encoder maximizes the divergence. By doing so, they implemented adversarial learning together with tractable inference and disentangled latent space without additional computational cost.

In addition, they added reconstruction loss terms in each space to guarantee each component to be reciprocal. These loss terms can be interpreted as imposing each function to be a one to one mapping function so as not to fall into mode collapse as similar to in MRGAN. Recall that a geometric regularizing term of MRGAN also aims to incorporate a supervised training signal which guides the reconstruction process to the correct location. Figure 10b shows an outline of Ulyanov et al. where $R$ in the red rectangle denotes the reconstruction loss term between the original and reconstructed samples.

2.2 Variational Autoencoder

Variational Autoencoder (VAE) is a popular generative model using an autoencoder framework. Assuming some unobserved latent variable $z$ affects a real sample $x$ in an unknown manner, VAE essentially finds the maximum of the marginal likelihood $p_{\theta}(x)$ for the model parameter $\theta$. VAE addresses the intractability of $p_{\theta}(x)$ by introducing a variational lower bound, learning the mapping of $\mathcal{X}\to\mathcal{Z}$ with an encoder and $\mathcal{Z}\to\mathcal{X}$ with a decoder. Specifically, VAE assumes a prior knowledge $p(z)$ and approximated posterior probability modeled by $Q_{\phi}(z|x)$ to be a standard normal distribution and a normal distribution with diagonal covariance, respectively for the tractability. More explicitly, VAE learns to maximize $p_{\theta}(x)$ where a variational lower bound of the marginal log-likelihood $\log p_{\theta}(x)$ can be derived as follows:

As $KL(Q_{\phi}(z|x)||p_{\theta}(z|X))$ is always nonnegative, a variational lower bound $L(\theta,\phi;x)$ of Equation 42 can be derived as follows:

where $p_{\theta}(x|z)$ is a decoder that generates sample $x$ given the latent $z$ and $Q_{\phi}(z|x)$ is an encoder that generates the latent code $z$ given sample $x$.

Maximizing $L(\theta,\phi;x)$ increases the marginal likelihood $p_{\theta}(x)$. The first term can be interpreted as leading an encoder $Q_{\phi}(z|x)$ to be close to a prior probability $p_{\theta}(z)$. It can be calculated analytically because $Q_{\phi}(z|x)$ and the prior probability are assumed to follow a Gaussian distribution. The second term can be estimated from the sample using a reparameterization method. To sum up, VAE learns by tuning the parameter of the encoder and the decoder to maximize the lower bound $L(\theta,\phi;x)$.

Recently, several approaches to incorporate each advantage of VAE and GAN have been proposed. Although VAE generates blurry images, VAE suffers less from the mode collapse problem because an autoencoder encourages all real samples to be reconstructed. However, GAN generates sharper images than VAE and does not need further constraints on the model while GAN suffers from mode collapse as mentioned in Section 2.4. In this section, we address two studies which attempt to combine VAE and GAN into one framework.

VAEGAN combined VAE with GAN by assigning GAN’s generator to a decoder. Its objective function combined VAE’s objective function with an adversarial loss term to produce sharp images while maintaining encoding ability for the latent space. Notably, it replaced the reconstruction of $x$ in Equation 42 with the intermediate features of the discriminator to capture more perceptual similarity of real samples. Figure 11a shows an outline of VAEGAN where the discriminator $D$ takes one real sample and two fake samples where one is sampled from an encoded latent space ($z_{VAE}$) and the other from a prior distribution ($z$).

Variational approaches for autoencoding GAN ($\alpha$-GAN) proposes adopting discriminators for the variational inference and transforms Equation 41 into a more GAN-like formulation. The most negative aspect of VAE is that we have to constrain a distribution form of $Q_{\phi}(z|x)$ to analytically calculate $L(\theta,\phi;x)$. $\alpha$-GAN treats the variational posteriori distribution implicitly by using the density ratio technique which can be derived as follows:

where $C_{\phi}(z)=\frac{Q_{\phi}(z|x)}{Q_{\phi}(z|x)+p_{\theta}(z)}$ which is an optimal solution of the discriminator for a fixed generator in standard GAN . $\alpha$-GAN estimates the KLD term using a learned discriminator from the encoder $Q_{\phi}(z|x)$ and the prior distribution $p_{\theta}(z)$ in the latent space. A KLD regularization term of a variational distribution, thus, no longer needs to be calculated analytically.

$\alpha$-GAN also modifies a reconstruction term in Equation 42 by adopting two techniques. One is using another discriminator which distinguishes real and synthetic samples, and the other adds a normal $l_{1}$ pixelwise reconstruction loss term in the data space. Specifically, $\alpha$-GAN changes a variational lower bound $L(\theta,\phi;x)$ into a more GAN-like formulation by introducing two discriminators using a density ratio estimation and reconstruction loss to prevent a mode collapse problem. Figure 11b shows an outline of $\alpha$-GAN where $D_{L}$ is the discriminator of the latent space, and $D_{D}$ represents the discriminator which acts on the data space and $R$ is the reconstruction loss term.

2.3 Examples

Antipov et al. proposed an encoder-decoder combined method for the face aging of a person. It produces facial aging of the target image under the given age vector $y$ while maintaining the identity of the target image. The encoder takes a role to output a latent vector $z$ which represents a personal identity to be preserved. However, practically, this CGAN-based approach has a problem in that the generator tends to ignore the latent variable $z$, concerning only conditional information $y$.

There are some approaches to training $z$ in an unsupervised manner with an auto encoder. Mainly, semi latent GAN (SL-GAN) proposed a method for changing a facial image from high-level semantic facial attributes (i.e., male/female, skin/hair color), by decomposing the latent space from an encoder into annotated attributes (ground-truth attribute of an image) and data-driven attributes (as $c$ of InfoGAN). Similarly to InfoGAN, SL-GAN tries to maximize the mutual information between the data-driven attribute and the generated image while using unsupervised training for the data-driven attributes.

In addition to that, disentangled representation GAN (DR-GAN) addresses pose-invariant face recognition, which is a difficult problem because of the drastic changes of an image for each different pose, adopting an encoder-decoder structure for the generator. As the purpose of DR-GAN is to generate a face of the same identity given a target pose, it has to learn the identity feature to be invariant regardless of a facial pose. To achieve this, DR-GAN designs an encoder of the generator to represent an identity feature, while a decoder of the generator produces an image under the pose representing vectors and the encoded identity.

Applications Using GANs

As discussed in earlier sections, GAN is a very powerful generative model in that it can generate real-like samples with an arbitrary latent vector $z$. We do not need to know an explicit real data distribution nor assume further mathematical conditions. These advantages lead GAN to be applied in various academic and engineering fields. In this section, we discuss applications of GANs in several domains.

Image translation involves translating images in one domain $X$ to images in another domain $Y$. Mainly, translated images have the dominant characteristic of domain $Y$ maintaining their attributes in the original images. Image translation can be categorized into supervised and unsupervised techniques such as the classical machine learning.

Image translation with paired images can be regarded as supervised image translation in that an input image ${x}$ $\in$ $X$ to be translated always has the target image ${y}$ $\in$ $Y$ where $X$ and $Y$ are two distinctive domains. Pix2pix suggests an image translation method with paired images using a CGAN framework in which a generator produces a corresponding target image conditioned on an input image as seen in Figure 12. In contrast, Perceptual Adversarial Networks (PAN) add the perceptual loss between a paired data $(x,y)$ to the generative adversarial loss to transform input image $x$ into ground-truth image $y$. Instead of using the pixelwise loss to push the generated image toward the target image, it uses hidden layer discrepancies of the discriminator between an input image $x$ and ground truth image $y$. It tries to transform $x$ to $y$ to be perceptually similar by minimizing perceptual information discrepancies from the discriminator.

Image translation in an unsupervised manner learns a mapping between two domains given unpaired data from two domains. CycleGAN and discover cross-domain relations with GAN (DiscoGAN) aim to conduct unpaired image-to-image translations using a cyclic consistent loss term in addition to an adversarial loss term. With a sole translator ${G}$: ${X}\to{Y}$, GAN may learn meaningless translation or mode collapse, resulting in an undesired translation. To reduce the space of mapping of the generator, they adopt another inverse translator ${T}$: ${Y}\to{X}$ and introduce the cyclic consistency loss which encourages ${T}({G}({x}))\approx{x}$ and ${G}({T}({y}))\approx{y}$ so that each translation finds a plausible mapping between the two domains as mentioned in Section 2.1.3.1. Their methods can be interpreted in a similar manner described in Section 3.2.1 in that they add a supervised signal for reconstruction.

Attribute guided image translation was also considered to transfer the visual characteristic of an image. Conditional CycleGAN utilizes CGAN with a cyclic consistency framework. Kim et al. attempted to transfer visual attributes. In addition to the cyclic consistency of an image, they also added an attribute consistency loss which forces the transferred image to have a target attribute of the reference image.

1.2 Super resolution

Acquiring super resolution images from low resolution images has the fundamental problem that the recovered high-resolution image misses high-level texture details during the upscaling of the image. Ledig et al. adopted a perceptual similarity loss in addition to an adversarial loss, instead of pixelwise mean-squared error loss. It focuses on feature differences from the intermediate layer of the discriminator, not pixelwise because optimizing pixel-wise mean squared error induces the pixelwise average of a plausible solution, leading to perceptually poor smoothed details and it is not robust to drastic pixel value changes.

1.3 Object detection

Detecting small objects in an image typically suffers from low-resolution of an object, and thus, it is necessary to train models with images of various scales similar to You Look Only Once (YOLO) and Single Shot Detection (SSD) methods. Notably, Li et al. tries to transform a small object with low resolution into a super resolved large object to make the object more discriminative. They utilized a GAN framework except decomposed the discriminator into two branches, namely, an adversarial branch and a perceptual branch. The generator produces a real-like large-scale object by the typical adversarial branch while the perceptual branch guarantees that the generated large-scale object is useful for the detection.

Ehsani et al. proposed another framework to detect objects occluded by other objects in an image. It uses a segmentor, a generator, and a discriminator to extract the entire occluded-object mask and to paint it as a real-like image. The segmentor takes an image and a visible region mask of an occluded object and produces a mask of the entire occluded object. The generator and the discriminator are trained adversarially to produce an object image in which the invisible regions of the object are reconstructed.

1.4 Object transfiguration

Object transfiguration is a conditional image generation that replaces an object in an image with a particular condition while the background does not change. Zhou et al. adopted an encoder-decoder structure to transplant an object, where the encoder decomposes an image into the background feature and the object feature, and the decoder reconstructs the image from the background feature and the object feature we want to transfigure. Importantly, to disentangle the encoded feature space, two separated training sets are required where one is the set of images having the object and the other is the set of images not having the object.

In addition, the GAN can be applied to an image blending task which implants an object into another image’s background and makes the composited copy-paste images look more realistic. Gaussian-Poisson GAN (GP-GAN) suggests a high-resolution image blending framework using GAN and a classic image blending gradient-based approach . It decomposes images into low-resolution but well-blended images using a GAN and detailed textures and edges using a gradient constraint. Then, GP-GAN attempts to combine the information by optimizing a Gaussian-Poisson equation to generate high-resolution well-blended images while maintaining captured high-resolution details.

1.5 Joint image generation

The GAN can be utilized to generate multiple domain images at once. Coupled GAN suggests a method of generating multidomain images jointly by weight sharing techniques among GAN pairs. It first adopts GAN pairs to match the number of domains we want to produce. Then, it shares the weights of some layers of each GAN pair that represents high-level semantics. Therefore, it attempts to learn joint distributions of a multidomain from samples drawn from a marginal domain distribution. It should be noted that because it aims to generate multidomain images which share high-level abstract representations, images from each domain have to be very similar in a broad view.

1.6 Video generation

In this paragraph, we discuss GANs generating video. Generally, the video is composed of relatively stationary background scenery and dynamic object motions. Video GAN (VGAN) considers a two-stream generator. A moving foreground generator using 3D CNN predicts plausible future frames while a static background generator using 2D CNN makes the background stationary. Pose-GAN takes a VAE and GAN combining approach. It uses a VAE approach to estimate future object movements conditioned on a current object pose and hidden representations of past poses. With a rendered future pose video and clip image, it uses a GAN framework to generate future frames using a 3D CNN. Recently, motion and content GAN (MoCoGAN) proposed to decompose the content part and motion part of the latent space, especially modeling the motion part with RNN to capture the time dependency.

2 Sequential Data Generation

GAN variants that generate discrete values mostly borrow a policy gradient algorithm of RL, to circumvent direct back-propagation of discrete values. To output discrete values, the generator, as a function, needs to map the latent variable into the domain where elements are not continuous. However, if we do the back-propagation as another continuous value generating process, the generator is steadily guided to generate real-like data by the discriminator, rather than suddenly jumping to the target discrete values. Thus, such a slight change of the generator cannot easily look for a limited real discrete data domain .

In addition, when generating a sequence such as music or language, we need to evaluate a partially generated sequence step-by-step, measuring the performance of the generator. However, the conventional GAN framework can only evaluate whole generated sequences unless there is a discriminator for each time-step. This, too, can be solved by the policy gradient algorithm, in that RL naturally addresses the sequential decision process of the agent.

When we want to generate music, we need to generate the note and a tone of the music step-by-step, and these elements are not continuous values. A simple and direct approach is continuous RNN-GAN (C-RNN-GAN) , where it models both the generator and discriminator as an RNN with long-short term memory (LSTM) , directly extracting whole sequences of music. However, as mentioned above, we can only evaluate whole sequences, and not a partially generated sequence. Furthermore, its results are not highly satisfactory since it does not consider the discrete property of the music elements.

In contrast, sequence GAN (SeqGAN) , object reinforced GAN (ORGAN) , and Lee et al. employed a policy gradient algorithm, and not generating whole sequences at once. The result of SeqGAN is shown in Figure 13. They treat a generator’s output as a policy of an agent and take the discriminator’s output as a reward. Selecting a reward with the discriminator is a natural choice as the generator acts to obtain a large output (reward) from the discriminator, similar to the agent learning to acquire a large reward in reinforcement learning. In addition, ORGAN is slightly different from SeqGAN, adding a hard-coded objective to the reward function to achieve the specified goal.

2.2 Language and speech

RankGAN suggests language (sentence) generation methods and a ranker instead of a conventional discriminator. In natural language processing, the expression power of natural language needs to be considered in addition to its authenticity. Thus, RankGAN adopts a relative ranking concept between generated sentences and reference sentences which are human-written. The generator tries its generated language sample to be ranked high, while the ranker evaluates the rank score of the human-written sentences higher than the machine-written sentences. As the generator outputs discrete symbols, it similarly adopts a policy gradient algorithm similar to SeqGAN and ORGAN. In RankGAN, the generator can be interpreted as a policy predicting next step symbol and the rank score can be thought of as a value function given a past generated sequence.

Variational autoencoding Wasserstein GAN (VAW-GAN) is a voice conversion system combining GAN and VAE frameworks. The encoder infers a phonetic content $z$ of the source voice, and the decoder synthesizes the converted target voice given a target speaker’s information $y$, similar to conditional VAE . As discussed in Section 3.2.2, VAE suffers from generating sharp results due to the oversimplified assumption of the Gaussian distribution. To address this issue, VAW-GAN incorporates WGAN similarly to VAEGAN . By assigning the decoder to the generator, it aims to reconstruct the target voice given the speaker representation.

3 Semi-Supervised Learning

Semi-supervised learning is a learning method that improves the classification performance by using unlabeled data in a situation where there are both labeled and unlabeled data. In the big data era, there exists a common situation where the size of the data is too large to label all the data, or the cost of labeling is expensive. Thus, it is often necessary to train the model with a dataset in which only a small portion of the total data has labels.

The GAN-based semi-supervised learning method demonstrates how unlabeled and generated data are available on the GAN framework. The generated data is allocated to a $K+1$ class beyond $1\dots K$ class for the labeled data. For labeled real data, the discriminator classifies their correct label ($1$ to $K$). For unlabeled real data and generated data, they are trained with a GAN minimax game. Their training objective can be expressed as follows:

where $L_{s}$ and $L_{us}$ stand for the loss functions of the labeled data and the unlabeled data, respectively. It is noted that because only generated data is classified as the $K+1$ class, we could think of $L_{us}$ as a GAN standard minimax game. The unlabeled data and the generated data serve to inform the model of the space where the real data resides. In other words, the unsupervised cost serves to guide the location of the optimum of the supervised cost of the real labeled data.

Categorical GAN (CatGAN) proposes an algorithm for the robust classification for which the generator regularizes the classifier. The discriminator has no classification head for distinguishing real and fake and is trained with three requirements: a small conditional entropy of $H(y|x)$ to make the correct class assignment for the real data, a large conditional entropy of $H(y|G(z))$ to make the class assignment for the generated data diverse and a large entropy of $H(y)$ to make a uniform marginal distribution with an assumption of a uniform prior $p(y)$ over classes where $x$, $y$ and $G(z)$ are real data, labels, and generated data respectively. The generator, meanwhile, is trained with two requirements: a small conditional entropy of $H(y|G(z))$ to make the class assignment for the generated data certain and a large entropy of $H(y)$ to generate equally distributed samples over classes. The unlabeled data and the generated data help the classification by balancing classes through the adversarial act of the generator, so it helps semi-supervised learning.

3.2 Semi-supervised learning with an auxiliary classifier

The above GAN variants in semi-supervised learning have two problems: the first one is that the discriminator has two incompatible convergence points, one for discriminating real and fake data and the other for predicting the class label; and the second problem is that the generator cannot generate data in a specific class. Triple-GAN addresses the two problems by a three-player formulation: a generator $G$, a discriminator $D$ and a classifier $C$. The model is illustrated in Figure 14 where $(X_{g},Y_{g})\sim p_{g}(X,Y)$, $(X_{l},Y_{l})\sim p(X,Y)$, and $(X_{c},Y_{c})\sim p_{c}(X,Y)$ refer to the generated data, the labeled data, and the unlabeled data with a predicted label, and $CE$ is the cross entropy loss. To summarize, Triple-GAN adopts an auxiliary classifier which classifies real labeled data and label-conditioned generated data, relieving the discriminator of classifying the labeled data. In addition, Triple-GAN generates data conditioned on $Y_{g}$, which means that it can generate label-specific data.

4 Domain Adaptation

The main difficulty in domain adaptation is the difference between the source distribution and the target distribution, called domain shift. This domain shift allows the classifier trained with only the source data to fail in the target domain. One of the methods to address the domain shift is to project each domain data into the common feature space where the distributions of the projected data are similar. There have been a few studies to achieve the common feature space via GAN for the domain adaptation task.

Domain adversarial neural network (DANN) first used GAN to obtain domain invariant features by making them indistinguishable as to whether it comes from the source domain or the target domain while still discriminative for the classifying task. There are two components sharing a feature extractor. One is a classifier which classifies the labels of the data. The other is a domain discriminator which discern where the data comes from. The feature generator acts like the generator in GAN producing source-like features from the target domain. To sum up, DANN takes CNN classification networks in addition to the GAN framework to classify data from the target domain without label information by learning the domain invariant features. Figure 15 shows the overall outline of DANN, ARDA and unsupervised pixel-level domain adaptation where $I_{S},I_{T}$, and $I_{f}$ stands for the source domain image, target domain image, and fake image, respectively. $F_{S}$ and $F_{T}$ means the extracted source features, and the target features, respectively. It should be noted that $Y_{s}$, which is the label information of $I_{s}$, is fed into the classifier, training the classifier with the cross entropy loss.

Although DANN may achieve domain invariant marginal feature distributions, if the label of the source feature and that of the target feature do not contain a mismatch, the learned classifier should not work well in the target domain. Cycle-consistent adversarial domain adaptation (CyCADA) , thus, adds a cycle consistency to preserve the content of the data which is the crucial characteristic in determining the label while inheriting the most of the architecture of DANN.

4.2 Examples

Bousmalis et al. used the domain adaptation via GAN for the grasping task. They used simulation data as the source domain data and ran the learned model in the real environment. Their method is slightly different than DANN in that it only adapts source images to be seen as if they were drawn from the target domain while DANN tries to obtain features similarly from both domains. It can be understood that the feature space of is the target domain space and the feature generator in the target domain is the identity function. In addition, they added a content similarity loss defined as the pixelwise mean-squared error between the source image and the adapted image to preserve the content of the source image. By doing so, they achieved better performance than the supervised method in the grasping task. It should also be noted that the method of can check whether the domain adaptation process is working well during the training phase because the transformed data is visible, while cannot visually check the domain adaptation process because the common representation space cannot be easily visualized.

Yoo et al. achieved autonomous navigation without any real labeled data using domain adaptation. As in , they also exploited the simulation data as the source data and showed autonomous navigation in a real outdoor environment. They used cycle consistency to preserve the content and style loss term motivated by the style transfer task to reduce the domain shift dramatically for the outdoor environment as seen in Figure 16. By doing so, they showed the applicability of the simulation via domain adaptation into the autonomous navigation tasks where collecting labels for various environments is difficult and expensive.

5 Other Tasks

Several variants of GAN have also been developed in other academic or practical fields other than the machine learning fields.

Xue et al. proposed a segmentor-critic structure to segment a medical image. A segmentor generates a predicted segmented image, and a critic maximizes the hierarchical feature differences between the ground-truth and the generated segmentation. This structure leads a segmentor to learn the features of the ground-truth segmentation adversarially similar to the GAN approach. There are also other medical image segmentation algorithms such as the deep image-to-image network (DI2IN) and structure correcting adversarial network (SCAN) . DI2IN conducts liver segmentation of 3D CT images through adversarial learning. SCAN tries to segment the lung and the heart from chest X-ray images through an adversarial approach with a ground-truth segmentation mask.

5.2 Steganography

It is also feasible to use GAN for steganography. Steganography is a technique that conceals secret messages in non-secret containers such as an image. A steganalyzer determines if a container contains a secret message or not. Some studies such as Volkhonskiy et al. , and Shi et al. propose a steganography model with three components: a generator producing real-looking images that are used as containers and two discriminators, one of which classifies whether an image is real or fake, and the other determines whether an image contains a secret message.

5.3 Continual Learning

Deep generative replay extends a GAN framework to continual learning. Continual learning solves multiple tasks and accumulates new knowledge continually. Continual learning in deep neural networks suffers from catastrophic forgetting which refers to forgetting a previously learned task while learning a new task. Inspired by the brain mechanism, catastrophic forgetting is addressed with a GAN framework called deep generative replay. Deep generative replay trains a scholar model in each task where a scholar model is composed of a generator. The generator produces samples of an old task, and the solver gives a target answer to an old task’s sample. By sequentially training scholar models with old task values generated by old scholars, it attempts to overcome catastrophic forgetting while learning new tasks.

Discussion

We have discussed how GAN and its variants work and how they are applied to various applications. Table 3 compares some famous variants of GAN with respect to model architectures and additional constraints. As we have viewed GAN from a microscopic perspective until now, we are going to discuss the macroscopic view of GAN in this section.

Measuring the performance of GAN is related to capturing the diversity and quality of generated data. As explained in Section 1, the generative models mostly model the likelihood and learn by maximizing it. Thus, it is natural to evaluate the generator using log likelihood. However, GAN generates real-like data without estimating likelihood directly, so evaluation with the likelihood is not quite proper. The most widely accepted evaluation metric for GAN is an Inception score. The Inception score was proposed by Salimans et al. , and the Inception score is defined as follows:

As seen in Equation 47, it computes the average KLD between the conditional label distribution $p(y|x)$ and the marginal distribution of the generated data’s label $p(y)$ with a pretrained classifier such as VGG and ImageNet data . Since the KLD term between $p(y|x)$ and $p(y)$ in the exponent function is equivalent to their mutual information $I(y;x)=H(y)-H(y|x)$ , the high entropy of $p(y|x)$ and low entropy of $p(y)$ leads to a high Inception score. The entropy of $p(y|x)$ measures how the generated data are sharp and clear to be well-classified. The other term $H(y)$ represents whether the generated data are diverse with respect to the generated class. In this way, the Inception score is believed to measure the diversity and visual quality of the generated data.

The method for measuring the performance of GAN is still a disputable subject. Since GAN is naturally unsupervised learning, we cannot measure the accuracy or error rate as in the supervised learning approaches. Evaluation metrics or different distances discussed in the above paragraph still do not measures the performance of GAN exactly, and there are many cases in which images that do not look natural have a high score . Thus, there is room to improve the evaluation for GAN.

2 Discrete Structured Data

Unlike other generative models such as VAE, GAN has an issue handling the discrete data such as text sequences or discretized images. Since discrete data is non-differentiable, gradient descent update via back-propagation cannot directly be applied for a discrete output. The content of this section may overlap with Section 4.2, but we will shortly cover this issues in the discussion because it is one of the troublesome issue in GAN.

To address this issue, some methods adopt a policy gradient algorithm in reinforcement learning (RL) in which the objective is to maximize the total rewards. By rolling out whole sequences, this method circumvents direct back-propagation for a discrete output. Intuitively, the generator which generates fake data maximizing the discriminator’s output can be thought of as a policy agent in RL which is a probability distribution to take action maximizing a reward in a given state.

3 Relationship to Reinforcement Learning

Reinforcement learning (RL) is a type of learning theory that focuses on teaching an agent to choose the best action given a current state. A policy $\pi(a|s)$, which is a probability for choosing an action $a$ at state $s$, is learned via an on-policy or off-policy algorithm. RL has a very similar concept to GAN in the aspect of a policy gradient where it is very important to estimate the value function correctly given state $s$ and action $a$.

Inverse reinforcement learning (IRL) is similar to reinforcement learning in that its objective is to find the optimal policy. However, in the IRL framework, the experts’ demonstrations are provided instead of a reward. It finds the appropriate reward function that makes the given demonstration as optimal as possible and then produces the optimal policy for the identified reward function. There are many variants in IRL. Maximal entropy IRL is one which finds the policy distribution that satisfies the constraints so that the feature expectations of the policy distribution and the given demonstration are the same. To solve such an ill-posed problem, maximal entropy IRL finds the policy distribution with the largest entropy according to the maximal entropy principle . Intuitively, the maximal entropy IRL finds the policy distribution which maximizes the likelihood of a demonstration and its entropy. Its constraint and convexity induce the dual minimax problem. The dual variable can be seen as a reward. The minimax formulation and the fact that it finds the policy which has the largest likelihood of demonstrations gives it a deep connection with the GAN framework. The primal variable is a policy distribution in IRL whereas it can be considered as a data distribution from the generator in GAN. The dual variable is a reward/cost in IRL while it can be seen as the discriminator in GAN.

Finn et al. , Yoo et al. , and Ho and Ermon showed a mathematical connection between IRL and GAN. Ho and Ermon converted IRL to the original GAN by constraining the space of dual variables and Yoo et al. showed the relationship between EBGAN and IRL using approximate inference.

4 Pros and Cons of GAN

As briefly mentioned in Section 1, the major advantage of GAN is that GAN does not need to define the shape of the probability distribution of the generator model. Thus, GAN naturally avoids concerning tractable density forms which need to represent complex and high-dimensional distributions. Compared to other models using explicitly defined probability density , GAN has following advantages:

where $x$ is the d-dimensional vector. For example, in image generation, autoregressive models generate an image pixel by pixel where the probability distribution of future pixel cannot be inherently computed until the value of the previous pixel is computed. Thus, the generation process is naturally slow, which becomes more severe for high-dimensional data generation such as speech synthesis .

On the other hand, the generator of GAN is a simple feed-forward network mapping from $\mathcal{Z}$ to $\mathcal{X}$. The generator produces data all at once, not pixel by pixel as autoregressive models. Therefore, GAN can generate samples in parallel, which results in a considerable speed up for sampling, and this property gives more opportunity for GAN to be used in various real applications.

GAN does not need to approximate a likelihood by introducing a lower bound, as in VAE. As we mentioned in Section 3.2.2, VAE tries to maximize a likelihood by introducing a variational lower bound. The strategy of VAE is to maximize a tractable variational lower bound, guaranteeing it to be at least as high as the lower bound, even when the likelihood is intractable. However, VAE still needs assumptions on a prior and posterior distributions, which do not guarantee the tight bound of Equation 42. This strong assumption on distributions makes the approximation to the maximum likelihood biased.

In contrast, GAN does not approximate the likelihood and does not need any probability distribution assumptions. Instead, GAN is designed to solve an adversarial game between the generator and the discriminator, and a Nash equilibrium of the GAN game corresponds to finding the real data distribution .

GAN is highly capable of capturing the high-frequency parts of an image. Since the generator tries to fool the discriminator to recover the real data distribution, the generator evolves to lead even the high-frequency parts to deceive the discriminator. In addition, some techniques such as PatchGAN in Section 2.3.3 helps GAN produce and capture sharper results more effectively.

4.2 Cons

GAN was developed to solve the minimax game between the generator and the discriminator. Though several studies discuss the convergence and the existence of the Nash equilibrium of the GAN game, training of GAN is highly unstable and difficult to converge as mentioned in Section 2.3.1 and 2.3.2. GAN solves the minimax game through the gradient descent method iteratively for the generator and the discriminator. In perspective of the cost function: $V(G,D)$ in Equation 1, a solution for the GAN game is the Nash equilibrium which is a point of parameters where the discriminator’s cost and the generator’s cost is minimum with respect to their parameters. However, the decrease of the discriminator’s cost function can cause the increase of the generator’s cost function and vice versa. Thus, a convergence of the GAN game may often fail and is prone to be unstable.

Another important issue for GAN is the mode collapse problem. This problem is very detrimental for GAN that is applied in real applications since a mode collapse restricts GAN’s ability of diversity. In Equation 1, the generator is only forced to deceive the discriminator, not for representing multimodality of a real data distribution. A mode collapse thus can happen even in a simple experiment , and this discourages applying GAN due to the low diversity. As mentioned in Section 2.4, various studies tried to address the mode collapse by using a new object function , or adding new components . However, for a highly complex and multimodal real data distribution, the mode collapse still remains a problem GAN has to solve.

4.3 Future research areas

As GAN has been popular throughout deep learning, the limitations of GAN mentioned above have recently been improved . With the development of GAN, new tasks are steadily conquered using GAN. For instance, CausalGAN combines a causal implicit generative model with the conditional GAN to replace conditioning by intervention, which enables the model to generate data with the desired characteristic combinations that do not exist in the dataset. In addition, new types of GANs have been proposed for new applications such as cipher cracking and object tracking . When you design a GAN for a new task, you can first identify the nature of the task and then use Tables LABEL:tab:overview1, LABEL:tab:overview_app, and 3 to determine which model to use as a baseline. A new loss function can be designed using the characteristics of the task. In the future, we anticipate that the limitations of existing GANs will be solved in novel ways, and GAN will remain as an important generative model by conquering areas that existing deep learning-based models cannot effectively solve.

Conclusion

We discussed how various object functions and architectures affect the behavior of GAN and the applications of GAN such as image translation, image attribute editing, domain adaptation, and other fields. The GAN originated from the theoretical minimax game perspective. In addition to the standard GAN , practical trials as well as mathematical approaches have been adopted, resulting in many variants of GAN. Furthermore, the relationship between GAN and other concepts such as imitation learning, and other generative models has been discussed and combined in various studies, resulting in rich theory and numerous application techniques. GAN has the potential to be applied in many application domains including those we have discussed. Despite GAN’s significant success, there remain unsolved problems in the theoretical aspects as to whether GAN actually converges and whether it can perfectly overcome mode collapse, as Arora et al. , Grnarova et al. , and Mescheder et al. discussed. However, with the power of deep neural networks and with the utility of learning a highly non-linear mapping from latent space into data space, there remain enormous opportunities to develop GAN further and to apply GAN to various applications and fields.

References