Toward Multimodal Image-to-Image Translation

Introduction

Deep learning techniques have made rapid progress in conditional image generation. For example, networks have been used to inpaint missing image regions , add color to grayscale images , and generate photorealistic images from sketches . However, most techniques in this space have focused on generating a single result. In this work, we model a distribution of potential results, as many of these problems may be multimodal in nature. For example, as seen in Figure 1, an image captured at night may look very different in the day, depending on cloud patterns and lighting conditions. We pursue two main goals: producing results which are (1) perceptually realistic and (2) diverse, all while remaining faithful to the input.

Mapping from a high-dimensional input to a high-dimensional output distribution is challenging. A common approach to representing multimodality is learning a low-dimensional latent code, which should represent aspects of the possible outputs not contained in the input image. At inference time, a deterministic generator uses the input image, along with stochastically sampled latent codes, to produce randomly sampled outputs. A common problem in existing methods is mode collapse , where only a small number of real samples get represented in the output. We systematically study a family of solutions to this problem.

We start with the pix2pix framework , which has previously been shown to produce high-quality results for various image-to-image translation tasks. The method trains a generator network, conditioned on the input image, with two losses: (1) a regression loss to produce similar output to the known paired ground truth image and (2) a learned discriminator loss to encourage realism. The authors note that trivially appending a randomly drawn latent code did not produce diverse results. Instead, we propose encouraging a bijection between the output and latent space. We not only perform the direct task of mapping the latent code (along with the input) to the output but also jointly learn an encoder from the output back to the latent space. This discourages two different latent codes from generating the same output (non-injective mapping). During training, the learned encoder attempts to pass enough information to the generator to resolve any ambiguities regarding the output mode. For example, when generating a day image from a night image, the latent vector may encode information about the sky color, lighting effects on the ground, and cloud patterns. Composing the encoder and generator sequentially should result in the same image being recovered. The opposite should produce the same latent code.

In this work, we instantiate this idea by exploring several objective functions, inspired by literature in unconditional generative modeling:

cVAE-GAN (Conditional Variational Autoencoder GAN): One approach is first encoding the ground truth image into the latent space, giving the generator a noisy “peek" into the desired output. Using this, along with the input image, the generator should be able to reconstruct the specific output image. To ensure that random sampling can be used during inference time, the latent distribution is regularized using KL-divergence to be close to a standard normal distribution. This approach has been popularized in the unconditional setting by VAEs and VAE-GANs .

cLR-GAN (Conditional Latent Regressor GAN): Another approach is to first provide a randomly drawn latent vector to the generator. In this case, the produced output may not necessarily look like the ground truth image, but it should look realistic. An encoder then attempts to recover the latent vector from the output image. This method could be seen as a conditional formulation of the “latent regressor" model and also related to InfoGAN .

BicycleGAN: Finally, we combine both these approaches to enforce the connection between latent encoding and output in both directions jointly and achieve improved performance. We show that our method can produce both diverse and visually appealing results across a wide range of image-to-image translation problems, significantly more diverse than other baselines, including naively adding noise in the pix2pix framework. In addition to the loss function, we study the performance with respect to several encoder networks, as well as different ways of injecting the latent code into the generator network.

We perform a systematic evaluation of these variants by using humans to judge photorealism and a perceptual distance metric to assess output diversity. Code and data are available at https://github.com/junyanz/BicycleGAN.

Related Work

Parametric modeling of the natural image distribution is a challenging problem. Classically, this problem has been tackled using restricted Boltzmann machines and autoencoders . Variational autoencoders provide an effective approach for modeling stochasticity within the network by reparametrization of a latent distribution at training time. A different approach is autoregressive models , which are effective at modeling natural image statistics but are slow at inference time due to their sequential predictive nature. Generative adversarial networks overcome this issue by mapping random values from an easy-to-sample distribution (e.g., a low-dimensional Gaussian) to output images in a single feedforward pass of a network. During training, the samples are judged using a discriminator network, which distinguishes between samples from the target distribution and the generator network. GANs have recently been very successful . Our method builds on the conditional version of VAE and InfoGAN or latent regressor models by jointly optimizing their objectives. We revisit this connection in Section 3.4.

Conditional image generation

All of the methods defined above can be easily conditioned. While conditional VAEs and autoregressive models have shown promise , image-to-image conditional GANs have lead to a substantial boost in the quality of the results. However, the quality has been attained at the expense of multimodality, as the generator learns to largely ignore the random noise vector when conditioned on a relevant context . In fact, it has even been shown that ignoring the noise leads to more stable training .

Explicitly-encoded multimodality

One way to express multiple modes is to explicitly encode them, and provide them as an additional input in addition to the input image. For example, color and shape scribbles and other interfaces were used as conditioning in iGAN , pix2pix , Scribbler and interactive colorization . An effective option explored by concurrent work is to use a mixture of models. Though able to produce multiple discrete answers, these methods are unable to produce continuous changes. While there has been some degree of success for generating multimodal outputs in unconditional and text-conditional setups , conditional image-to-image generation is still far from achieving the same results, unless explicitly encoded as discussed above. In this work, we learn conditional image generation models for modeling multiple modes of output by enforcing tight connections between the latent and image spaces.

Multimodal Image-to-Image Translation

Our goal is to learn a multi-modal mapping between two image domains, for example, edges and photographs, or night and day images, etc. Consider the input domain $\mathcal{A}\!\subset\!\mathds{R}^{H\!\times W\!\times 3}$, which is to be mapped to an output domain $\mathcal{B}\!\subset\!\mathds{R}^{H\!\times W\!\times 3}$. During training, we are given a dataset of paired instances from these domains, \big{\{}(\mathbf{A}\!\in\!\mathcal{A},\mathbf{B}\!\in\!\mathcal{B})\big{\}}, which is representative of a joint distribution $p(\mathbf{A},\mathbf{B})$. It is important to note that there could be multiple plausible paired instances $\mathbf{B}$ that would correspond to an input instance $\mathbf{A}$, but the training dataset usually contains only one such pair. However, given a new instance $\mathbf{A}$ during test time, our model should be able to generate a diverse set of output $\widehat{\mathbf{B}}$’s, corresponding to different modes in the distribution $p(\mathbf{B}|\mathbf{A})$.

While conditional GANs have achieved success in image-to-image translation tasks , they are primarily limited to generating a deterministic output $\widehat{\mathbf{B}}$ given the input image $\mathbf{A}$. On the other hand, we would like to learn the mapping that could sample the output $\widehat{\mathbf{B}}$ from true conditional distribution given $\mathbf{A}$, and produce results which are both diverse and realistic. To do so, we learn a low-dimensional latent space $\mathbf{z}\in\mathds{R}^{Z}$, which encapsulates the ambiguous aspects of the output mode which are not present in the input image. For example, a sketch of a shoe could map to a variety of colors and textures, which could get compressed in this latent code. We then learn a deterministic mapping $G:(\mathbf{A},\mathbf{z})\rightarrow\mathbf{B}$ to the output. To enable stochastic sampling, we desire the latent code vector $\mathbf{z}$ to be drawn from some prior distribution $p(\mathbf{z})$; we use a standard Gaussian distribution $\mathcal{N}(0,I)$ in this work.

We first discuss a simple extension of existing methods and discuss its strengths and weakness, motivating the development of our proposed approach in the subsequent subsections.

The recently proposed pix2pix model has shown high quality results in the image-to-image translation setting. It uses conditional adversarial networks to help produce perceptually realistic results. GANs train a generator $G$ and discriminator $D$ by formulating their objective as an adversarial game. The discriminator attempts to differentiate between real images from the dataset and fake samples produced by the generator. Randomly drawn noise $\mathbf{z}$ is added to attempt to induce stochasticity. We illustrate the formulation in Figure 2(b) and describe it below.

In this scenario, there is little incentive for the generator to make use of the noise vector which encodes random information. Isola et al. note that the noise was ignored by the generator in preliminary experiments and was removed from the final experiments. This was consistent with observations made in the conditional settings by , as well as the mode collapse phenomenon observed in unconditional cases . In this paper, we explore different ways to explicitly enforce the latent coding to capture relevant information.

One way to force the latent code $\mathbf{z}$ to be “useful" is to directly map the ground truth $\mathbf{B}$ to it using an encoding function $E$. The generator $G$ then uses both the latent code and the input image $\mathbf{A}$ to synthesize the desired output $\widehat{\mathbf{B}}$. The overall model can be easily understood as the reconstruction of $\mathbf{B}$, with latent encoding $\mathbf{z}$ concatenated with the paired $\mathbf{A}$ in the middle – similar to an autoencoder . This interpretation is better shown in Figure 2(c).

This approach has been successfully investigated in Variational Autoencoder in the unconditional scenario without the adversarial objective. Extending it to conditional scenario, the distribution $Q(\mathbf{z}|\mathbf{B})$ of latent code $\mathbf{z}$ using the encoder $E$ with a Gaussian assumption, $Q(\mathbf{z}|\mathbf{B})\triangleq E(\mathbf{B})$. To reflect this, Equation 1 is modified to sampling $\mathbf{z}\sim E(\mathbf{B})$ using the re-parameterization trick, allowing direct back-propagation .

where $\mathcal{D}_{\text{KL}}(p||q)=-\int p(z)\log\frac{p(z)}{q(z)}dz$. This forms our cVAE-GAN objective, a conditional version of the VAE-GAN as

As a baseline, we also consider the deterministic version of this approach, i.e., dropping KL-divergence and encoding $\mathbf{z}=E(\mathbf{B})$. We call it cAE-GAN and show a comparison in the experiments. There is no guarantee in cAE-GAN on the distribution of the latent space $\mathbf{z}$, which makes the test-time sampling of $\mathbf{z}$ difficult.

We explore another method of enforcing the generator network to utilize the latent code embedding $\mathbf{z}$, while staying close to the actual test time distribution $p(\mathbf{z})$, but from the latent code’s perspective. As shown in Figure 2(d), we start from a randomly drawn latent code $\mathbf{z}$ and attempt to recover it with $\widehat{\mathbf{z}}=E(G(\mathbf{A},\mathbf{z}))$. Note that the encoder $E$ here is producing a point estimate for $\widehat{\mathbf{z}}$, whereas the encoder in the previous section was predicting a Gaussian distribution.

We also include the discriminator loss $L_{\text{GAN}}(G,D)$ (Equation 1) on $\widehat{\mathbf{B}}$ to encourage the network to generate realistic results, and the full loss can be written as:

4 Our Hybrid Model: BicycleGAN

We combine the cVAE-GAN and cLR-GAN objectives in a hybrid model. For cVAE-GAN, the encoding is learned from real data, but a random latent code may not yield realistic images at test time – the KL loss may not be well optimized. Perhaps more importantly, the adversarial classifier $D$ does not have a chance to see results sampled from the prior during training. In cLR-GAN, the latent space is easily sampled from a simple distribution, but the generator is trained without the benefit of seeing ground truth input-output pairs. We propose to train with constraints in both directions, aiming to take advantage of both cycles ($\mathbf{B}\rightarrow\mathbf{z}\rightarrow\widehat{\mathbf{B}}$ and $\mathbf{z}\rightarrow\widehat{\mathbf{B}}\rightarrow\widehat{\mathbf{z}}$), hence the name BicycleGAN.

where the hyper-parameters $\lambda$, $\lambda_{\text{latent}}$, and $\lambda_{\text{KL}}$ control the relative importance of each term.

In the unconditional GAN setting, Larsen et al. observe that using samples from both the prior $\mathcal{N}(0,I)$ and encoded $E(\mathbf{B})$ distributions further improves results. Hence, we also report one variant which is the full objective shown above (Equation 9), but without the reconstruction loss on the latent space $\mathcal{L}_{1}^{\text{latent}}$. We call it cVAE-GAN++, as it is based on cVAE-GAN with an additional loss $\mathcal{L}_{\text{GAN}}(G,D)$, which allows the discriminator to see randomly drawn samples from the prior.

Implementation Details

The code and additional results are publicly available at https://github.com/junyanz/BicycleGAN. Please refer to our website for more details about the datasets, architectures, and training procedures.

For generator $G$, we use the U-Net , which contains an encoder-decoder architecture, with symmetric skip connections. The architecture has been shown to produce strong results in the unimodal image prediction setting when there is a spatial correspondence between input and output pairs. For discriminator $D$, we use two PatchGAN discriminators at different scales, which aims to predict real vs. fake for $70\times 70$ and $140\times 140$ overlapping image patches. For the encoder $E$, we experiment with two networks: (1) ECNN: CNN with a few convolutional and downsampling layers and (2) EResNet: a classifier with several residual blocks .

Training details

Injecting the latent code $\mathbf{z}$ to generator. We explore two ways of propagating the latent code $\mathbf{z}$ to the output, as shown in Figure 3: (1) add_to_input: we spatially replicate a $Z$-dimensional latent code $\mathbf{z}$ to an $H\!\times W\!\times Z$ tensor and concatenate it with the $H\!\times W\!\times 3$ input image and (2) add_to_all: we add $\mathbf{z}$ to each intermediate layer of the network $G$, after spatial replication to the appropriate sizes.

Experiments

Datasets We test our method on several image-to-image translation problems from prior work, including edges $\rightarrow$ photos , Google maps $\rightarrow$ satellite , labels $\rightarrow$ images , and outdoor night $\rightarrow$ day images . These problems are all one-to-many mappings. We train all the models on $256\times 256$ images.

Methods We evaluate the following models described in Section 3: pix2pix+noise, cAE-GAN, cVAE-GAN, cVAE-GAN++, cLR-GAN, and our hybrid model BicycleGAN.

We show qualitative comparison results on Figure 5. We observe that pix2pix+noise typically produces a single realistic output, but does not produce any meaningful variation. cAE-GAN adds variation to the output, but typically at a large cost to result quality. An example on facades is shown on Figure 4.

We observe more variation in the cVAE-GAN, as the latent space is encouraged to encode information about ground truth outputs. However, the space is not densely populated, so drawing random samples may cause artifacts in the output. The cLR-GAN shows less variation in the output, and sometimes suffers from mode collapse. When combining these methods, however, in the hybrid method BicycleGAN, we observe results which are both diverse and realistic. Please see our website for a full set of results.

2 Quantitative Evaluation

We perform a quantitative analysis of the diversity, realism, and latent space distribution on our six variants and baselines. We quantitatively test the Google maps $\rightarrow$ satellites dataset.

Diversity We compute the average distance of random samples in deep feature space. Pretrained networks have been used as a “perceptual loss" in image generation applications , as well as a held-out “validation" score in generative modeling, for example, assessing the semantic quality and diversity of a generative model or the semantic accuracy of a grayscale colorization .

In Figure 6, we show the diversity-score using the LPIPS metric proposed by Learned Perceptual Image Patch Similarity (LPIPS) metric computes distance in AlexNet feature space (conv1-5, pretrained on Imagenet ), with linear weights to better match human perceptual judgments.. For each method, we compute the average distance between 1900 pairs of randomly generated output $\widehat{\mathbf{B}}$ images (sampled from 100 input $\mathbf{A}$ images). Random pairs of ground truth real images in the $\mathbf{B}\in\mathcal{B}$ domain produce an average variation of .262. As we are measuring samples $\widehat{\mathbf{B}}$ which correspond to a specific input $\mathbf{A}$, a system which stays faithful to the input should definitely not exceed this score.

The pix2pix system produces a single point estimate. Adding noise to the system pix2pix+noise produces a small diversity score, confirming the finding in that adding noise does not produce large variation. Using the cAE-GAN model to encode a ground truth image $\mathbf{B}$ into a latent code $\mathbf{z}$ does increase the variation. The cVAE-GAN, cVAE-GAN++, and BicycleGAN models all place explicit constraints on the latent space, and the cLR-GAN model places an implicit constraint through sampling. These four methods all produce similar diversity scores. We note that high diversity scores may also indicate that unnatural images are being generated, causing meaningless variations. Next, we investigate the visual realism of our samples.

Perceptual Realism To judge the visual realism of our results, we use human judgments, as proposed in and later used in . The test sequentially presents a real and generated image to a human for 1 second each, in a random order, asks them to identify the fake, and measures the “fooling" rate. Figure 6(left) shows the realism across methods. The pix2pix+noise model achieves high realism score, but without large diversity, as discussed in the previous section. The cAE-GAN helps produce diversity, but this comes at a large cost to the visual realism. Because the distribution of the learned latent space is unclear, random samples may be from unpopulated regions of the space. Adding the KL-divergence loss in the latent space, used in the cVAE-GAN model, recovers the visual realism. Furthermore, as expected, checking randomly drawn $\mathbf{z}$ vectors in the cVAE-GAN++ model slightly increases realism. The cLR-GAN, which draws $\mathbf{z}$ vectors from the predefined distribution randomly, produces similar realism and diversity scores. However, the cLR-GAN model resulted in large mode collapse - approximately $15\%$ of the outputs produced the same result, independent of the input image. The full hybrid BicycleGAN gets the best of both worlds, as it does not suffer from mode collapse and also has the highest realism score by a significant margin.

Encoder architecture In pix2pix, Isola et al. conduct extensive ablation studies on discriminators and generators. Here we focus on the performance of two encoder architectures, ECNN and EResNet, for our applications on the maps and facades datasets. We find that EResNet better encodes the output image, regarding the image reconstruction loss $||\mathbf{B}-G(\mathbf{A},E(B))||_{1}$ on validation datasets as shown in Table 1. We use EResNet in our final model.

Methods of injecting latent code We evaluate two ways of injecting latent code $\mathbf{z}$: add_to_input and add_to_all (Section 4), regarding the same reconstruction loss $||\mathbf{B}-G(\mathbf{A},E(B))||_{1}$. Table 1 shows that two methods give similar performance. This indicates that the U_Net can already propagate the information well to the output without the additional skip connections from $\mathbf{z}$. We use add_to_all method to inject noise in our final model.

Latent code length We study the BicycleGAN model results with respect to the varying number of dimensions of latent codes $\{2,8,256\}$ in Figure 7. A very low-dimensional latent code may limit the amount of diversity that can be expressed. On the contrary, a very high-dimensional latent code can potentially encode more information about an output image, at the cost of making sampling difficult. The optimal length of $\mathbf{z}$ largely depends on individual datasets and applications, and how much ambiguity there is in the output.

Conclusions

In conclusion, we have evaluated a few methods for combating the problem of mode collapse in the conditional image generation setting. We find that by combining multiple objectives for encouraging a bijective mapping between the latent and output spaces, we obtain results which are more realistic and diverse. We see many interesting avenues of future work, including directly enforcing a distribution in the latent space that encodes semantically meaningful attributes to allow for image-to-image transformations with user controllable parameters.

Acknowledgments We thank Phillip Isola and Tinghui Zhou for helpful discussions. This work was supported in part by Adobe Inc., DARPA, AFRL, DoD MURI award N000141110688, NSF awards IIS-1633310, IIS-1427425, IIS-1212798, the Berkeley Artificial Intelligence Research (BAIR) Lab, and hardware donations from NVIDIA. JYZ is supported by Facebook Graduate Fellowship, RZ by Adobe Research Fellowship, and DP by NVIDIA Graduate Fellowship. Much of this work was done while JYZ and RZ were Adobe Research interns.

Changelog

v1 initial preprint release v2 NIPS camera ready. Added additional implementation details and related work. Replaced VGG-16 with LPIPS for diversity score. Miscellaneous changes to text. v3 Fixed a minor bug in computation of LPIPS score. Figure 6 updated; scores barely change. v4 Updated Adobe Research internship affiliation in acknowledgments.

References