DRIT++: Diverse Image-to-Image Translation via Disentangled Representations

Introduction

Image-to-Image (I2I) translation aims to learn the mapping between different visual domains. Numerous vision and graphics problems can be formulated as I2I translation problems, such as colorization larsson2016colorization ; zhang2016colorization2 (grayscale $\rightarrow$ color), super-resolution lai2017deep ; ledig2016photo ; li2016deep ; li2019joint (low-resolution $\rightarrow$ high-resolution), and photorealistic image synthesis chen2017photographic ; park2019semantic ; wang2017pix2pixhd (label $\rightarrow$ image). In addition, I2I translation can be applied to synthesize images for domain adaptation bousmalis2017unsupervisedda ; Chen2019CrDoCo ; hoffman2017cycada ; murez2018image ; shrivastava2017apple .

Learning the mapping between two visual domains is challenging for two main reasons. First, aligned training image pairs are either difficult to collect (e.g., day scene $\leftrightarrow$ night scene) or do not exist (e.g., artwork $\leftrightarrow$ real photo). Second, many such mappings are inherently multimodal — a single input may correspond to multiple possible outputs. To handle multimodal translation, one possible approach is to inject a random noise vector to the generator for modeling the multimodal data distribution in the target domain. However, mode collapse may still occur easily since the generator often ignores the additional noise vectors.

Several recent efforts have been made to address these issues. The Pix2pix isola2017pix2pix method applies conditional generative adversarial network to I2I translation problems. Nevertheless, the training process requires paired data. A number of recent approaches choi2017stargan ; liu2017unit ; taigman2016unsupervised ; yi2017dualgan ; zhu2017cyclegan relax the dependency on paired training data for learning I2I translation. These methods, however, generate a single output conditioned on the given input image. As shown in isola2017pix2pix ; zhu2017bicyclegan , the strategy of incorporating noise vectors as additional inputs to the generator does not increase variations of generated outputs due to the mode collapse issue. The generators in these methods are likely to overlook the added noise vectors. Most recently, the BicycleGAN zhu2017bicyclegan algorithm tackles the problem of generating diverse outputs in I2I translation by encouraging the one-to-one relationship between the output and the latent vector. Nevertheless, the training process of BicycleGAN requires paired images.

In this paper, we propose a disentangled representation framework for learning to generate diverse outputs with unpaired training data. We propose to embed images onto two spaces: 1) a domain-invariant content space and 2) a domain-specific attribute space as shown in Figure 2. Our generator learns to perform I2I translation conditioned on content features and a latent attribute vector. The domain-specific attribute space aims to model variations within a domain given the same content, while the domain-invariant content space captures information across domains. We disentangle the representations by applying a content adversarial loss to encourage the content features not to carry domain-specific cues, and a latent regression loss to encourage the invertible mapping between the latent attribute vectors and the corresponding outputs. To handle unpaired datasets, we propose a cross-cycle consistency loss using the proposed disentangled representations. Given a pair of unaligned images, we first perform a cross-domain mapping to obtain intermediate results by swapping the attribute vectors from both images. We can then reconstruct the original input image pair by applying the cross-domain mapping one more time and use the proposed cross-cycle consistency loss to enforce the consistency between the original and the reconstructed images. Furthermore, we apply the mode seeking regularization MSGAN to further improve the diversity of generated images. At test time, we can use either 1) randomly sampled vectors from the attribute space to generate diverse outputs or 2) the transferred attribute vectors extracted from existing images for example-guided translation. Figure 1 shows examples of diverse outputs produced by our model.

We evaluate the proposed model with extensive qualitative and quantitative experiments. For various I2I tasks, we show diverse translation results with randomly sampled attribute vectors and example-guided translation with transferred attribute vectors from existing images. In addition to the common dual-domain image-to-image translation, we extend our proposed framework to the more general multi-domain image-to-image translation and demonstrate diverse translation among domains. We measure realism of our results with a user study and the Fréchet inception distance (FID) heusel2017gans , and evaluate diversity using perceptual distance metrics zhang2018perceptual . However, the diversity metric alone does not effectively measure similarity between the distribution of generated images and the distribution of real data. Therefore, we use the Jensen-Shannon Divergence (JSD) distance which measures the similarity between distributions, and the Number of Statistically-Different Bins (NDB) richardson2018NDB metric which determines the relative proportions of samples within clusters predetermined by real data.

We make the following contributions in this work:

1) We introduce a disentangled representation framework for image-to-image translation. We apply a content discriminator to facilitate the factorization of domain-invariant content space and domain-specific attribute space, and a cross-cycle consistency loss that allows us to train the model with unpaired data.

2) Extensive qualitative and quantitative experiments show that our model performs favorably against existing I2I models. Images generated by our model are both diverse and realistic.

3) The proposed disentangled representation and cross-cycle consistency can be applied to multi-domain image-to-image translation for generating diverse images.

Related Work

Generative adversarial networks. The recent years have witnessed rapid advances of generative adversarial networks (GANs) arjovsky2017wgan ; goodfellow2014GAN ; radford2016dcgan for image generation. The core idea of GANs lies in the adversarial loss that enforces the distribution of generated images to match that of the target domain. The generators in GANs can map from noise vectors to realistic images. Several recent efforts exploit conditional GAN in various contexts including conditioned on text reed2016text2img , audio lee2019dancing2music , low-resolution images ledig2016photo , human pose ma2017pose ; albahar2019guided , video frames vondrick2016videogan , and image isola2017pix2pix . Our work focuses on using GAN conditioned on an input image. In contrast to several existing conditional GAN frameworks that require paired training data, our model generates diverse outputs without paired data. As such, our method has wider applicability to problems where paired training datasets are scarce or not available.

Image-to-image translation. I2I translation aims to learn the mapping from a source image domain to a target image domain. The Pix2pix isola2017pix2pix method applies a conditional GAN to model the mapping function. Although high-quality results have been shown, the model training requires paired training data. To train with unpaired data, the CycleGAN zhu2017cyclegan , DiscoGAN kim2017discogan , and UNIT liu2017unit schemes leverage cycle consistency to regularize the training. However, these methods perform generation conditioned solely on an input image and thus produce one single output. Simply injecting a noise vector to a generator is usually not an effective solution to achieve multimodal generation due to the lack of regularization between the noise vectors and the target domain. On the other hand, the BicycleGAN zhu2017bicyclegan algorithm enforces the bijection mapping between the latent and target space to tackle the mode collapse problem. Nevertheless, the method is only applicable to problems with paired training data. Unlike existing work, our method enables I2I translation with diverse outputs in the absence of paired training data.

We note several concurrent methods almahairi2018augmented ; cao2018dida ; huang2018munit ; lin2019explore ; lin2018conditional ; ma2018exemplar (all independently developed) also adopt disentangled representations similar to our work for learning diverse I2I translation from unpaired training data. Furthermore, several approaches StarGAN2018 ; UFDN2018 extend the conventional dual-domain I2I to general multi-domain settings. However, these methods can only achieve one-to-one mapping among domains.

Disentangled representations. The task of learning disentangled representation aims at modeling the factors of data variations. Previous work makes use of labeled data to factorize representations into class-related and class-independent components cheung2014discovering ; kingma2014semi ; makhzani2015adversarial ; mathieu2016disentangling . Recently, numerous unsupervised methods have been developed chen2016infogan ; denton2017unsupervised to learn disentangled representations. The InfoGAN chen2016infogan algorithm achieves disentanglement by maximizing the mutual information between latent variables and data variation. Similar to DrNet denton2017unsupervised that separates time-independent and time-varying components with an adversarial loss, we apply a content adversarial loss to disentangle an image into domain-invariant and domain-specific representations to facilitate learning diverse cross-domain mappings.

Disentangled Representation for I2I Translation

Our approach embeds input images onto a shared content space $\mathcal{C}$, and domain-specific attribute spaces, $\mathcal{A}_{\mathcal{X}}$ and $\mathcal{A}_{\mathcal{Y}}$. Intuitively, the content encoders should encode the common information that is shared between domains onto $\mathcal{C}$, while the attribute encoders should map the remaining domain-specific information onto $\mathcal{A}_{\mathcal{X}}$ and $\mathcal{A}_{\mathcal{Y}}$.

To achieve representation disentanglement, we apply two strategies: weight-sharing and a content discriminator. First, similar to liu2017unit , based on the assumption that two domains share a common latent space, we share the weight between the last layer of $E^{c}_{\mathcal{X}}$ and $E^{c}_{\mathcal{Y}}$ and the first layer of $G_{\mathcal{X}}$ and $G_{\mathcal{Y}}$. Through weight sharing, we enforce the content representation to be mapped onto the same space. However, sharing the same high-level mapping functions does not guarantee the same content representations encode the same information for both domains. Thus, we propose a content discriminator $D^{c}$ which aims to distinguish the domain membership of the encoded content features $z_{x}^{c}$ and $z_{y}^{c}$. On the other hand, content encoders learn to produce encoded content representations whose domain membership cannot be distinguished by the content discriminator $D^{c}$. We express this content adversarial loss as:

2 Cross-cycle Consistency Loss

With the disentangled representation where the content space is shared among domains and the attribute space encodes intra-domain variations, we can perform I2I translation by combining a content representation from an arbitrary image and an attribute representation from an image of the target domain. We leverage this property and propose a cross-cycle consistency. In contrast to cycle consistency constraint in zhu2017cyclegan (i.e., $\mathcal{X}\to\mathcal{Y}\to\mathcal{X}$) which assumes one-to-one mapping between the two domains, the proposed cross-cycle constraint exploit the disentangled content and attribute representations for cyclic reconstruction.

Our cross-cycle constraint consists of two stages of I2I translation.

Forward translation. Given a non-corresponding pair of images $x$ and $y$, we encode them into $\{z_{x}^{c},z_{x}^{a}\}$ and $\{z_{y}^{c},z_{y}^{a}\}$. We then perform the first translation by swapping the attribute representation (i.e., $z_{x}^{a}$ and $z_{y}^{a}$) to generate $\{u,v\}$, where $u\in\mathcal{X},v\in\mathcal{Y}$.

Backward translation. After encoding $u$ and $v$ into $\{z_{u}^{c},z_{u}^{a}\}$ and $\{z_{v}^{c},z_{v}^{a}\}$, we perform the second translation by once again swapping the attribute representation (i.e., $z_{u}^{a}$ and $z_{v}^{a}$).

Here, after two I2I translation stages, the translation should reconstruct the original images $x$ and $y$ (as illustrated in Figure 3). To enforce this constraint, we formulate the cross-cycle consistency loss as:

where $u=G_{\mathcal{X}}(E_{\mathcal{Y}}^{c}(y),E_{\mathcal{X}}^{a}(x))$ and $v=G_{\mathcal{Y}}(E_{\mathcal{X}}^{c}(x),E_{\mathcal{Y}}^{a}(y))$, respectively.

3 Other Loss Functions

In addition to the proposed content adversarial loss and cross-cycle consistency loss, we also use several other loss functions to facilitate network training. We illustrate these additional losses in Figure 4. Starting from the top-right, in the counter-clockwise order:

The full objective function of our network is:

where the hyper-parameters $\lambda$s control the importance of each term.

4 Mode Seeking Regularization

We incorporate the mode seeking regularization MSGAN method to alleviate the mode-collapse problem in conditional generation tasks. Given a conditional image $\mathbf{I}$, latent vectors $\mathbf{z}_{1}$ and $\mathbf{z}_{2}$, and a conditional generator $G$, we use the mode seeking regularization term to maximize the ratio of the distance between $G(\mathbf{I},\mathbf{z}_{1})$ and $G(\mathbf{I},\mathbf{z}_{2})$ with respect to the distance between $\mathbf{z}_{1}$ and $\mathbf{z}_{2}$,

where $d_{\ast}(\cdot)$ denotes the distance metric.

The regularization term can be easily incorporated into the proposed framework:

5 Multi-Domain Image-to-Image Translation

In addition to the translation between two domains, we apply the proposed disentangle representation to the multi-domain setting. Different from typical I2I designed for two domains, multi-domain I2I aims to perform translation among multiple domains with a single generator $G$.

We then perform the forward and backward translation similar to the dual-domain translation.

Experimental Results

Implementation details. We implement the proposed model with PyTorch paszke2017pytorch . We use the input image size of $216\times 216$ for all of our experiments. For the content encoder $E^{c}$, we use an architecture consisting of three convolution layers followed by four residual blocks. For the attribute encoder $E^{a}$, we use a CNN architecture with four convolution layers followed by fully-connected layers. We set the size of the attribute vector to $z^{a}\in R^{8}$ for all experiments. For the generator $G$, we use an architecture consisting of four residual blocks followed by three fractionally strided convolution layers.

Datasets. We evaluate the proposed model on several datasets include Yosemite zhu2017cyclegan (summer and winter scenes), pets (cat and dog) cropped from Google images, artworks zhu2017cyclegan (Monet), and photo-to-portrait cropped from subsets of the WikiArt datasethttps://www.wikiart.org/ and the CelebA dataset liu2015celeb .

Evaluated methods. We perform the evaluation on the following algorithms:

DRIT DRIT , and MUNIT huang2018munit : Multimodal generation frameworks trained with unpaired data.

DRIT w/o $D^{c}$: DRIT model without the content discriminator.

Cycle/Bicycle: We construct a baseline using a combination of CylceGAN and BicycleGAN. Here, we first train CycleGAN on unpaired data to generate corresponding images as pseudo image pairs. We then use this pseudo paired data to train BicycleGAN.

CycleGAN zhu2017cyclegan , and BicycleGAN zhu2017bicyclegan

The proposed DRIT++ method extends the original DRIT method by 1) incorporating mode-seeking regularization for improving sample diversity and 2) generalizing the two-domain model to handle multi-domain image-to-image translation problems. The DRIT++ (multi-domain) algorithm is backward compatible with the DRIT++ (two-domain) and DRIT methodss with comparable performance (as shown in Section LABEL:subsec:subsec:quan). Thus, the DRIT++ (two-domain) method can be viewed as a special case of the DRIT++ (multi-domain) algorithm. The DRIT++ (two-domain) algorithm can improve the visual quality slightly over the DRIT++ (multi-domain) scheme with a category-specific generator and discriminator under the two-domain setting.

Diversity. We first compare the proposed model with other methods in Figure 6. In Figure 7, demonstrate the visual artifacts of images generated by baseline methods. Both our model without $D^{c}$ and Cycle/Bicycle can generate diverse results. However, the results contain clearly visible artifacts. Without the content discriminator, our model fails to capture domain-specific details (e.g., the color of tree and sky). Therefore, the variations of synthesized images lie in global color differences. As the Cycle/Bicycle methods are trained on pseudo paired data generated by CycleGAN, the quality of the pseudo paired data is not high. As a result, the generated images contain limited diversity.

To better analyze the learned domain-specific attribute space, we perform linear interpolation between two given attributes and generate the corresponding images as shown in Figure 9. The interpolation results validate the continuity in the attribute space and show that our model can generalize in the distribution, rather than simply retain visual information.

Mode seeking regularization. We demonstrate the effectiveness of the mode seeking regularization term in Figure 8. The mode seeking regularization term substantially alleviates the mode collapse issue in DRIT DRIT , particularly in the challenging shape-variation translation (i.e., dog-to-cat translation).

Attribute transfer. We demonstrate the results of the attribute transfer in Figure 10. By disentangling content and attribute representations, we are able to perform attribute transfer from images of desired attributes, as illustrated in Figure 3(c). Furthermore, since the content space is shared between two domains, we can generate images conditioned on content features encoded from either domain. Thus our model can achieve not only inter-domain but also intra-domain attribute transfer. Note that intra-domain attribute transfer is not explicitly involved in the training process.

Multi-domain I2I. Figure 11 shows the results of applying the proposed method on the multi-domain I2I. We perform translation among three domains (real images and two artistic styles) and four domains (different weather conditions). Using one single generator, the proposed model is able to perform diverse translation among multiple domains.

2 Quantitative Evaluation

Metrics We conduct quantitative evaluations using the following metrics:

FID. To evaluate the quality of the generated images, we use the FID heusel2017gans metric to measure the distance between the generated distribution and the real one through features extracted by Inception Network szegedy2015going . Lower FID values indicate better quality of the generated images.

LPIPS. To evaluate diversity, we employ LPIPS zhang2018perceptual metric to measure the average feature distances between generated samples. Higher LPIPS scores indicate better diversity among the generated images.

JSD and NDB. To measure the similarity between the distribution between real images and generated one, we adopt two bin-based metrics, JSD and NDB richardson2018NDB . These metrics evaluate the extent of mode missing of generative models. Similar to richardson2018NDB , we first cluster the training samples using K-means into different bins. These bins can be viewed as modes of the real data distribution. We then assign each generated sample to the bin of its nearest neighbor. We compute the bin-proportions of the training samples and the synthesized samples to evaluate the difference between the generated distribution and the real data distribution. The NDB and JSD metrics of the bin-proportion are then computed to measure the level of mode collapse. Lower NDB and JSD scores mean the generated data distribution approaches the real data distribution better by fitting more modes. More discussions on these metrics can be found in richardson2018NDB .

User preference. For evaluating realism of synthesized images, we conduct a user study using pairwise comparison. Given a pair of images sampled from real images and translated images generated from various methods, each subject needs to answer the question “Which image is more realistic?”

Realism vs. diversity. We conduct the experiment using winter $\rightarrow$ summer and cat $\rightarrow$ dog translation with the Yosemite and pets datasets, respectively. Table 2, Table 5, and Figure 12 present the quantitative comparisons with other methods as well as baseline methods. In Table 2, the DRIT++ method performs well on all metrics. The DRIT++ method generates images that are not only realistic, but also diverse and close to the original data distribution. Table 5 validates the effectiveness of the content discriminator, latent regression loss, and mode-seeking regularization in the proposed algorithm. Figure 12 shows the results of user study. The DRIT++ algorithm performs favorably against the state-of-the-art approaches as well as baseline methods.

Multi-domain translation We compare the performance of DRIT++, StarGAN StarGAN2018 , and DosGAN lin2019explore in terms of realism on the weather dataset. For each trial, We translate 1000 testing images to one of four domains and measure the visual quality (in terms of FID) and diversity (using the LPIPS metric). We report the averaged results of 5 trials. Table 6 shows that the disentangled representations by our method not only enable diverse translation, but also improve the quality of generated images. Figure 14 presents qualitative results by the evaluated methods.

Multi-domain model on two-domain translation Two-domain translation is a special case of multi-domain translation problems. We conduct an experiment under the same settings described in Table 2 and 3. As shown in Table 3, our multi-domain model performs well in all metrics against the two-domain translation model that consists of the domain-specific generator and discriminator.

Ablation study on the content discriminator In practice, the content discriminator helps align distributions of the latent content representations of two domains. We conduct experiments on both cat2dog and the Yosemite datasets to illustrate this. The distance between the means of the content representations from two domains is measured by:

Table 4 shows the quantitative results. Furthermore, Figure 13 visualizes the distributions of the latent content representations from two domains using t-SNE. The distance (15) between the content representations of the two domains is much smaller with the help of the content discriminator.

3 High Resolution I2I

We demonstrate that the proposed scheme can be applied to the translation tasks with high-resolution images. We perform image translation on the street scene (GTA Richter_2016_ECCV $\leftrightarrow$ Cityscape cordts2016cityscapes ) dataset. The size of the input image is $720\times 360$ pixels. During the training, we randomly crop the image to the size of $340\times 340$ for memory efficiency consideration. To enhance the quality of the generated high-resolution images, we adopt a multi-scale generator-discriminator structure similar to the StackGAN zhang2017stackgan++ scheme. As shown in Figure 15, we extract the intermediate feature of the generator and pass through a convolutional layer to generate low-resolution images. We utilize an additional discriminator which takes low-resolution images as input. This discriminator enforces the first few layers of the generator to capture the distribution of low-level variations such as colors and image structures. We find such multi-scale generator-discriminator structure facilitate the training and yields more realistic images on high-resolution translation task. To validate the effectiveness of the multi-scale architecture, we show the comparison between (1) adding two more layers to generators and (2) using the multi-scale generator-discriminator architecture in Table 7 and Figure 16. We report the FID and LPIPS scores of the generated images by the two methods on the GTA5 $\rightarrow$ Cityscape translation task. As shown in Table 7, using the multi-scale architecture we can generate more photo-realistic images on the translation task with high-resolution images.

4 Limitations

The performance of the proposed algorithm is limited in several aspects. First, due to the limited amount of training data, the attribute space is not fully exploited. Our I2I translation fails when the sampled attribute vectors locate in under-sampled space, see Figure 17(a). Second, it remains difficult when the domain characteristics differ significantly. For example, Figure 17(b) shows a failure case on the human figure due to the lack of human-related portraits in Monet collections. Third, we use multiple encoders and decoders for the cross-cycle consistency during training, which requires large memory usage. The memory usage limits the application on high-resolution image-to-image translation.

Conclusions

In this paper, we present a novel disentangled representation framework for diverse image-to-image translation with unpaired data. we propose to disentangle the latent space to a content space that encodes common information between domains, and a domain-specific attribute space that can model the diverse variations given the same content. We apply a content discriminator to facilitate the representation disentanglement. We propose a cross-cycle consistency loss for cyclic reconstruction to train in the absence of paired data. Qualitative and quantitative results show that the proposed model produces realistic and diverse images. We also apply the proposed algorithm to domain adaptation and achieve competitive performance compared to the state-of-the-art methods.

Acknowledgements

This work is supported in part by the NSF CAREER Grant #1149783, the NSF Grant #1755785, and gifts from Verisk, Adobe and Google.

References