Generate To Adapt: Aligning Domains using Generative Adversarial Networks

Swami Sankaranarayanan, Yogesh Balaji, Carlos D. Castillo, Rama Chellappa

Introduction

The development of powerful learning algorithms such as Convolutional Neural Networks (CNNs) has provided an effective pipeline for solving many classification problems . The abundance of labeled data has resulted in remarkable improvements for tasks such as the Imagenet challenge: beginning with the CNN framework of AlexNet and more recently ResNets and its variants. Another example is the steady improvements in performance on the LFW dataset . The common theme across all these approaches is the dependence on large amounts of labeled data. While labeled data is available and getting labeled data has been easier over the years, the lack of uniformity of label distributions across different domains results in suboptimal performance of even the most powerful CNN-based algorithms on realistic unseen test data. For example, labeled synthetic data is available in plenty but algorithms trained only on synthetic data perform poorly on real data. This is of vital importance in cases where labeled real data is unavailable. The use of such unlabeled target data to mitigate the shift between source and target distributions is the most useful direction among domain adaptation approaches. Hence this paper focuses on the topic of unsupervised domain adaptation. In this work, we learn an embedding that is robust to the shift between source and target distributions. We achieve this by using unsupervised data sampled from the target distribution to guide the supervised learning procedure that uses data sampled from the source distribution. We propose an adversarial image generation approach to directly learn the shared feature embedding using labeled data from source and unlabeled data from the target. It should be noted that while there have been a few approaches that use an adversarial framework for solving the domain adaptation problem, the novelty of the proposed approach is in using a joint generative discriminative method: the embeddings are learned using a combination of classification loss and an image generation procedure that is modeled using a variant of Generative Adversarial Networks (GANs) .

Figure 1 illustrates the pipeline of the proposed approach. During training, the source images are passed through the feature extraction network (encoder) to obtain an embedding which is then used by the label prediction network (classifier) for predicting the source label and also used by the generator to generate a realistic source image. The realistic nature of the images from the generator ( $G$ ) is controlled by the discriminator ( $D$ ). The encoder is updated based on the discriminative gradients from the classifier and generative gradients from the adversarial framework. Given unlabeled target images, the encoder is updated using only gradients from the adversarial part, since the labels are unavailable. Thus, the encoder learns to discriminate better even in the target domain using the knowledge imparted by the generator-discriminator pair. By using the discriminator as a multi-class classifier, we ensure that the gradient signals backpropagated by the discriminator for the unlabeled target images belong to the feature space of the respective classes. By sampling from the distribution of the generator after training, we show that the network has indeed learned to bring the source and target distributions closer.

The main contribution of this work is to provide an adversarial image generation approach for unsupervised domain adaptation that directly learns a joint feature space in which the distance between source and target distributions is minimized. Different from contemporary approaches that achieve a similar objective by using a GAN as a data augmenter, our approach achieves superior results even in cases where a stand along image generation process is bound to fail (such as in the OFFICE dataset). This is done by utilizing the GAN framework to address the domain shift directly in the feature space learnt by the encoder. Our experiments show that the proposed approach yields superior results compared to similar approaches which update the embedding based on auto-encoders or disentangling the domain information from the embedding by learning a separate domain classifier .

This paper is organized as follows: We begin in Section 2 by describing existing approaches for the unsupervised domain adaptation problem. In Section 3, we describe in detail the formulation of our approach and the iterative training procedure. The experimental setups and the results are discussed in Section 4 using both quantitative and qualitative experiments, followed by discussion and conclusion in Section 5

Related Work

Domain adaptation is an actively researched topic in many areas of Artificial Intelligence including Machine Learning, Natural Language Processing and Computer Vision. In this section, we describe techniques related to visual domain adaptation. Earlier approaches to domain adaptation focused on building feature representations that are invariant across domains. This was accomplished either by feature reweighting and selection mechanisms , or by learning an explicit feature transformation that aligns source distribution to the target distribution (, , ). The ability to deep neural networks to learn powerful representations [, ] has been harnessed to perform unsupervised domain adaptation in recent works [, , , , ]. The underlying idea behind such methods is to minimize a suitable loss function that captures domain discrepancy, in addition to the task being solved.

Deep learning methods for visual domain adaptation can be broadly grouped into few major categories. One line of work uses Maximum Mean Discrepancy(MMD) as a metric to measure the shift across domains. Deep Domain Confusion (DDC) jointly minimizes the classification loss and MMD loss of the last fully connected layer. Deep Adaptation Networks (DAN) extends this idea by embedding all task specific layers in a reproducing kernel Hilbert space and minimizing the MMD in the projected space. In addition to MMD, Residual Transfer Networks (RTN) uses a gated residual layer for classifier adaptation. Joint Adaptation Networks learn a transfer network by aligning the joint distributions of multiple domain-specific layers across domains based on a Joint Maximum Mean Discrepancy (JMMD) criterion.

Another class of methods uses adversarial losses to perform domain adaptation. Revgrad employs a domain classification network which aims to discriminate the source and the target embeddings. The goal of the feature extraction network is to produce embeddings that maximize the domain classifier loss, while at the same time minimizing the label prediction loss. This is accomplished by negating the gradients coming from the domain classification network. Adversarial Discriminative Domain Adaptation (ADDA) on the other hand learns separate feature extraction networks for source and target, and trains the target CNN so that a domain classifier cannot distinguish the embeddings produced by the source or target CNNs.

While methods discussed above apply adversarial losses in the embedding space, there has been a lot of interest recently to perform adaptation by applying adversarial losses in the pixel space. Such approaches primarily use generative models such as GANs to perform cross-domain image mapping. and use adversarial networks to map source images to target and perform adaptation in the transferred space. Coupled GAN (CoGAN) on the other hand trains a coupled generative model that learns the joint data distribution across the two domains. A domain invariant classifier is learnt by sharing weights with the discriminator of the CoGAN network.

Comparison to recent GAN-based DA approaches: While previous approaches such as and use GANs as a data augmentation step, we use a GAN to obtain rich gradient information that makes the learned embeddings domain adaptive. Unlike the previous methods, our approach does not completely rely on a successful image generation process. As a result, our method works well in cases where image generation is hard (eg. in the OFFICE dataset where the number of samples per class is limited). We observed that in such cases, even though the generator network we use performs a mere style transfer, yet this is sufficient for providing good gradient information for successfully aligning the domains, as demonstrated by our superior performance on the OFFICE dataset.

Approach

Several approaches including learning entropy-based metrics , learning a domain classifier based on a embedding network or denoising autoencoders have been used to transfer information between source and target distributions. In this work, we propose a GAN-based approach to bridge the gap between source and target domains. We accomplish this by using both generative and a discriminative processes thus ensuring a rich information transfer to the learnt embedding.

Overview of GANs: In a traditional GAN, two competing mappings are learned: the discriminator $D$ and the generator $G$ , both of which are modeled as deep neural networks. $G$ and $D$ play a minmax game where $D$ tries to classify the generated samples as fake and $G$ tries to fool $D$ by producing examples that are as realistic as possible. More formally, to train a GAN, the following optimization problem is solved in an iterative manner:

$D(x)$ represents the probability that $x$ came from the real data distribution rather than the distribution modeled by the generator $G$ . As an extension to traditional GANs, conditional GANs enable conditioning the generator and discriminator mappings on additional data such as a class label or an embedding. They have been shown to generate images of digits and faces conditioned on the class label or the embedding respectively . Training a conditional GAN involves optimizing the following minimax objective:

In this work, we employ a variant of the conditional GAN called Auxiliary Classifier GAN (AC-GAN) where the discriminator is modeled as a multi-class classifier instead of providing conditioning information at the input. We modify the AC-GAN set up for the domain adaptation problem as follows:

(c) The discriminator mapping $D$ takes the real image $x$ or the generated image $G(x_{g})$ as input and outputs two distributions: (1) $D_{data}(x)$ : the probability of the input being real, which is modeled as a binary classifier. (2) $D_{cls}(x)$ : the class probability distribution of the input $x$ , which is modeled as a ( $N_{c}$ )-way classifier. To clarify notation, we use ${D_{cls}(x)}_{y}$ to imply the probability assigned by the classifier mapping $D_{cls}$ for input $x$ to class $y$ . It should be noted that, for target data, since class labels are unknown, only $D_{data}$ is used to backpropagate the gradients.

Now, we describe our optimization procedure in detail. To jointly learn the embedding and the generator-discriminator pair, we optimize the $D$ , $G$ , $F$ and $C$ networks in an alternating manner:

Given source images as input, $D$ outputs two distributions $D_{data}$ and $D_{cls}$ . $D_{data}$ is optimized by minimizing a binary cross entropy loss $L_{data,src}$ and $D_{cls}$ is optimized by minimizing the cross entropy loss $L_{cls,src}$ between the source labels and the model predictive distribution $D_{cls}(x)$ . In the case of source inputs, the gradients are generated using the following loss functions:

Using the gradients from $D$ , $G$ is updated using a combination of adversarial loss and classification loss to produce realistic class consistent source images.

$F$ and $C$ are updated based on the source images and source labels in a traditional supervised manner. $F$ is also updated using the adversarial gradients from $D$ so that the feature learning and image generation processes co-occur smoothly.

In the final step, the real target images are presented as input to $F$ . The target embeddings output by $F$ along with the random noise vector $z$ and the fake label encoding $l$ are input to $G$ . The generated target images $G(x_{g})$ are then given as input to $D$ . As described above, $D$ outputs two distributions but the loss function is evaluated only for $D_{data}$ since in the unsupervised case considered here, target labels are not provided during training. Hence, $D$ is updated to classify the generated target images as fake as follows:

In order to transfer the knowledge of target distribution to the embedding, $F$ is updated using the gradients from $D_{data}$ that corresponds to the generated target images being classified as real:

The proposed iterative optimization procedure is summarized as a pseudocode in Algorithm 1. $\alpha$ and $\beta$ are the coefficients that trade off between the classification loss and the source and target adversarial losses. Based on our experiments, we find that our approach is not overly sensitive to the cost coefficients $\alpha$ and $\beta$ . However, the value of the parameter is dependent on the application and size of the dataset. Such specifications are mentioned in the supplementary material.

The main strength of our approach is how the target images are used to update the embedding. Given a batch of target images as input, we update the embedding $F$ by using the following binary loss term:

where $x_{g}$ is the concatenated input to $G$ as described earlier and $\beta$ is the weight coefficient for the target adversarial loss. The use of target data is intended to bring the source and target distributions closer in the feature space learned by $F$ . To achieve this, we update the $F$ network to produce class consistent embeddings for both source and target data. Performing this update for source data is straightforward since the source labels are available during training. Since labels are unavailable for target data, we use the generative ability of the $G$ - $D$ pair for obtaining the required gradients.

Given source inputs, $G$ is updated to fool $D$ using gradients from Eq. (7) which provide the conditioning required for $G$ to produce class consistent fake images. Given target inputs, the update in Eq. (11) encourages $F$ to produce target embeddings that are aligned with the source distribution. As training progresses, the class conditioning information learned by $G$ during the source update (Eq. (7)) was found to be sufficient for it to produce class consistent images for target embeddings as well. This symbiotic relationship between the embedding and the adversarial framework contributes to the success of the proposed approach.

Experiments and Results

This section reports the experimental validation of our approach. We perform a thorough study by conducting experiments across three adaptation settings: (1) low domain shift and simple data distribution: DIGITS dataset, (2) moderate domain shift and complex data distribution: OFFICE dataset, (3) high domain shift and complex data distribution: Synthetic to real adaptation. By complex data distribution, we denote datasets containing images with high variability and limited number of samples. Our methods performs well in all three regimes, thus demonstrating the versatility of our approach. Training code: https://goo.gl/zUVeqC

The first set of experiments involve digit classification in three standard DIGITS datasets: MNIST , USPS and SVHN . Each dataset contains digits belonging to $10$ classes (0-9). MNIST and USPS are large datasets of handwritten digits captured under constrained conditions. SVHN dataset, on the other hand was obtained by cropping house numbers in Google Street View images and hence captures much more diversity. We test the three common domain adaptation settings: SVHN $\to$ MNIST, MNIST $\to$ USPS and USPS $\to$ MNIST. In each setting, we use the label information only from the source domain, thus following the unsupervised protocol.

For all digit experiments, following other recent works , we use a modified version of Lenet architecture as our $F$ network. For $G$ and $D$ networks, we use architectures similar to those used in DCGAN .

We start with the easy case of adaptation involving MNIST and USPS. The MNIST dataset is split into $60000$ training and $10000$ test images, while the USPS dataset contains $7291$ training and $2007$ test images. We run our experiments in two settings: (1) using the entire training set of MNIST and USPS (MNIST $\leftrightarrow$ USPS (f)), and (2) using the protocol established in , sampling $2000$ images from MNIST and $1800$ images from USPS (MNIST $\leftrightarrow$ USPS (p)). Table. 1 presents the results of the proposed approach in comparison with other contemporary approaches. The reported numbers are averaged over 5 independent runs with different random samplings or initializations. We can observe that our approach achieves the best performance in all cases except in the MNIST $\to$ USPS full protocol case where our accuracy is very close to the best performing method.

(b) SVHN →→\rightarrow MNIST

Compared to the previous experiment, SVHN $\rightarrow$ MNIST presents a harder case of domain adaptation owing to larger domain gap. Following other works , we use the entire training set (labeled $73257$ SVHN images and unlabeled $60000$ MNIST images) to train our model, and evaluate on the training set of the target domain (MNIST dataset). From Table. 2, we observe that our method significantly improves the performance of the source-only model from $60.3\%$ to $92.4\%$ , which results in a performance gain of $32.1\%$ . We also outperform other methods by a large margin, obtaining at least $10.4\%$ performance improvement. A visualization of this improvement in performance is done in figure 2, where we show a t-SNE plot of the features of the embedding network $F$ for the adapted and non-adapted cases.

2 OFFICE experiments

The next set of experiments involve the OFFICE dataset, which is a small scale dataset containing images belonging to $31$ classes from three domains - Amazon, Webcam and DSLR, each containing 2817, 795 and 498 images respectively. The small dataset size poses a challenge to our approach since we rely on GAN which demands more data for better image generation. Nevertheless, we perform experiments on the OFFICE dataset to demonstrate that though our method does not succeed in generating very realistic images, the approach still results in improved performance by using the generative process to obtain domain invariant feature representations.

Training deep networks with randomly initialized weights on small datasets give poor performance. So, an effective technique used in practice is to fine-tune networks trained on a related task having large data . Following this rationale, we initialized the $F$ network using a pre-trained ResNet-50 model trained on Imagenet. For $D$ and $G$ networks, we used architectures similar to the ones used in the Digits experiments. It should be noted that even though the inputs are $224\times 224$ , the $G$ network is made to generate a downsampled version of size $64\times 64$ . Standard data augmentation steps involving mean normalization, random cropping and mirroring were performed.

In all our experiments, we follow the standard unsupervised protocol - using the entire labeled data in the source domain and unlabeled data in the target domain. Table 2 reports the performance of our method in comparison to other methods. We observe that our method obtains the state-of-the-art performance in all the settings. In particular, we get good performance improvement consistently in all hard transfer cases: $A\rightarrow W$ , $A\rightarrow D$ , $W\rightarrow A$ and $D\rightarrow A$ .

3 Synthetic to Real experiments

To test the effectiveness of the proposed approach further, we perform experiments in the hardest case of domain adaptation involving adaptation from synthetic to real datasets. This setting is particularly interesting because of its enormous practical implications. In this experiment, we use CAD synthetic dataset and a subset of PASCAL VOC dataset as our source and target sets respectively. The CAD synthetic dataset contains multiple renderings of 3D CAD models of the $20$ object categories contained in the PASCAL dataset. To create the datasets, we follow the protocol described in : The CAD dataset contains six subsets with different configurations (i.e. RR-RR, W-RR, W-UG, RR-UG, RG-UG, RG-RR). Of these, we use images with white background (W-UG subset) as our training set. To generate the target set, we crop 14976 patches from 4952 images of the PASCAL VOC 2007 test set using the object bounding boxes provided. The lack of realistic background and texture in the CAD synthetic dataset increases the disparity from the natural image manifold, thus making domain adaptation extremely challenging.

Due to the high domain gap, we observed that models trained on the CAD synthetic dataset with randomly initialized weights performed very poorly on the target dataset. So, similar to the previous set of experiments, we initialized the $F$ network with pretrained models. In particular, we removed the last fully connected layer from the VGG16 model trained on Imagenet and used it as our $F$ network. Note that the same $F$ network is used to train all other methods for fair comparison. Table. 6 reports the results of the experiments we ran on the Synthetic to real setting. We can observe that our method improves the baseline performance from $38.1\%$ to $50.4\%$ in addition to outperforming all other compared methods.

4 VISDA challenge

In this section, we present the results on VISDA dataset - a large scale testbed for unsupervised domain adaptation algorithms. The task is to train classification models on synthetic dataset generated from the renderings of 3D CAD models and adapt these models to real images which are drawn from Microsoft COCO (validation set) and Youtube Bounding Box dataset (test set). We train our models using the same hyper-parameter settings and data augmentation scheme as the previous experiment. Table. 4 presents the results on the VISDA classification challenge. We find that our method achieves significant performance gains compared to the baseline model.

5 Ablation Study

In this experiment, we study the effect of each individual component to the overall performance. The embedding network $F$ is updated using a combination of losses from two streams (1) supervised classification stream and (2) adversarial stream, as shown in Figure 1. The adversarial stream consists of the G-D pair, with D containing two components - real/fake classifier which we denote as $C_{1}$ , and auxiliary classifier which we denote as $C_{2}$ . We report the performance on the following three settings: (1) using only the Stream 1 and only using source data to train - this corresponds to the Source-only setting (2) Using stream 1 + $C_{1}$ classifier from stream 2 - this corresponds to the case where source and target embeddings are forced to produce source-like images, but class information is not provided to the discriminator and (3) Using stream 1 + stream2 ( $C_{1}$ + $C_{2}$ ) - this is our entire system. For settings (2) and (3) we utilized labeled source data and unlabeled target data during training. Table 5 presents the results of this experiment.

We observe that using only the real/fake classifier $C_{1}$ in the discriminator does improve performance, but the auxiliary classifier $C_{2}$ is needed to get the full performance benefit. This can be attributed to the mode collapse problem in traditional GANs (we observed that training without $C_{2}$ resulted in missing modes and mismatched mappings where embeddings get mapped to images of wrong classes), hence resulting in sub-optimal performance. Use of an auxiliary classifier objective in $D$ stabilizes the GAN training as observed in and significantly improves the performance of our approach.

Conclusion and Future Work

In this paper, we addressed the problem of unsupervised visual domain adaptation. We proposed a joint adversarial-discriminative approach that transfers the information of the target distribution to the learned embedding using a generator-discriminator pair. We demonstrated the superiority of our approach over existing methods that address this problem using experiments on three different tasks, thus making our approach more generally applicable and versatile. Some avenues for future work include using stronger encoder architectures and applications of our approach to more challenging domain adaptation problems such as RGB-D object recognition and medical imaging.

Acknowledgement

This research is based upon work supported by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA R&D Contract No. 2014-14071600012. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon.

Network Architectures and Hyperparameters

This section describes the details of the network architectures used in our experiments. A detailed description of all the architectures can be found in Fig. 3

For $SVHN\to MNIST$ experiment, we used $DigF1$ , $DigC1$ , $DigG$ and $DigD$ architectures mentioned in Fig. 3 as our $F$ , $C$ , $G$ and $D$ networks respectively. For all other digit experiments, we use $DigF2$ , $DigC2$ , $DigG$ and $DigD$ . All models were trained from scratch and were initialized using random Gaussian noise with standard deviation $0.01$ . We used Adam solver with base learning rate of $0.0005$ and momentum $0.8$ to train our models. The cost coefficients $\alpha$ and $\beta$ are set as $0.1$ and $0.03$ respectively based on validation splits. We resize all input images to $32\times 32$ and scale their values to the range $$.

For OFFICE experiments, we used $OfcC$ , $OsG$ and $OsD$ architectures mentioned in Fig. 3 as our $C$ , $G$ and $D$ networks respectively. The $F$ network is initialized with pretrained Resnet50 model trained on ImageNet, the last layer of which is removed and the resulting $2048$ dimensional vector is used as the feature embedding. We use Adam solver for optimization with a base learning rate of 0.0004 and momentum 0.7 for all the experiments. The dimension of the random noise vector is set as $128$ and the cost coefficient $\alpha$ and $\beta$ are both set as $0.01$ .

Similar to OFFICE experiments, we used $SynC$ , $OsG$ and $OsD$ architectures mentioned in Fig. 3 as our $C$ , $G$ and $D$ networks respectively. We remove the last layer of the pretrained VGG16 model trained on Imagenet, and initialize it as our $F$ network. The resulting $4096$ dimensional vector is used as the feature embedding. For all the experiments, we used the same hyperparameter settings as those used in the Office experiments.

Noise Analysis

Generation visualization

In Fig. 5, we show some sample images generated by the $G$ network in two experimental settings - $SVHN\to MNIST$ and Office $A\to W$ . The top set of images show the generations when the input to the system are the samples taken from the source dataset, while the bottom set are the generations when inputs are the images from the target dataset. We make the following observations: (1) The quality of image generation is better in the digits experiments compared to the Office experiments (2) The generator is able to produce source-like images for both the source and target inputs in a class-consistent manner (3) There is mode collapse in the generations produced in the Office experiments.

The difficulty of GANs in generating realistic images in the Office and Synthetic to real datasets makes it significantly hard for the methods that use cross-domain image generation as a data augmentation step. Since we rely on the image generation as a mode for deriving rich gradients to the feature extraction network, our method works well even in the presence of severe mode collapse and poor generation quality.

Synthetic to Real adaptation with ResNet

This experiment is an extension to the Synthetic to Real experiments in the main paper. Instead of initializing $F$ network with the pretrained VGG16 model, we initialize it with pretrained Resnet-50 model trained on ImageNet as done in the OFFICE experiments. The results of the experiments are presented in Table. 6. We observe that the model trained only on source domain achieves $30.2\%$ performance, which is $7.9\%$ less than the VGG16 baseline performance mentioned in the main paper. However, our method achieves a performance of $46.5\%$ (which is $16.3\%$ above the baseline) and outperforms other compared approaches.