Learning from Extrinsic and Intrinsic Supervisions for Domain Generalization

Introduction

The rise of deep neural networks has achieved promising results in various computer vision tasks. Most of these achievements are based on supervised learning, which assumes that the models are trained and tested on the samples drawn from the same distribution or domain. However, in many real-world scenarios, the training and test samples are often acquired under different criteria. Therefore, the trained network may perform poorly on “unseen” test data with domain discrepancy from the training data. To address this limitation, researchers have studied how to alleviate the performance degradation of a trained network among different domains. For instance, by utilizing labeled (or unlabeled) target domain samples, various domain adaptation methods have been proposed to minimize the domain discrepancy by aligning the source and target domain distributions .

Although these domain adaptation methods can achieve better performance on the target domain, there exists an indispensable demand to pre-collect and access target domain data during the network training. Moreover, it needs to re-train the network to adapt to every new target domain. However, in real-world applications, it is often the case that adequate target domain data is not available during the training process . For example, it is difficult for an automated driving system to know which domain (e.g., city, weather) the self-driving car will be used. Therefore, it has a broad interest in studying how to learn a generalizable network that can be directly applied to new “unseen” target domains. Recently, the community develops domain generalization methods to improve the model generalization ability on unseen target domains by utilizing the multiple source domains.

Most existing domain generalization methods attempt to extract the shared domain-invariant semantic features among multiple source domains . For example, Li et al. extend an adversarial auto-encoder by imposing the Maximum Mean Discrepancy (MMD) measure to align the distributions among different domains. Since there is no specific prior information from target domains during the training, some works have investigated the effectiveness of increasing the diversity of the inputs by creating synthetic samples to improve the generalization ability of networks . For instance, Yue et al. propose a domain randomization method with Generative Adversarial Networks (GANs) to learn a model with high generalizability. Meta-learning has also been introduced to address the domain generalization problem via an episodic training . Very recently, Carlucci et al. introduce a self-supervision task by predicting relative positions of image patches to constrain the semantic feature learning for domain generalization. This shows that the self-supervised task can discover invariance in images with different patch orders and thus improve the network generalization. Such self-supervision task only considers the regularization within images but does not explore the valuable relationship among images across different domains to further enhance the discriminability and transferability of semantic features.

The generalization of deep neural networks relies crucially on the ability to learn and adapt knowledge across various domains. We argue that a generalized object recognition system should well understand the relationships among different objects and the objects themselves at the same time. Particularly, on the one hand, exploring the relationship among different objects (i.e., extrinsic supervision) guides the network to extract domain-independent yet category-specific representation, facilitating decision-boundary learning. On the other hand, exploring context or shape constraint within a single image (i.e., intrinsic supervision) introduces necessary regularization for network training, broadening the network understanding of the object.

To this end, we present a new framework called EISNet that learns how to generalize across domains by simultaneously incorporating extrinsic supervision and intrinsic supervision for images from multi-source domains. We formulate our framework as a multi-task learning paradigm for general feature learning, as shown in Fig. 1. Besides conducting the common supervised recognition task, we seamlessly integrate a momentum metric learning task and a self-supervised auxiliary task into our framework to utilize the extrinsic and intrinsic supervisions, respectively. Specifically, we develop an effective momentum metric learning scheme with the $K$-hard negative selector to encourage the network to explore the image relationship and enhance the discriminability learning. The $K$-hard negative selector is able to filter the informative hard triplets, while the momentum updated encoder guarantees the consistency of embedded features stored in the memory bank, which stabilizes the training process. We then introduce a jigsaw puzzle solving task to learn the spatial relationship of images parts. The three kinds of tasks share the same feature encoder and are optimized in an end-to-end manner. We demonstrate the effectiveness of our approach on two object recognition benchmarks. Our EISNet achieves the state-of-the-art performance.

Related Work

The goal of unsupervised domain adaptation is to learn a general model with source domain images and unlabeled target domain images, so that the model could perform well on the target domain. Under such a problem setting, images from the target domain can be utilized to guide the optimization procedure. The general idea of domain adaption is to align the source domain and target domain distributions in the input level , semantic feature level , or output space level . Most methods adopt Generative Adversarial Networks and achieve better performance on the target domain data. However, training domain adaptation models need to access unlabeled target domain data, making it impractical for some real-world applications.

Domain generalization is an active research area in recent years. Its goal is to train a neural network on multiple source domains and produce a trained model that can be applied directly to unseen target domain. Since there is no specific prior guidance from the target domain during the training procedure, some domain generalization methods proposed to generate synthetic images derived from the given multiple source domains to increase the diversity of the input images, so that the network could learn from a larger data space . Another promising direction is to extract domain-invariant features over multiple source domains . For example, Li et al. developed a low-rank parameterized CNN model for domain generalization and proposed the domain generalization benchmark dataset PACS. Motiian et al. presented a unified framework by exploiting the Siamese architecture to learn a discriminative space. A novel framework based on adversarial autoencoders was presented by Li et al. to learn a generalized latent feature representation across domains. Recently, meta-learning-based episodic training was designed to tackle domain generalization problems . Li et al. developed an episodic training procedure to expose the network to domain shift that characterizes a novel domain at runtime to improve the robustness of the network. Our work is most related to , which introduced self-supervision signals to regularize the semantic feature learning. However, besides the self-supervision signals within a single image, we further exploit the extrinsic relationship among image samples across different domains to improve the feature compactness.

0.2 Metric learning

Our work is also related to metric learning, which aims to learn a metric to minimize the intra-class distances and maximize the inter-class variations . With the development of deep learning, distance metric also benefits the feature embedding learning for better discrimination . Recently, the metric learning strategies have attracted a lot of attention on face verification and recognition , fine-grained object recognition , image retrieval , and so on. Different from previous applications, in this work, we adopt the conventional triplet loss with more informative negative selection and momentum feature extraction for domain generalization.

0.3 Self-supervision

Self-supervision is a recent paradigm for unsupervised learning. The idea is to design annotation-free (i.e., self-supervised) tasks for feature learning to facilitate the main task learning. Annotation-free tasks can be predictions of the image colors , relative locations of patches from the same image , image inpainting , and image rotation . Typically, self-supervised tasks are used as network pre-train to learn general image features. Recently, it is trained as an auxiliary task to promote the mainstream task by sharing semantic features . In this paper, we inherit the advantage of self-supervision to boost the network generalization ability.

Method

The overall framework of the proposed EISNet is illustrated in Fig. 1. We adopt the classical classification loss, i.e., Cross-Entropy, to minimize the objective $\mathcal{L}_{c}(h_{\psi}(f_{\theta}(x)),y)$ that measures the difference between the ground truth $y$ and the network prediction $\hat{y}=h_{\psi}(f_{\theta}(x))$. To avoid performance degradation on unseen target domain, we introduce two additional complementary supervisions to our framework. One is an extrinsic supervision with momentum metric learning, and the other is an intrinsic supervision with a self-supervised auxiliary task. The momentum metric learning is employed by a triplet loss with a $K$-hard negative selector on the momentum updated embeddings stored in a large memory bank. We implement a self-supervised auxiliary task by predicting the order of patches within an image. All these tasks adopt a shared encoder $f$ and are seamlessly integrated into an end-to-end learning framework. Below, we introduce the extrinsic supervision and intrinsic self-supervision in detail.

For the domain generalization problem, it is necessary to ensure the features of samples with the same label close to each other, while the features of different class samples being far apart. Otherwise, the predictions on the unseen target domain may suffer from ambiguous decision boundaries and performance degradation . This is well aligned with the philosophy of metric learning. Therefore, we design a momentum metric-learning scheme to encourage the network to learn such domain-independent yet class-specific features by considering the mutual relation among samples across domains. Specifically, we propose a novel $K$-hard negative selector for triple loss to improve the training effectiveness by selecting informative triplets in the memory bank, and a momentum updated encoder to guarantee the representation consistency among the embeddings stored in the memory bank.

The triplet loss is widely used to learn feature embedding based on the relative similarity of the sampled pairs. The goal of the original triplet loss is to assign close distance to pairs of similar samples (i.e., positive pair) and long distance to pairs of dissimilar samples (i.e., negative pair). For example, we can extract the feature representation $v_{i}$ of each image $x_{i}$ from multi-source domains with the feature Encoder $f_{\theta}$. Then by fixing an anchor sample $x_{a}$, we choose a corresponding positive sample $x_{p}$ with the same class label as $x_{a}$, and a random negative sample $x_{n}$ with different class label from $x_{a}$ to form a triplet $\mathcal{T}=\{(x_{a},x_{p},x_{n})|y_{a}=y_{p},\ y_{a}\neq y_{n}\}$. Accordingly, the objective of the original triplet loss is formulated as

where $[\cdot]_{+}=max(0,\cdot)$, $d(x_{i},x_{j})$ represents the distance between the samples, and the margin is a standard relaxation coefficient. In general, we use the Euclidean distance to measure distances between the embedded features. Then, the distance between samples $x_{i}$ and $x_{j}$ is defined as

The negative sample selection process in the original triplet loss is shown in Fig. 2(a). Since the selected negative sample may already obey the triplet constraint, the training with the original triplet loss selector may not be efficient. To avoid useless training, inspired by , we propose a novel $K$-hard negative online selector, which extends the triplet with $K$ negatives that violate the triplet constraint within a certain of margin. Specifically, given a sampled anchor, we randomly choose one positive sample with the same class label as the anchor, and select $K$ hard negative samples $x_{n_{i}},i=\{1,2,...,K\}$ following

In an extreme case, the number of hard negative samples may be zero, then we random select negative samples without the distance constraint. Therefore, the objective of the proposed triple loss with $K$-hard negative selector can be represented as

We illustrate the triplet selection process of semi-hard selector ($K=1$) and $K$-hard selector ($K=2$) in Fig. 2 (b) (c) for a better understanding. Compared with the original triplet loss, our proposed triple loss equipped with the $K$-hard negative selector considers more informative hard negatives for each anchor, thus facilitating the feature encoder to learn more discriminative features.

1.2 Efficient learning with memory bank

The way to select informative triplet pairs has a large influence on the feature embedding. Good features can be learned from a large sample pool that includes a rich set of negative samples . However, selecting $K$-hard triplets from the whole sample pool is not efficient. To increase the diversity of selected triplet pairs while reducing the computation burden, we maintain memory bank $\mathcal{V}$ to store the feature representation $v_{i}$ of historical samples with a size of $m$. Instead of calculating the embedded features of all the images at each iteration, we utilize the stored features to select the $K$-hard triplet samples. Note that we also keep the class label $y_{i}$ along with representation $v_{i}$ in the memory bank to filter the negatives, as shown in Fig. 1. During the network training, we dynamically update the memory bank by discarding the oldest items and feeding the new batch of embedded features, where the memory bank acts as a queue.

1.3 Momentum updated encoder

With the memory bank, we can improve the efficiency of triplet sample selection. However, the representation consistency between the current samples and historical samples in the memory bank is reduced due to the rapidly-changed encoder . Therefore, instead of utilizing the same feature encoder to extract the representation of current samples and historical samples, we adopt a new Momentum updated Encoder (MuEncoder) to generate feature representation for the samples in the memory bank. Formally, we denote the parameters of Encoder and MuEncoder as $\theta_{f}$ and $\theta_{g}$, respectively. The Encoder parameter $\theta_{f}$ is optimized by a back-propagation of the loss function, while the MuEncoder parameter $\theta_{g}$ is updated as a moving average of Encoder parameters $\theta_{f}$ following

where $\delta$ is a momentum coefficient to control the update degree of MuEncoder. Since the MuEncoder evolves more smoothly than Encoder, the update of different features in the memory bank is not rapid, thereby easing the triplet loss update. This is confirmed by the experimental results. In our preliminary experiments, we found that a large momentum coefficient $\delta$ by slowly updating $\theta_{g}$ could generate better results than rapid updating, which indicates that a slow update of MuEncoder is able to guarantee the representation consistency.

2 Intrinsic Supervision with Self-supervised Auxiliary Task

To broaden the network understanding of the data, we propose to utilize the intrinsic supervision within a single image to impose a regularization into the feature embedding by adding auxiliary self-supervised tasks on all the source domain images. A similar idea has been adopted in domain adaptation and Generative Adversarial Networks training . The auxiliary self-supervised task is able to exploit the intrinsic semantic information within a single image to provide informative feature representations for the main task.

There are plenty of works focusing on designing auxiliary self-supervised tasks, such as rotation degree prediction and relative location prediction of two patches in one image . Here, we employ the recently-proposed solving jigsaw puzzles as our auxiliary task. However, most of the self-supervised tasks focusing on high-level semantic feature learning can be incorporated into our framework. Specifically, we first divide an image into nine ($3\times 3$) patches, and shuffle these patches within the $30$ different combinations following . As pointed by , the model achieves the highest performance when the class number is set as $30$ and the order prediction performance decreases when the task becomes more difficult with more orders. A new auxiliary task branch $h_{a}$ follows the extracted feature representation $f_{\theta}$ to predict the ordering of the patches. A Cross-Entropy loss is applied to tackle this order classification task:

where $y^{a}$ and $p^{a}$ are the ground-truth order and predicted order from the auxiliary task branch, respectively. We use $c_{a}=0$ to represent the original images without patch shuffle, leading to a total of 31 classes.

Overall, we formulate the whole framework as a multi-task learning paradigm. The total objective function to train the network is represented as

where $\alpha,\ \beta,\ \text{and}\ \gamma$ are hyper-parameters to balance the weights of the basic classification supervision, extrinsic relationship supervision, and intrinsic self-supervision, respectively.

Experiments

We evaluate our method on two public domain generalization benchmark datasets: VLCS and PACS. VLCS is a classic domain generalization benchmark for image classification, which includes five object categories from four domains (PASCAL VOC 2007, LabelMe, Caltech, and Sun datasets). PACS is a recent domain generalization benchmark for object recognition with larger domain discrepancy. It consists of seven object categories from four domains (Photo, Art Paintings, Cartoon, and Sketches datasets) and the domain discrepancy among different datasets is more severe than VLCS, making it more challenging.

2 Network Architecture and Implementation Details

Our framework is flexible and one can use different network backbones as the feature Encoder. We utilized a fully-connected layer with $31$-dimensional output as the self-supervised auxiliary classification layer following the setting in for a fair comparison. To enable the momentum metric learning, we further employed a fully-connected layer with $128$ output channels following the Encoder part and added an L2 normalization layer to normalize the feature representation $v$ of each sample. The MuEncoder has the same network architecture as the Encoder, and the weight of MuEncoder was initialized with the same weight as Encoder. We followed the previous works in the literature and employed the leave-one-domain-out cross-validation strategy to produce the experiment results, i.e., we take turns to choose each domain for testing, and train a network model with the remaining three domains.

We implemented our framework with the PyTorch library on one NVIDIA TITAN Xp GPU. Our framework was optimized with the SGD optimizer. We totally trained 100 epochs, and the batch size was 128. The learning rate was set as 0.001 and decreased to 0.0001 after 80 epochs. We empirically set the margin of the triplet loss as $2$. We also adopted the same on-the-fly data augmentation as JiGen , which includes random cropping, horizontal flipping, and jitter.

3 Results on VLCS Dataset

We followed the same experiment setting in previous work to train and evaluate our method. The extrinsic metric learning and intrinsic self-supervised learning was developed upon the “FC7” features of AlexNet pretrained on ImageNet . We set the size of the memory bank as $1024$ and the number of negatives $K$ in the triplet loss Eq. (4) as $256$. The hyper-parameters $\alpha$, $\beta$, and $\gamma$ in total objective function Eq. (7) were set as $1$, $0.1$, and $0.05$, respectively. For our results, we report the average performance and standard deviation over three independent runs.

We compare our method with other nine previous state-of-the-art methods. D-MTAE utilized the multi-task auto-encoders to learn robust features across domains. CIDDG was a conditional invariant adversarial network that learns the domain-invariant representations under distribution constraints. CCSA exploited a Siamese network to learn a discriminative embedding subspace with distribution distances and similarities. DBADG developed a low-rank parametrized CNN model for domain generalization. MMD-AAE aligned the distribution through an adversarial auto-encoder by Maximum Mean Discrepancy. MLDG was a meta-learning method by simulating train/test domain shift during training. Epi-FCR was an episodic training method. JiGen solved a jigsaw puzzle auxiliary task based on self-supervision. MASF employed a meta-learning based strategy with two complementary losses for encoder regularization. Moreover, we include the Within domain performance of all the datasets as a comparison to reveal the performance drop due to domain discrepancy. We trained Within domain using a supervised way with training and test images from the same domain.

The comparison results with the above methods are shown in Table 1. It is observed that our EISNet achieves the best performance on both Caltech and Sun datasets and comparable results on PASCAL VOC and LabelMe datasets. Overall, EISNet achieves an average accuracy of $74.67\%$ over four domains, outperforming the previous state-of-the-art method MASF . Our method also outperforms JiGen on three domains and achieves comparable results on the remaining PASCAL VOC domain, demonstrating that utilizing extrinsic relationship supervision can further improve the network generalization ability.

4 Results on PACS Dataset

To show the effectiveness of our framework under different network backbones on PACS dataset, we evaluate our method with three different backbones: AlexNet, ResNet-18, and ResNet-50 . The size of memory bank was set as $1024$ and $K$ in the triplet loss Eq. (4) was set as $256$. The hyper-parameters in total objective function Eq. (7) were set as $1$, $0.5$, and $0.7$ for $\alpha$, $\beta$, and $\gamma$, respectively. For our results, we also report the average performance and standard deviation over three independent runs.

Table 2 summarizes the experimental results developed with AlexNet backbone. We compare our methods with eight other methods that achieved previous best results on this benchmark dataset. MetaReg utilized a novel classifier regularization in the meta-learning framework. As we can observe from Table 2, by simultaneously utilizing momentum metric learning and intrinsic self-supervision for images across different source domains, our method achieves the best performance on three datasets. Across all domains, our method achieves an average accuracy of $75.86\%$, setting a new state-of-the-art performance.

We also compare our method with baseline method (DeepAll) and the state-of-the-art method MASF using ResNet-18 and ResNet-50 backbones in Table 3. In the ResNet-50 experiment, we reduce the batch size to 64 to fit the limited GPU memory. The DeepAll method is trained with all the source domains without any specific network design. As shown in Table 3, our method consistently outperforms MASF about 1.11% and 3.17% on average accuracy with ResNet-18 and ResNet-50 backbone, respectively. This indicates that our designed framework is very general and can be migrated to different network backbones. Note that the improvement over MASF is more obvious with a deeper network backbone, showing that our proposed algorithm is more beneficial for domain generalization with deeper feature extractors.

5 Analysis of Our Method

We conduct extensive analysis of our method. Firstly, we investigate the effectiveness of extrinsic and intrinsic supervision using ResNet-50 backbone on PACS dataset, and the experimental results are illustrated in Table 4. The Extrinsic supervision indicates that the momentum metric learning is used, while Intrinsic supervision denotes that the auxiliary self-supervision loss is optimized. The method without these two supervisions is the baseline model, which is the same with DeepAll results in Table 3. From the results in Table 4, we observe that each supervision plays an important role in our framework. Specifically, equipping the extrinsic supervision into the baseline model yields about 2.99% average accuracy improvement. Meanwhile, we also achieve 2.73% average accuracy improvement over the baseline model by incorporating intrinsic self-supervision of the images. By combing extrinsic and intrinsic supervision, performance is further improved across all settings, indicating these two supervisions are complementary.

We then analyze five key components in our framework, that is a) the number of different negative samples $K$ in momentum metric learning, b) the effectiveness of momentum update coefficient $\delta$, c) the effectiveness of hard negative selector, d) the size of memory bank $m$, and e) time cost. All below comparison experiments are implemented with AlexNet backbone on the PACS benchmark.

The number of negative samples $K$ is a key parameter of our designed $K$-hard negative selector in momentum metric learning. We investigate the network performance under different options. We select six $K$ values at different magnitudes, which are 1, 8, 64, 128, 256, and 512. The Sketch dataset results are shown in Fig. 3 (a), We can observe that a large number of negative samples would lead to better results in general and the network generates the best result with $K=256$. However, the performance drops drastically if we set $K=512$, demonstrating that too large $K$ will produce a burden on the metric distance calculation and make the network difficult to learn.

The momentum update coefficient $\delta$ is important to control the feature consistency among different batches of embedded features in the memory bank. We show the accuracy with different momentum coefficient $\delta$ in Fig. 3 (b). It is observed that the network performs well when $\delta$ is relatively large, i.e., 0.999. A small coefficient would degrade the network performance, suggesting that a slow updating MuEncoder is beneficial to the feature consistency.

To validate the effectiveness of $K$-hard negative selector in our proposed metric learning, we compare our proposed $K$-hard negative selector with original random triplet selector and semi-hard negative selector. The Sketch dataset results are shown in Table 5. Equipped with semi-hard negative selector, the accuracy improves $2.70\%$. By selecting more negative pairs from the memory bank, we obtain the accuracy of $70.25\%$, demonstrating the effectiveness of the proposed $K$-hard negative selector.

The size of memory bank $m$ can be adjusted according to different tasks. Here, we show the results of four different settings with the number of negatives changing as well in Table 6. In general, our method is able to generate better results with a large memory bank size and negative samples. However, a too large memory bank will increase the burden to calculate the pair-wise distance in triplet loss. Therefore, we need to balance the accuracy and computation burden.

Apart from the performance improvement over other methods, our method has much lower computation cost. Under the same server setting (one TITAN XP GPU) and AlexNet backbone, our method only takes 1.5 hours to train the network on PACS dataset, while the total training time of the state-of-the-art MASF is about 17 hours. Therefore, our method could save more than $91\%$ time cost on training phase.

We also employ t-SNE to analyze the feature level discrimination of our method and the visualization results are shown in Fig. 4. Compared with the feature extracted from the ImageNet pre-trained network, the distance between different class clusters in our method becomes evident, indicating that equipped with our proposed extrinsic and intrinsic supervision, the model is able to learn more discriminative features among different object categories regardless domains.

Conclusions

We have presented a multi-task learning paradigm to learn how to generalize across domains for domain generalization. The main idea is to learn a feature embedding simultaneously from the extrinsic relationship of different images and the intrinsic self-supervised constraint within the single image. We design an effective and efficient momentum metric learning module to facilitate compact feature learning. Extensive experimental results on two public benchmark datasets demonstrate that our proposed method is able to learn discriminative yet transferable feature, which lead to state-of-the-art performance for domain generalization. Moreover, our proposed framework is flexible and can be migrated to various network backbones. Acknowledgments. We thank anonymous reviewers for the comments and suggestions. The work described in this paper was supported in parts by the following grants: Key-Area Research and Development Program of Guangdong Province, China (2020B010165004), Hong Kong Innovation and Technology Fund (Project No. ITS/426/17FP and ITS/311/18FP), and National Natural Science Foundation of China with Project No. U1813204.

References