Penalizing Top Performers: Conservative Loss for Semantic Segmentation Adaptation

Introduction

Deep convolutional neural networks have brought impressive advances to the state of the art across a multitude of tasks in computer vision . At the same time, these significant leaps require a large amount of labeled data. For some pixel-level tasks, e.g., semantic segmentation, obtaining a fine-grained label is expensive and time-consuming. In , they report that it takes more than 90 minutes for manually labeling a single image. Recent advances in Computer Graphics offer an alternative solution to address the data issue. In , they automatically capture both images and fine-grained labels from GTAV game with the speed faster than human in several orders of magnitude.

However, models trained on the synthetic data fail to perform well on the real-world images. The main reason is the shift between training and test domains . In the presence of the domain shift, the model trained on the synthetic data often tends to be biased towards the source domain (synthetic images), making them incapable to generalize to the target domain (real images).

Traditional approaches for domain adaptation mainly focus on the image classification task, which can be summarized as two lines: (1) minimizing the distance between the source and target distributions ; (2) explicitly ensuring that two distributions close to each other by adversarial learning . Existing works used the similar idea, i.e., gradient reversal layer, to our proposed loss in the domain adaptation for image classification, which was achieved by multiplying a negative scalar during the backpropagation. Those drawbacks limit the gradient reversal layer to generalize to the segmentation adaptation.

Semantic segmentation provides pixel-level label for input image, which carries more dense and structured information than image classification, and thus making its domain adaptation difficult. Hence, the domain adaptation techniques in the classification task which focus on sparsely high-level features do not translate well to the segmentation adaptation . Few works have explored the domain adaptation for segmentation . Orthogonal to those works focusing on manipulating the data statistics or applying the curriculum learning to adaptation, we propose the novel Conservative Loss to realize it without introducing extra computational overhead.

We observe that with training step goes by, the performance on the target domain first rises and then falls. We show the trends of mIoU on the experiment of synthetic (GTAV data ) to real (Cityscapes data ) segmentation adaptation in Fig 1. It can be observed that the performance on source domain and target domain would not reach the best at the same time because of the domain shift. Since there is no ground truth for target domain during training, it is required to find the saddle point of target domain on the source domain. It is note-worthy that the saddle point for target domain does bias to the best score on the source domain but not reach, which delivers a balance between the discriminativeness and domain-invariant. This phenomenon is consistent with many domain adaptation theories . Therefore, we focus on learning representations with two following characteristics which are: (i) discriminative for semantic segmentation on the source domain (corresponding to the ‘first rises’) and (ii) invariant to the change of domains.

In this paper, this is achieved by training with the Conservative Loss in an adversarial framework. The Conservative Loss is extremely simple. It holds two attributes corresponding to the properties of desired representations. First, when the probability of ground truth label on the source domain is low, the Conservative Loss enforces the network to learn more discriminative features via gradient descent, which corresponds to the first property of discriminativeness. Second, when the probability of ground truth label is much high, our loss penalizes this case by giving a negative value, which prevents the model from biasing to source domain training data further increasing the generalization capability. This corresponds to the second property of domain-invariant. Our loss function can be seen to seek the optimal parameters that deliver a saddle point of those two objectives. Furthermore, the generative adversarial network (GAN) is also introduced to our model. Unlike some works where they apply the feature-level discriminator, we utilize the GAN to further supplement the domain alignment by enforcing reconstructed images to be indistinguishable for the discriminator.

We conduct extensive experiments on synthetic to real segmentation adaptation. The proposed method considerably improves over previous state-of-the-art and achieves 9.3 points of mIoU gain on Synthia to Cityscapes experiment without introducing any extra computational overhead during evaluation. Ablation studies verify the effect of different components to our performance and give more insights into properties of our Conservative Loss. More discussions and visualization demonstrate the Conservative Loss has good flexibility rather than limiting to a fixed instantiation.

Related Work

Semantic Segmentation. Semantic segmentation is a highly active field, which is a task of assigning object label to each pixel of image. With the surge of deep segmentation model , most recent top-performing methods are built on the CNNs .

Huge amount of human effort is required to annotate the fined-grained semantic segmentation ground truth. According to , it did take about 60 minutes to manually segment each image. On the contrary, collecting data from video games such as GTAV is much faster and cheaper compared with the human annotator. For example, extracted 24,966 GTAV images with annotations within 49 hours by using a GPU parallel method. However, it is hard to apply the model trained on the synthetic image to the real-world image because of their discrepant data distributions.

Domain Adaptation. Many machine learning methods rely on the assumption that the training and test data are in the same distribution. However, it is often the case that there exists some discrepancies , which leads to significant performance drop on the test data. Domain adaptation aims to alleviate the impact of the discrepancy between training and test data.

Domain Adaptation for Image Classification. Existing works on domain adaptation mostly focus on image classification problem. Conventional methods include Maximum Mean Discrepancy (MMD) , geodesic flow kernel , sub-space alignment , asymmetric metric learning , etc. Recently, domain adaptation approaches aim to improve the adaptability of deep neural networks .

Domain Adaptation for Semantic Segmentation. Much less attention has been given to domain adaptation for semantic segmentation task. The pioneering work in this task is , which combines the global and local alignment methods with a domain adversarial training. Another work applies the curriculum learning to solve the domain adaptation from easy to hard. In , they propose an unsupervised learning to adapt road scene segmenters across different cities. In , they perform output space adaptation at feature level by an adversarial module. Unlike them constraining the distribution or the output of the network , we propose the Conservative Loss to naturally seek the discriminative and domain-invariant representations.

Adversarial Learning. Recently, Generative Adversarial Network (GAN) has raised great attention. Some works extend this framework for domain adaptation. CoGAN achieves the domain adaptation by generating cross-domain instances. Domain adversarial neural networks consider adversarial training for suppressing domain biases. In , they incorporate adversarial discriminative setting to help mitigate performance degradation. In our work, we also incorporate the GAN into our model, whose discriminator drives the source image towards the target one for promoting domain alignment.

Methodology

As presented above, the key to realize unsupervised domain adaptation is the discriminative and domain-invariant representations. The Conservative Loss is proposed to penalize the extreme cases and its goal is to deliver a balance between the discriminative and the domain-invariant representations. Furthermore, we introduce the generative adversarial networks to align the source and target embedding. Below, we first describe the framework of our model and its network blocks. Then, the Conservative Loss and its background are presented in details. Finally, the alternative optimization is provided.

Our framework is illustrated in Figure 2. In our setting, there are two domains: source domain (image and label) and target domain (image only). Our framework aims to achieve good performance on the target domain by applying the model trained on the source domain.

Our model consists of two major parts, i.e., GAN and Segmentation part. The GAN aims to align the source and target embedding. More specifically, the generator and discriminator are playing a minimax game , in which the generator takes source embedding as input and generates the target-like image to fool the discriminator, while the discriminator tries to classify the reconstructed image . The segmentation part can be seen as a regular segmentation model. For each part, the detailed components are shown in the following:

The encoder(E) performs the feature embedding given source or target image, whose architecture is a fully convolutional network. The generator(G) reconstructs the image based on the embedding. The discriminator(D) does classify the reconstructed images as real or fake. S is the pixel-wise classifier.

The GAN consists of encoder, generator and discriminator.

The segmentation part consists of encoder and pixel-level classifier. Note that the encoder does work in both GAN and Segmentation.

The detailed architecture of generators and discriminators is described in the supplementary material because of the limited page space.

2 Background

In this section, we briefly introduce the theory of domain adaptation and present its relation to our proposed loss.

Many theoretical analyses of domain adaptation have offered a upper bound on the expected risks of target domain, which depends on its source domain error (test-time) and the divergence between two domains. Formally,

where $\mathcal{S}$ and $\mathcal{T}$ denote the source domain and target domain, respectively. $\epsilon$ is the expected risk. $d$ is the domain divergence, which has different notions, for example $\mathcal{H}$-divergence . $\mathcal{C}$ is a constant term.

It can be observed that two terms $\epsilon_{\mathcal{S}}$ and $d(\mathcal{S},\mathcal{T})$ closely relate to the properties in the desired representations. The first term $\epsilon_{\mathcal{S}}$ indicates that the model should produce discriminative representations for getting smaller expected risks on the source domain, which corresponds to the first property of discriminativeness. The second term $d(\mathcal{S},\mathcal{T})$ defines the discrepancy distance between two distributions, in which the more similar the representations of both domains are, the smaller it is. This correlates with the second property of domain-invariant. More theoretical analyses are shown in the supplementary material.

3 Conservative Loss

As explained above, the desired representations should be discriminative for the main task on source domain and possess good generalization ability rather than getting into the overfitting. We thus propose the Conservative Loss for the semantic segmentation on the source domain, which carries the two following properties:

When the probability of ground truth class is low, the loss function gives a positive value, which enables the network to learn a more discriminative feature by using gradient descent method.

When the probability is high, the loss function delivers the negative value, which makes the network avoid the bias towards the source domain via the gradient ascend further learning the better generalization.

In the following, we have raised two lemmas to analysis the appealing property of our Conservative Loss.

Lemma 1: The objective function of domain adaptation system contains a saddle point, which relates to the zero point of Conservative Loss.

As the pipeline in Fig 2 shown, the full objective consists of two parts, including the loss $\mathcal{L}_{seg,p_{t}}^{s}$ for Segmentation and the loss $\mathcal{L}_{GAN}$ for GAN . The sign of $\mathcal{L}_{seg,p_{t}}^{s}$ dynamically depends on $p_{t}$. When $p_{t}$ is much high, the negative value leads to the gradient ascend for escaping the bias to source domain. Otherwise, the positive value makes the features discriminative. It can be seen that our loss balances the two objectives (discriminativeness and domain-invariant) that shape the representations during learning, and its zero point acts as the saddle point. More details are shown in the supplementary material.

Lemma 2: Our loss encourages the moderate examples in large range, which makes the overall optimization more stable.

In practice we use a $\lambda$-balanced variant of the Conservative Loss:

As our experiments will show, different balanced factors $\lambda$ yield slightly different performance. While in our main experiments we use the Conservative Loss defined above, its exact form is not crucial. In Section 4.5 we offer other forms of our loss which also maintain the two properties, and experimental results demonstrate that they can also be effective.

4 Model Objective

Our full objective is to alternatively update the three network blocks, i.e., discriminators(D), generators(G) and encoder(E). Note that S is a pixel-level classifier which has no learnable parameters in our model. Hence, the objective contains three terms: $\mathcal{L}_{D}$, $\mathcal{L}_{G}$ and $\mathcal{L}_{E}$. We then explain the various losses used in our method and describe the alternative optimization scheme.

Adversarial Loss. Inheriting from GAN , we apply the adversarial losses which are derived from the discriminator to all three blocks. We term them as $\mathcal{L}_{\text{GAN,D}},\mathcal{L}_{\text{GAN,G}}$ and $\mathcal{L}_{\text{GAN,E}}$. For each adversarial loss it consists of two parts, i.e., $\mathcal{L}_{GAN}^{s}$ for the source image and $\mathcal{L}_{GAN}^{t}$ for the target image. Thus we can obtain the adversarial loss by $\mathcal{L}_{GAN}=\mathcal{L}_{GAN}^{s}+\mathcal{L}_{GAN}^{t}$. It is noted that for the encoder, the adversarial loss does a cross-domain update (i.e., classifying the image as real or fake from source domain to target domain and vice versa), which enforces the network to generate similar embeddings for two domains.

Reconstructed Loss. The generator performs the image reconstruction. We use L1 distance as $\mathcal{L}_{rec}$ because L1 encourages less blurring.

Segmentation Loss. As Section 3.3 introduced, the Conservative Loss is applied to the semantic segmentation in the domain adaptation setting.

During training, we iteratively optimize all three learnable parts (Encoder, Generator and Discriminator). During inference, only the encoder and segmentation classifier are used to produce the results on target domain. The alternating update scheme is described as following:

Update discriminators: the overall loss is $\mathcal{L}_{D}=\mathcal{L}_{GAN,D}$.

Update generators: the loss involves the adversarial loss and reconstructed loss. The overall loss is $\mathcal{L}_{G}=\mathcal{L}_{GAN,G}+\mathcal{L}_{rec}$.

Update encoder: since the encoder does work in both two components, i.e., GAN and Segmentation, the overall loss is a combination of several losses, including adversarial loss and segmentation loss on source domain; $\mathcal{L}_{E}=\mathcal{L}_{GAN,E}+\mathcal{L}_{seg}^{s}$.

Experiments

Following previous works , we use GTAV or Synthia dataset as the source domain with pixel-level labels, and we use Cityscapes dataset as the target domain. We briefly introduce the datasets as following:

GTAV has 24,966 urban scene images rendered by the gaming engine GTAV. The semantic categories are compatible with the Cityscapes dataset. We take the whole GTAV dataset with labels as the source domain data.

Synthia is a large dataset which contains different video sequences rendered from a virtual city. We take SYNTHIA-RAND-CITYSCAPES as the source domain data which provides 9,400 images from all the sequences with Cityscape-compatible annotations. Inheriting from existing methods , we take 16 common object categories for the evaluation.

Cityscapes is a real-world image dataset focused on the urban scene, which consists of 2,975 images in training set and 500 images for validation. The resolution of images is 2048 $\times$ 1024 and 19 semantic categories are provided with pixel-level labels. We take the unlabeled training set as the target domain data. The adaptation results are reported on the validation set.

2 Training Setup

Warm Start. In our experiments, two different training strategies are employed, which are cold start and warm start. The cold start is that the whole model is trained by using the Conservative Loss from scratch. The warm start indicates the model is trained by first using cross entropy loss and then using our Conservative Loss. Many works demonstrate that the warm start strategy to gradient update provides a more stable training compared with cold start. As the ablation study will show, the warm start performs better than the cold start. In the next domain adaptation experiments, the model is trained using warm start strategy for fairness.

3 Results

In this section, we provide a quantitative evaluation by performing two adaptation experiments, i.e., from GTAV to Cityscapes and from Synthia to Cityscapes. We compare our method with several existing models, including FCNWild , CDA and . FCNWild applies the dilated network as the backbone and the base model of is the FCN8s-VGG19 . Tsai et al. adopts adversarial learning in the output space to perform feature adaptation. The detailed results of each category are available in the supplementary material.

GTAV $\rightarrow$ Cityscapes. For a fairness, the result is evaluated over the 19 common classes. From Table 2 shown, our proposed method achieves the best performance (mIoU=38.1), which has 9.2 points higher than and 11 points higher than . Due to the different experimental settings and backbone network (baseline method also mentions the difference), our own baseline performance is higher than other methods. However, the highlight is the performance gain. We can find that the proposed method yields an improvement of 8.1 points higher than 6.0 in and 6.6 in .

Synthia $\rightarrow$ Cityscapes. We report the results of mIoU in Table 2. It is noted that reported the results on Synthia to Cityscapes adaptation with only 13 object categories (excluding wall, fense and pole). We also report this results as the mIoU-2. Our proposed model achieves a mIoU of 34.2, and more importantly our model obtains a 9.3 points of performance gain which is higher than the performance gain of (7.0) and (2.8). Compared with on 13 categories, our method also achieves the better performance. In particular, our model does not use any additional scene parsing data except the source domain and target domain data, while the uses another dataset, i.e., PASCAL CONTEXT dataset, to obtain the superpixel label.

4 Ablation Study

In this section, we perform the thorough ablation experiments, including experiments with different components, different factors in the Conservative Loss and different training strategies. Those experiments demonstrate different contributions of components and provide more insights of our method.

Effect of different components. In this experiment, we show how each component in our model affects the final performance. We consider several cases as following: (1): the baseline model, which contains only the base segmentation model (FCN8s in our model) and is trained using source data only. (2) the FCN8s and GAN component, which consists of base model and GAN and is trained using both source data and target data with the cross entropy loss. (3) the full model, which involves three parts, i.e., base model, GAN and Conservative Loss. We perform the ablation experiments on GTAV$\rightarrow$Cityscapes setting.

The results of ablation study are shown in Table 4. It can be observed that each component plays an important role in performance improvement. More specifically, our full model achieves the best results and obtains 8.1 points performance gain. The GAN part also gets 4.4 performance gain compared with FCN8s+CE. Note that the GAN component could introduce the unlabeled target domain data into the whole model, so the Conservative Loss is applied based on the GAN and there is no variant of FCN8s+CL.

Effect of $a$ and $\lambda$ in the Conservative Loss. In this part, we design the ablation experiments for $a$ and $\lambda$ in the Conservative Loss. As shown in Equation 2, $a$ is the base of logarithm and denotes the intersection point with x-axis. $\lambda$ is a balanced factor. We show the impacts of different $a$ and $\lambda$ in Table 4.

In Table 5, it can be observed that the Conservative Loss with cold start outperforms with a large margin (6.3 points). The warm start performs better than the cold start because it enables the network to train stably.

5 Discussion

In this section, we design several experiments to verify the capability of the proposed method. We show the effect of adaptation on distribution to measure how domain gap is reduced in the feature level. We compared with several classification losses and homogeneous losses to show its superiority and flexibility.

To verify the effect of adaptation on the distribution, we use t-SNE to visualize feature distributions in Figure 4. 100 images are randomly selected from each domain and for each image the features from last convolutional layer (the channel size equals to class categories.) are extracted. We compare the distributions of our model with FCN8s (No adaptation). Four categories are sampled to display for a clearly visual effect. We observe that with the adaptation applying, the distance between two domains with same class becomes closer and the discrepancy between different classes also gets clear.

Comparison with other classification losses. In this experiment, we compare the Conservative Loss to Cross Entropy Loss and Focal Loss . The Cross Entropy Loss is given by $\text{CE}(p_{t})=-\log(p_{t})$, which is plotted in Figure 5 with green line. To ensure fairness, we utilize the $\alpha$-balanced Focal Loss $\text{FL}(p_{t})=-\alpha_{t}(1-p_{t})^{2}\log(p_{t})$ and warm start in the experiment of Focal Loss, and apply $\alpha_{t}=5$ by using a cross-validation.

From the right table in Figure 5, it can be observed that the Focal Loss obtains a better performance compared with the cross entropy loss because it focuses learning on hard negative examples. However, in the domain adaptation, the domain-invariant representations are crucial to achieve good adaptation performance. The Conservative Loss does enable the network to be insensitive to domain changes by punishing the extreme cases. It can be seen that the Conservative Loss yields higher result (38.1), and obtains more performance gain (3.7) than the Focal Loss (1.4) based on the cross entropy loss.

Effect of homogeneous losses. As shown in Section 3.3, the Conservative Loss has two properties: (1) when the $p_{t}$ is low, the Conservative Loss enforces the network to learn discriminative features. (2) when the $p_{t}$ is high, the loss enables the network to learn domain invariant features by gradient ascend method, which aims to penalize the extremely good cases. There are several losses that also maintain these two properties, for example the cubic equation. In this experiment, we propose several homogeneous losses to verify the effect of these two properties, which are given by:

We apply the adaptation experiment on GTAV $\rightarrow$ Cityscapes to verify their capabilities. The results are reported in Table 6. In order to ensure fairness, all experiments are performed based on the warm start and those hyper-parameters ($\lambda_{1},\lambda_{2},\alpha,\beta$) are chosen by using the cross-validation. We can observe that all homogeneous losses perform better than the cross entropy loss (34.4) and Focal Loss (35.8). Therefore, we can find that the exact form of the Conservative Loss is not crucial, and several homogeneous losses also yield comparable results and perform better than cross entropy loss and Focal Loss. Generally, we expect any loss function with similar properties as Conservative Loss to be equally effective.

Conclusion

In this paper, we have proposed a novel loss, the Conservative Loss, for the semantic segmentation adaptation. To enforce the network to learn the discriminative and domain-invariant representations, our loss combines the gradient descend and gradient ascend method together, in which it penalizes the extreme cases and encourages moderate cases. We further introduce the adversarial networks to our full model for supplementing the domain alignment. Extensive experiments demonstrate our model achieves state-of-the-art. Exploratory experiments show that the Conservative Loss has high flexibility without limiting to exact form.

Acknowledgments This work is partially supported by the Big Data Collaboration Research grant from SenseTime Group (CUHK Agreement No. TS1610626).

References