Self-Supervised Difference Detection for Weakly-Supervised Semantic Segmentation

Introduction

Semantic segmentation is a promising image recognition technology that enables the detailed analysis of images for various practical applications. However, semantic segmentation methods require training data with pixel-level annotation, which is costly to create. On the other hand, image-level annotation is much easier to obtain than pixel-level annotation. In recent years, various weakly-supervised semantic segmentation (hereinafter WSS) methods that required only image-level annotation have been proposed to resolve the annotation problems. However, there is still a large performance gap between fully-supervised and weakly-supervised methods.

In weakly-supervised segmentation methods, visualization-based approaches have been widely adopted. The visualization results highlight the regions that contributed to the classification, and we can roughly estimate the regions of the target objects by visualization. Class Activation Map (CAM) is a standard method to visualize the classification results. However, the visualization results do not always match actual segmentation results; therefore, it is usually necessary to consider the mapping from the visualization results to the semantic segmentation in weakly-supervised segmentation. Conditional Random Field (CRF) is widely used as a mapping function. CRF is a method for optimizing the probability distribution to be fitted to the edge of regions by using color and position information as features. The iterative approach for the learning segmentation models proposed by Wei et al. is a versatile approach for improving weakly supervised segmentation results. In this method, we generate pseudo pixel-level labels under weakly supervised conditions, and we train a segmentation model with the pseudo labels. Subsequently, we generate pseudo pixel-level labels from the outputs of the trained segmentation model, and we re-train a new segmentation model using the generated pseudo labels. Wei et al. showed that repeating this process absorbed outliers and gradually improved the accuracy. These methods can be regarded as mapping functions that bring inputs closer to the segmentation. However, the mapping functions of these methods do not guarantee any improvement in the accuracy of the semantic segmentation; therefore, the mapping results contain noise. In this paper, the mapping functions that make the above inputs close to the segmentation are treated as supervision containing noise, and we propose a robust learning method for such noise.

In this paper, we denote the information used as the inputs of the mapping functions as knowledge, and we consider the supervision containing the noise as advice. The supervision for fully supervised learning that allows one-to-one mapping is teacher. We assume that the advice provides supervision, which includes some correct and incorrect information. To make effective use of the information obtained from this advice, it is necessary to select useful information. In this paper, we regard the regions where opinions differ between knowledge and advice as difference. Since difference in the two segmentation masks can be obtained by simple processing without annotation, it is a kind of self-supervised learning to train a model, which predicts difference. Self-supervised learning is a pretext task as a form of indirect supervision. For example, as notable works, colorization and predicting the patch ordering have been proposed.

Inferring difference in knowledge and advice from knowledge leads to predicting the advisor’s advice in advance. In predicting advice, there are predictable advice and unpredictable advice. Certain advice can be easily inferred because many similar samples are included during training. Here, we assumed that advice contains a sufficient number of good information, and predictable information can be considered to be useful information. Based on this idea, we propose a method for selecting information by finding the true information in advice that can be predicted from the inference results of difference detection. Fig.1 shows the concept of the proposed approach.

In this paper, we demonstrate that the proposed Self-Supervised Difference Detection (SSDD) module can be used in both the seed generation stage and the training stage of fully supervised segmentation. In the seed generation stage, we refine the CRF results for pixel-level semantic affinity (PSA) by using the SSDD module. In the training stage, we introduce two SSDD modules inside the training loop of a fully supervised segmentation network. In the experiments, we demonstrate the effectiveness of the SSDD modules in both stages. In particular, the SSDD modules greatly boosted the performance of the WSS on the PASCAL visual object classes (VOC) 2012 dataset, and achieved new state-of-the-art. To summarize it, our contributions are as follows:

We propose an SSDD module, which estimates the noise of the mapping functions of the weakly supervised segmentation and select useful information.

We show that the SSDD modules can be effectively applied to both the seed generation stage and the training stage of a fully supervised segmentation model.

We obtained the best results on the PASCAL VOC 2012 dataset with 64.9% mean IoU on the val set and 65.5% on the test set.

Related Works

In this section, we review related research on CNN-based WSS methods by classifying them into several types.

Visualization In the early works of CNN-based WSS, visualization-based methods were studied. The pixels that contributed to the classification were correlated to the regions of the target objects; therefore, the visualization methods can be used as segmentation methods under weakly supervised settings. Zeiler et al. showed that the derivatives obtained by back-propagation from the CNN models trained for classification tasks highlight the region of a target object in an image. Simonyan et al. used derivatives such as the GrabCut seeds and extended the visualization method to the WSS method. They also demonstrated that the regions of multi-class objects could also be captured by the difference in class-specific derivatives . Oquab et al. visualized the attention region by the forwarding process using activation and trained a classification model with large input images by using global max pooling. After this approach, several derived methods employing global pooling were also proposed . In particular, CAM has been widely adopted in recent weakly supervised segmentation methods.

Region refinement for WSS results using CRF In general, the segmentation results based on fully convolutional neural network (FCN) tend to output ambiguous outlines. CRF can refine the ambiguous outlines using low-level features such as the pixel colors. Chen et al. and Pathak et al. adopted CRF as a post-processing method for region refinement and demonstrated the effectiveness of the CRF for WSS. Kolesnikov et al. proposed the use of CRF during the training of a semantic segmentation model. Ahn et al. proposed a method to learn pixel-level similarity from the CRF results, and apply a random walk-based region refinement, which achieved the best results on the PASCAL VOC 2012 dataset. CRF plays an important role to improve the accuracy of weakly supervised segmentation. Furthermore, various researches employed the CRF for refining the coarse segmentation masks . However, CRF does not guarantee any improvement in the mean intersection over union (IoU) score, and it often degrades the segmentation masks and the scores. Therefore, we focus on preventing a segmentation mask from being degraded by applying CRF. We estimate the confidence maps of both the initial mask and the mask after CRF post-processing, and we integrate both masks based on the estimated confidence maps.

Training fully supervised segmentation model under weakly supervised setting Certain researchers trained a fully supervised semantic segmentation (hereinafter FSS) model under a weakly supervised setting. First, Papandreou et al. proposed MIL-FCN, which trained a fully supervised semantic segmentation model with a global max-pooling loss using only image-level labels. Wei et al. proposed a novel approach to train an FSS model using pixel-level labels obtained by saliency maps . This method is simple, and the obtained results are impressive. Wei et al. also demonstrated that the outputs of the trained semantic segmentation model could be used as a new pixel-level annotation for re-training, and the re-trained FSS model achieved better results than the original model.

Generating pixel-level labels during training of an FSS model Constrained convolutional neural network (CCNN) and EM-adopt generated pixel-level labels during training using class labels and outputs of the segmentation model. In both the studies similar constraints were made for generating pixel-level labels to obtain better results. They set the ratios of the foreground and the background in an image and generated pixel-level labels within the ratio. Wei et al. proposed an online prohibitive segmentation learning (PSL). They generated pixel-level seed labels of training samples before the first training of an FSS model and re-generated pixel-level labels using the outputs of the segmentation model and the classification results. The semantic segmentation model was trained by both the pixel-level labels, and they achieved good performance without costly manual pixel-level annotation. We expected that the pixel-level seed labels would play the role of the constraint. Huang et al. proposed deep seeded region growing (DSRG), which is a method to expand the seed region during training. Before training, the authors prepared pixel-level seed labels that had unlabeled regions for unconsidered pixels. In this research, we proposed new constraints for generating pixel-level labels during the training of the FSS model. We trained an FSS model and the difference detection model in an end-to-end manner. Then, we interpolated a few pixel-level seed labels, that had different regions in the newly generated pixel-level labels and these labels could also be predicted by the difference detection model.

WSS methods using additional information A few recent weakly supervised approaches achieved high accuracy by using additional annotations for image-level labels. Researchers have proposed the bounding box annotation for WSS , and they showed that the bounding box annotation substantially boosted performance. As weaker additional annotation, point annotation and scribble annotation were also proposed . Saleh et al. proposed an approach to check the generated initial masks by minimal additional supervision by human visions. Motion segmentation of videos as additional training information for weakly supervised segmentation has also been proposed . There are also reports that web images were helpful for improving the weakly supervised segmentation accuracy . Recently, fully supervised saliency methods are being widely used for detecting the background regions, and certain researchers have reported that this approach could substantially boost performance . Region proposal methods trained with fully supervised foreground masks such as MCG have also been used in . Hu et al. used instance-level saliency maps for WSS. The concept of saliency can be used and helpful in various situation; however, the fully supervised saliency model was affected by its training data domain, which may cause negative effects on applications. WSS methods without saliency maps are also beneficial. In this paper, we do not use any additional information, and we use only PASCAL VOC images with image-level labels and CNN models pre-trained with ImageNet images and their image-level labels.

Method

There was no supervision for the mapping functions of segmentation in the weakly supervised setting; therefore, it was necessary to consider a mapping for bringing the input close to the better segmentation results by using a method that incorporated human knowledge. In this paper, we propose a method for selecting useful information from the results of the mapping functions by treating the results as supervision containing noise. We define the inputs of the mapping functions as knowledge, and the mapped results as advice. We predict the regions of differences between knowledge and advice, and we call this as the difference detection task. Using the inference results, we select the information of the advice.

where $S$ is a set of pixels of the input spaces, and $J()$ is assumed to be a function that returns a loss for the binary cross entropy.

Note that the parameters of the embedding function $\theta_{e}$ are independent of the optimization of $\theta_{d}$. The training of DD-Net is self-supervised; therefore, neither special annotation nor additional data are needed.

2 Self-supervised difference detection module

Here, $bias$ is a hyper parameter for a threshold of the selection obtained by the difference detection, and it is also an enhanced value for the categories in the presence labels of the input image. The refined masks $m^{D}$ obtained from $m^{K}$ and $m^{A}$ are defined by the following expression:

We denote this processing flow for generating new segmentation mask as an SSDD module in the after notation.

Introducing SSDD modules into the processing flow of WSS

In this section, we explain how to use SSDD modules in the processing flow of WSS. The proposed method can be adapted to various cases by applying inputs of the mapping function as knowledge and the results of the mapping function as advice. The processing flow that we adopted in this paper consists of two stages: the seed generation stage with static region refinement and the training stage of a segmentation model with dynamic region refinement. In the first stage, we adapted the proposed method by applying the results of PSA as knowledge and its CRF results as advice (Sec.4.1). In the second stage, we adapted the proposed method by applying the results of the first stage (Sec.4.1) as knowledge, and the outputs of the segmentation models trained by the masks were applied as advice (Sec.4.2).

PSA is a method to propagate label responses to nearby areas that belong to the same semantic entity. Though PSA employs CRF for the refinement of the segmentation mask, CRF often fails to improve the segmentation masks; in fact, it degrades the masks. In this section, we refine the outputs of CRF in PSA by using the proposed SSDD module. We illustrate the processing flow of the first seed generation stage in Fig.4. Note that we omitted the input of the given image to an SSDD module for the sake of simplifying in the figure.

We denote an input image as $x$; the probability maps obtained by PSA are denoted as $p^{K0}=PSA(x;\theta_{psa})$, and its CRF results are denoted as $p^{A0}$. We obtain the segmentation masks $(m^{K0},m^{A0})$ from the probability maps $(p^{K0},p^{A0})$ by taking the argument of the maximum of the presence labels including a background category. We computed the loss of the DD-Net as follows:

The proposed method is not effective when either of the segmentation masks or both of them do not have the correct labels. These cases are not only meaningless for the proposed refinement approach, but they may also harm the training of the DD-Net. We define the bad training samples by simple processing based on the difference in the number of the class-specific pixels, and we exclude them from the training.

In this work, we also train the embedding function by training a segmentation network with $m^{K0}$ to obtain good representation for the inputs of high-level features and low-level features:

where $S_{k}^{m}$ is a set of locations that belong to the class $k$ on the mask $m$; $h_{u}^{k}$ is the conditional probability of observing any label $k$ at any location $u\in\{1,2,...,n\}$; and ${\mathcal{C}}$ is a set of class labels. $\theta_{e0}$ are parameters of embedding functions and $\theta_{base}$ are parameters for the segmentation branch. The training of $\theta_{e0}$ is independent of $\theta_{d0}$. The final loss function for the static region refinement using the difference detection is as follows:

After training, we integrate the masks $(m^{K0},m^{A0})$ and obtain the integrated masks $m^{D0}$ using the SSDD module with the trained parameter $\theta_{d0}$ as follows:

2 Training stage of a fully supervised segmentation model with a dynamic region refinement

When we train a fully supervised semantic segmentation model with pixel-level seed labels, the accuracy of the seed labels directly effects the performance of the segmentation. The performance gain is expected by replacing the seed labels to better the pixel-level labels during training. In this study, we propose a novel approach to constrain the interpolation of the seed labels during the training of a segmentation model. The idea of the constraint is to limit the interpolation of seed labels only to predictable regions of difference detection between newly generated pixel-level labels and seed labels.

In practice, we interpolate the pixel-level seed labels in two steps of each iteration as shown in Fig.5. Note that “SegNet” in the figure does not represent a specific segmentation network; it represents any fully supervised segmentation network. In the first step, for an input image $x$, we obtain the outputs of the segmentation model $p^{K1}=Seg(e(x);\theta_{main})$ and its CRF outputs $p^{A1}$. We obtain the segmentation masks $(m^{K1},m^{A1})$ from the probability maps $(p^{K1},p^{A1})$ by taking the argument of the maximum of the presence labels including a background category. Then, we obtain the refined pixel-level labels $m^{D1}$ by applying the proposed refinement method as follows: $m^{D1}=SSDD(e(x),m^{K1},m^{A1};\theta_{d1})$. In the second step, we apply the proposed method to the seed labels $m^{D0}$ and to the mask $m^{D1}$ obtained in the first step. The further refined mask $m^{D2}$ is obtained by $m^{D2}=SSDD(e(x),m^{D0},m^{D1};\theta_{d2})$. We generate the mask $m^{D2}$ in each iteration and train the segmentation model using the generated mask $m^{D2}$. We train the semantic segmentation model with the generated mask $m^{D2}$ as follows:

The loss of DD-Net for $m^{A1}$ and $m^{K1}$ is as follows:

In the second stage, we also exclude the bad samples (as done in Sec.static) based on the change ratio of pixels because the proposed method is not effective if the input segmentation masks do not have correct regions.

We explain how to train the DD-Net for ($m^{D0}$,$m^{D1}$). The masks ($m^{K1},m^{A1},m^{D1}$) depend on the outputs of the segmentation model $Seg(e(x),\theta_{main})$. Therefore, if the learning of the segmentation model falls into a local minimum, the masks will become meaningless; all the pixels become background pixels or single foreground pixels. In this case, the inference results of the difference detection is also always constant, that is, ($D^{K}=1,d^{A}=1,d^{A}=d^{K}$), and Eq.(3) becomes $w=bias$. To escape from this local minimum, we create a new branch of a segmentation model and use it for learning the difference detection between $m^{D0}$ and $m^{D1}$. Assume that the mask $m^{sub}$ was obtained from outputs of the branch of the new segmentation model $p^{sub}=Seg(e(x);\theta_{sub})$. In the training of difference detection, we trained the network to learn the differences among ($m^{D0}$, $m^{sub}$) and ($m^{sub}$, $m^{D1}$) as follows:

If $m^{sub}$ is the output, which is halfway between $m^{D0}$ and $m^{D1}$, the replacement of the training samples will let the segmentation model exit from the situation ($d^{K}=1,d^{A}=1,d^{A}=d^{K}$), and the inference results of the difference detection will predict the regions that correlate with the difference between $m^{D0}$ and $m^{D1}$. We train the parameters $\theta_{sub}$ from the following loss to achieve the outputs that are halfway between $m^{D0}$ and $m^{D1}$.

where $\alpha$ is a hyper parameter of the mixing ratio of $m^{D0}$ and $m^{D1}$ .

The final loss function of the proposed dynamic region refinement method is calculated as follows:

Experiments

We evaluated the proposed methods using the PASCAL VOC 2012 data. The PASCAL VOC 2012 segmentation dataset has 1464 training images, 1449 validation images, and 1456 test images including 20 class pixel-level labels and image-level labels. Similar to the methodology followed by , we used the augmented PASCAL VOC training data provided by as well, wherein the training image number was 10,582. For evaluation, we used an IoU metric, which is the official evaluation metric in the PASCAL VOC segmentation task. For calculating the mean IoU on the val and test sets, we used the official evaluation server. We compared the best performance of our method with the state-of-the-art methods on both the val and test sets.

Our experiments are heavily based on the previous research . For the generating results of PSA results, we used implementations and trained parameters provided by the authors that are publicly available. We followed the methodology of and set hyperparameters that gave the best performance. For the CRF parameters, we used the default settings provided by . For the semantic segmentation model, we used a ResNet-38 model, which had almost the same architecture as that in . The only difference was in the last upsampling rate; in the paper on PSA, the authors set the upsampling rate to 8, while we set the rate to 2 for reducing the computational cost of CRF. The input image size was 448 for training, and the test images and the output feature map size before the upsampling was 56. In the DD-Net, we used features obtained from the segmentation model before the last layer as the high-level features $e^{h}$ and the features obtained before the second pooling layer as the low level features $e^{l}$. These feature map sizes were adjusted to 112 by 112 using the simple linear interpolation approach. We initialized the parameters of the segmentation models by using parameters trained with the PASCAL VOC images and their image-level labels with a pre-trained model using ImageNet, which was also provided in . The codes provided by did not include the training and test code for the segmentation models; therefore, we implemented our own codes. In the original paper on PSA, though the authors optimized the segmentation models by Adam; however, the performance was unstable in our re-implementation, and there were several unclear settings. Therefore, we used SGD for training the entire networks. We set an initial learning rate to 1e-3 (1e-2 for initialization without the pre-trained model), and we decreased learning rate with cosine LR ramp down . For the static region refinement, we trained the network with batch sizes of 16 and 10 epochs. For the dynamic region refinement, we trained the network with batch sizes of 8 and 30 epochs. For the data augmentation and inference technique, we carefully followed the methodology used in . We implemented the proposed method using PyTorch. All the networks are trained using four NVIDIA Titan X PASCAL. We will open the results of the proposed method and training codeshttps://github.com/shimoda-uec/ssdd.

2 Analysis of static region refinement

In the proposed method, we used fully connected CRF with the same parameter settings as those for PSA , ($w_{g}=3$, $w_{rgb}=10$,$\theta_{\alpha}=80$, $\theta_{\beta}=13$, $\theta_{\gamma}=3$) in the following kernel potentials: $k(f_{i},f_{j})=w_{g}exp\left(-\frac{|p_{i}-p_{j}|}{2\theta_{\alpha}^{2}}-\frac{|I_{i}-I_{j}|}{2\theta_{\beta}^{2}}\right)+w_{rbg}exp\left(-\frac{|p_{i}-p{j}|^{2}}{2\theta_{\gamma}^{2}}\right)$. To examine the relationship between the CRF params and results, we changed the values of ($w_{g}$, $w_{rgb}$) and evaluated the accuracy. Fig.6 shows a comparison of the proposed static region refinement with the PSA and its CRF results on the training set. The weakening of $w_{rgb}$ decreases the difference only between the CRF and the SSDD+CRF results; therefore the effectiveness of the proposed method reduces. However, the proposed method always indicates a high accuracy. The optimal weights are different for each image, and it is expected to be difficult to search them for each image. We consider that the proposed method realized the improvement of CRF by correcting the partial failure of CRF.

Fig.7 shows the difference detection results and their refined segmentation masks. In the fourth and fifth rows of Fig.7, we show the typical failure cases of the proposed method. The regions of small objects tend to vanish in the CRF, and the DD-Net also learns such tendencies, which causes the failure of the proposed re-refinement method. In the fifth row, both of the input segmentation masks fail to provide segmentation. In such cases, the proposed method is also not effective.

3 Analysis of the whole proposed method

We denote the dynamic region refinement as “SSDD” in all the tables. The score of the SSDD is with the CRF with parameters ($w_{g}=3$, $w_{rgb}=10$) that are default values from the author’s public implementation. We also used the parameters for the CRF during training.

Comparison with PSA Table 1 shows the comparison of the dynamic region refinement method with the PSA. We observe that the proposed method outperforms PSA by more than 3.2 point margins. This clearly proves the effectiveness of the interpolation for the seed labels with the novel constraint by difference detection. The accuracy is greatly improved as compared with the results of the static region refinement because of the increase in the number of good advice by end-to-end learning of the segmentation model, that is, $|S^{A1,T}|>|S^{A0,T}|$.

In Table 1, we also show the gains between the proposed method and PSA for detailed analysis. We obtain over 10% gain on the cat, cow, horse, and sheep classes. Interestingly, all the classes that gave the large gain belonged to the animal category. However, in the potted plant, airplane, and person class objects, it was hard to improve the segmentation mask by using the proposed method. In the proposed method, we considered the precondition that advise, which is a true value, was larger than the value that was not a true value($|S^{A,T}|>|S^{A,F}|$). When this precondition was satisfied, the accuracy of the classes improved. If the precondition was not satisfied, the accuracy did not improve or the accuracy decreased.

Fig.8 shows the examples of the results of re-implementation of PSA, the static region refinement, and the dynamic region refinement. Dynamic region refinement shows more accurate predictions on object location and boundary. The results of the static region refinement are outputs of a segmentation model re-trained with the masks in case of ($w_{g}=3$, $w_{rgb}=10$) in Fig.6. Note that we show the results of before the CRF for detailed comparisons.

Comparison with the state-of-the-art methods Table 3 shows the results of the proposed method and the recent weakly supervised segmentation methods that do not use additional supervisions on the PASCAL VOC 2012 validation data and PASCAL VOC 2012 test data. We observed that our method achieves the highest score as compared with all the existing methods, which use the same types of supervision . The proposed method outperforms the recent previous works on MEFF and TPL by large margins. As discussed earlier, the proposed method also outperforms the current state-of-the-art methods . This result clearly indicates the effectiveness of the proposed method.

Table 3 shows the comparison of the proposed method with a few weakly supervised segmentation methods that employ relatively cheap additional information. Surprisingly, the proposed method also outperforms all the listed weakly supervised segmentation methods. The proposed methods outperformed the following methods: SeeNet , DSRG , MDC , GAIN , and MCOF that employed fully supervised saliency methods. In addition, the score of the proposed method was also better than the results of AISC , which used instance-level saliency map methods. Note that AISC achieved 64.5% on the val set and 65.6% on the test set using an additional 24,000 ImageNet images for training. The score of the proposed method was also higher than the score of Shen et al. , which used 76.7k web images for training. It is not possible to have a completely fair comparison for them because of the difference of the network model, the augmentation technique, the number of iteration epochs, and so on. However, the proposed method demonstrates comparable performance or better performance without any additional training information.

Conclusions

In this paper, we proposed a novel method to refine a segmentation mask from a pair of segmentation masks before and after the refinement process such as the CRF by using the proposed SSDD module. We demonstrated that the proposed method could be used effectively in two stages: the static region refinement in the seed generation stage and the dynamic region refinement in the training stage. In the first stage, we refined the CRF results of PSA by using the SSDD module. In the second stage, we refined the generated semantic segmentation masks by using a fully supervised segmentation model and CRF during the training. We demonstrated that three SSDD modules could greatly boost the performance of WSS and achieve the best results on the PASCAL VOC 2012 dataset over all the weakly supervised methods with and without additional supervision.

Acknowledgements This work was supported by JSPS KAKENHI Grant Number 17J10261, 15H05915, 17H01745, 17H06100 and 19H04929.

References

A.1 Details of the simple decision

In the proposed method, we select advice by inference results of difference detection. The confidence score is calculated from the viewpoint of how close the value of $d^{K}$ to $d^{A}$. In the proposed method, if this difference is large enough, we ignore the advice. Therefore, if the inferences of the difference detection are too easy, the values of $d^{K}$ for advice that is not true become close to $d^{A}$, and the proposed method does not work effectively. In particular, if the inference results of the difference detection are ($d^{K}=1,d^{A}=1,d^{A}=d^{K}$), we cannot distinguish whether the advice belongs to the set of true values $|S^{A,T}|$ or the set of false values $|S^{A,F}|$ based on the results of the difference detection. Therefore, we judge the typical failure examples of $advice$ and excluded them from the training sample so that the differences between $d^{K}$ and $d^{A}$ were large in the inference of the bad advice. To be concrete, when the number of differences in the pixels in each class of mask is obviously large, we assume that the advice has failed. We define the bad training samples as the pair of the masks for the difference detection that satisfies the following equation:

where ${\cal C}$ is a set of image-level label of the input image. We decide the threshold 0.5 empirically.

A.2 Details of the bias in Eq.(3)

In Eq.(3), we use $bias$, which is a kind of hyperparameter. In this section, we discuss this $bias$. We define the $bias$ as follows:

where $\forall c\in\hat{{\cal C}}\hskip 4.2679pt$ satisfy$\hskip 4.2679pt\frac{|S^{m^{A}}_{c}|}{|S^{m^{K}}_{c}|}<0.5\hskip 4.2679pt$and$\hskip 4.2679ptc\in{\cal C}$. $b_{dd}$ is a bias for the difference between knowledge and advice, and $b_{class}$ is a bias for the class category. When the number of differences in the pixels in each class of mask is obviously large, it is assumed that the advice has failed, and to prioritize the label of that class over the results of the difference detection, we use the bias $b_{class}$. We defined the values of $b_{dd}$ and $b_{class}$ by using the grid search.

A.3 Values of hyperparameters

We explore good hyperparameters by a grid search and verify the effect of the hyperparameters. We change the values of the hyperparameters and measure the mean IoU scores. Table A-1 shows the hyper parameter values and the mean IoU scores. The hyperparameters ($b_{dd},b_{class}$) are used in Eq.(17) as the bias values. In $b_{dd}=0.4$, the mean IoU score becomes the maximum value. We also set the bias $b_{class}$ for the missing categories. We observe that the setting $b_{class}=1.0$ achieved a maximum mean IoU. It is expected that the class biases for the missing categories help to the train for robustness. In addition, we also verify the effect of hyperparameters for coefficients of losses in Eq.(11). Though we had expected that the value of $\alpha$ would affect the performance, the hyper parameter was not critical for the change of the mean IoU. The balanced setting, that is, $\alpha=0.5$ showed the best score.

A.4 Detailed comparison with existing works on the PASCAL VOC 2012 val and test sets