Object Region Mining with Adversarial Erasing: A Simple Classification to Semantic Segmentation Approach

Introduction

Deep neural networks (DNNs) have achieved remarkable success on semantic segmentation tasks , arguably benefiting from available resources of pixel-level annotated masks. However, collecting a large amount of accurate pixel-level annotation for training semantic segmentation networks on new image sets is labor intensive and inevitably requires substantial financial investments. To relieve the demand for the expensive pixel-level image annotations, weakly-supervised approaches provide some promising solutions.

Among various levels of weak supervision information, the simplest and most efficient one that can be collected for training semantic segmentation models is the image-level annotation . However, to train a well-performing semantic segmentation model given only such image-level annotation is rather challenging – one obstacle is how to accurately assign image-level labels to corresponding pixels of training images such that DNN-based approaches can learn to segment images end-to-end. To establish the desired label-pixel correspondence, some approaches are developed that can be categorized as proposal-based and classification-based. The proposal-based methods often exhaustedly examine each proposal to generate pixel-wise masks, which are quite time-consuming. In contrast, the classification-based methods provide much more efficient alternatives. Those methods employ a classification model to select the regions that are most discriminative for the classification target and employ the regions as pixel-level supervision for semantic segmentation learning. However, object classification models usually identify and rely on a small and sparse discriminative region (as highlighted in the heatmaps produced by the classification network shown in Figure 1 (a)) from the object of interest. It deviates from requirement of the segmentation task that needs to localize dense, interior and integral regions for pixel-wise inference. Such deviation makes the main obstacle to adapting classification models for solving segmentation problems and harms the segmentation results. To address this issue, we propose a novel adversarial erasing (AE) approach that is able to drive a classification network to learn integral object regions progressively. The AE approach can be viewed as establishing a line of competitors, trying to challenge the classification networks to discover some evidence of a specific category until no supportable evidence is left.

Concretely, we first train an image classification network using the image-level weak supervision information, i.e. the object category annotation. The classification network is applied to localize the most discriminative region within an image for inferring the object category. We then erase the discovered region from the image to breakdown the performance of the classification network. To remedy the performance drop, the classification network needs to localize another discriminative region for classifying the image correctly. With such repetitive adversarial erasing operation, the classification network is able to mine other discriminative regions belonging to the object of interest. The process is illustrated by an example in Figure 1 (a), in which head is the most discriminative part for classifying the “dog” image. After erasing head and re-training the classification network, another discriminative part body would pop out. Repeating such adversarial erasing can localize increasingly discriminative regions diagnostic for image category until no more informative region left. Finally, the erased regions are merged to form a pixel-level semantic segmentation mask that can be used for training a segmentation model. More visualization examples are shown in Figure 1 (b).

However, the AE approach may miss some object-related regions and introduce some noise due to less attention on boundaries. To exploit those ignored object-related regions as well as alleviate noise, we further propose a complementary online prohibitive segmentation learning (PSL) approach to work with AE together to discover more complete object regions and learn better semantic segmentation models. In particular, PSL uses the predicted image-level classification confidences to modulate the corresponding category-specific response maps and form them into an auxiliary segmentation mask, which can be updated in an online manner. Those category-specific segmentation maps with low classification confidences are prohibited for contributing to the formed supervision mask, thus noise can be reduced effectively.

To sum up, our main contributions are three-fold:

We propose a new AE approach to effectively adapt an image classification network to continuously mining and expanding target object regions, and it eventually produces contiguous object segmentation masks that are usable for training segmentation models.

We propose an online PSL method to utilize image-level classification confidences to reduce noise within the supervision mask and achieve better training of the segmentation network, collaborating with AE.

Our work achieves the mIoU 55.0% and 55.7% on val and test of the PASCAL VOC segmentation benchmark respectively, which are the new state-of-the-arts.

Related Work

To reduce the burden of pixel-level annotation, various weakly-supervised methods have been proposed for learning to perform semantic segmentation with coarser annotations. For example, Papandreou et al. and Dai et al. proposed to estimate segmentation using annotated bounding boxes. More recently, Lin et al. employed scribbles as supervision for semantic segmentation. In , the required supervised information is further relaxed to instance points. All these annotations can be considered much simpler than pixel-level annotation.

Some works propose to train the segmentation models by only using image-level labels, which is the simplest supervision for training semantic segmentation models. Among those works, Pinheiro et al. and Pathak et al. proposed to utilize multiple instance learning (MIL) to train the models for segmentation. Pathak et al. introduced a constrained CNN model to address this problem. Papandreou et al. adopted an alternative training procedure based on the Expectation-Maximization algorithm to dynamically predict semantic foreground and background pixels. However, the performance of those methods is not satisfactory. Recently, some new approaches are proposed to further improve the performance of this challenging task. In particular, Wei et al. presented a simple to complex learning method, in which an initial segmentation model is trained with simple images using saliency maps for supervision. Then, samples of increasing complexity are progressively included to further enhance the ability of the segmentation model. In , three kinds of loss functions, i.e. seeding, expansion and constrain-to-boundary, are proposed and integrated into a unified framework to train the segmentation network. Both and our work propose to localize object cues according to classification networks. However, Kolesnikov et al. can only obtain small and sparse object-related seeds for supervision. In contrast, the proposed AE approach is able to mine dense object-related regions, which can provide richer supervised information for learning to perform semantic segmentation. In addition, Qi et al. proposed an augmented feedback method, in which GrabCut and object proposals are employed to generate pixel-level annotations for supervision. To the best of our knowledge, Qi et al. achieved the state-of-the-art mIoU scores using Selective Search (52.7%) and MCG (55.5%) segmentation proposals on the PASCAL VOC benchmark. However, note that MCG has been trained from PASCAL train images with pixel-level annotations, and thus the corresponding results of are obtained by using stronger supervision inherently.

Classification to Semantic Segmentation

The proposed classification to semantic segmentation approach includes two novel components, i.e. object region mining with AE and online PSL for semantic segmentation.

To address the problem that classification networks are only responsive to small and sparse discriminative regions, we propose the AE approach for localizing and expanding object regions progressively. As shown in Figure 2, the AE iteratively performs two operations: learning a classification network for localizing the object discriminative regions and adversarially erasing the discovered regions. In particular, the classification network is initialized based on the DeepLab-CRF-LargeFOV model. Global average pooling is applied on conv7 and the generated representations pass through a fully-connected layer for predicting classification. In the first operation, we train the classification network by minimizing squared label prediction loss as suggested by . In the second operation of performing erasing, we first produce the heatmap for each image-level label using the classification activation maps (CAM) method . Then, the discriminative object regions are obtained by applying a hard threshold to the heatmap. We erase the mined region from training images by replacing its internal pixels by the mean pixel values of all the training images. The processed image with erased regions is then fed into the next classification learning iteration. As the discriminative regions have been removed and no longer contribute to the classification prediction, the classification network is naturally driven to discover new object discriminative regions for maintaining its classification accuracy level. We repeat the classification learning and the AE process for several times until the network cannot well converge on the produced training images, i.e. no more discriminative regions left for performing reasonably good classification.

We now explain the AE process more formally. Suppose the training set $\mathcal{I}=\{(I_{i},\mathcal{O}_{i})\}_{i=1}^{N}$ includes $N$ images and $\mathcal{F}=\{F_{i}\}_{i=1}^{N}$ represents the mined object regions by AE. We iteratively produce the object regions $F_{i,t}$ for each training image $I_{i,t}$ with the classification model $M_{t}$ at the $t^{th}$ learning step. Denote $\mathcal{C}$ as the set of object categories and CAM$(\cdot)$ as the operation of heatmap generation. Thus, the $c^{th}$ heatmap $H_{i,t}^{c}$ of $I_{i,t}$, in which $c\in\mathcal{O}_{i}$ and $\mathcal{O}_{i}\subseteq\mathcal{C}$ is the image-level label set of $I_{i,t}$, can be obtained according to CAM$(I_{i,t},M_{t},c)$. To enforce the classification network to expand object regions from $I_{i,t}$, we erase the pixels whose values on $H_{i,t}^{c}$ are larger than $\delta$. Then, $\mathcal{F}$ is obtained through the procedure summarized in Algorithm 1.

Beyond mining foreground object regions, finding background localization cues is also crucial for training the segmentation network. Motivated by , we use the saliency detection technology to produce the saliency maps of training images. Based on the generated saliency maps, the regions whose pixels are with low saliency values are selected as background. Suppose $B_{i}$ denotes the selected background regions of $I_{i}$. We can obtain the segmentation masks $\mathcal{S}=\{S_{i}\}_{i=1}^{N}$, where $S_{i}=F_{i}\cup B_{i}$. We ignore three kinds of pixels for producing $\mathcal{S}$: 1) those erased foreground regions of different categories which are in conflict; 2) those low-saliency pixels which lie within the object regions identified by AE; 3) those pixels that are not assigned semantic labels. One example of the segmentation mask generation process is demonstrated in Figure 3 (a). “black” and “purple” regions refer to the background and the object, respectively.

2 Online PSL for Semantic Segmentation

The proposed AE approach provides the initial segmentation mask for each training image that can be used for training segmentation networks. However, some object-related or background-related pixels may be missed (as those “blue” pixels on the AE outputs shown in Figure 3 (a)). In addition, semantic labels of some labeled pixels may be noisy due to the limitation of AE on capturing boundary details. To exploit those pixels unlabeled by AE for training and gain robustness to falsely labeled pixels, we propose an online Prohibitive Segmentation Learning (PSL) approach to further learn to perform semantic segmentation upon the masks provided by AE. The online PSL exploits image classification results to identify reliable category-wise segmentation maps and form them into a less noisy auxiliary supervision map, offering auxiliary information to the AE output. PSL updates the produced auxiliary segmentation map along with training of the segmentation networks in an online manner and produces increasingly more reliable auxiliary supervision. As shown in Figure 3 (b), the proposed PSL builds a framework that includes two branches, one for classification and the other for semantic segmentation. In particular, PSL uses the squared loss as the optimization objective for the classification branch, whose produced classification confidences are used by PSL to weight the corresponding category-specific segmentation score maps. With the help of classification results, the online PSL is able to integrate the multi-category segmentation maps into an auxiliary segmentation mask and provides supervision in addition to the AE output. With PSL, those segmentation maps corresponding to categories with low classification confidences are prohibited from contributing to the auxiliary segmentation map. Thus, noise from those irrelevant categories can be effectively alleviated.

Formally, denote the set of semantic labels for segmentation task as $\mathcal{C}^{seg}$ and the image-specific label set for a given image $I$ as $\mathcal{O}^{seg}$, in which background category is included. During each training epoch, we denote the image-level prediction from the classification branch as $\bm{v}$. Suppose $S$ is the segmentation mask produced by AE. The online PSL exploits the image prediction over $\mathcal{C}^{seg}$ to train a segmentation network $f(I;\theta)$ parameterized by $\theta$, which predicts the pixel-wise probability of each label $c\in\mathcal{C}^{seg}$ at every location $u$ of the image plane $f_{u,c}(I,\theta)$. To produce the additional segmentation mask $\hat{S}$ for training the segmentation network, PSL uses $\bm{v}$ to weight foreground category segmentation score maps as shown in Figure 3 (b). With this prohibitive operation, large response values from negative score maps can be suppressed by multiplying a small classification category score. Meanwhile, the score maps of dominant categories (i.e. the corresponding objects that occupy a large area of the image) can also be enhanced. Denote the weighting operator as $\otimes$, and $\hat{S}$ is then produced by

Here the appended element 1 is for weighting the background category. Suppose $S_{c}$ and $\hat{S}_{c}$ represent the pixels annotated with category $c$. The cross-entropy loss used for noise-prohibitive semantic segmentation is formulated as

With online training, the segmentation ablity of the network is progressively improved, which can produce increasingly more accurate $\hat{S}$ for supervising the later training process.

During the testing process, we take a more strict pohibitive policy for those categories with low classification confidences. In particular, we set those classification confidences that are smaller than $p$ to zero and keep others unchanged, and apply them to weight the predicted segmentation score maps and produce the final segmentation result.

Experiments

Dataset and Evaluation Metrics We evaluate our proposed approach on the PASCAL VOC 2012 segmentation benchmark dataset , which has 20 object categories and one background category. This dataset is split into three subsets: training (train, 1,464 images), validation (val, 1,449 images) and testing (test, 1,456 images). Following the common practice , we increase the number of training images to 10,582 by image augmentation. In our experiments, only image-level labels are utilized for training. The performance is evaluated in terms of pixel IoU averaged on 21 categories. Experimental analysis of the proposed approach is conducted on the val set. We compare our method with other state-of-the-arts on both val and test sets. The result on the test set is obtained by submitting the predicted results to the official PASCAL VOC evaluation server.

Training/Testing Settings We adopt DeepLab-CRF-LargeFOV from as the basic network for the classification network and segmentation network in AE and PSL, whose parameters are initialized by the VGG-16 pre-trained on ImageNet . We use a mini-batch size of 30 images where patches of 321 $\times$ 321 pixels are randomly cropped from images for training the network. We follow the training procedure in at this stage. The initial learning rate is 0.001 (0.01 for the last layer) and decreased by a factor of 10 after 6 epochs. Training terminates after 15 epochs. Both two networks are trained on NVIDIA GeForce TITAN X GPU with 12GB memory. We use DeepLab code in our experiments, which is implemented based on the publicly available Caffe framework .

For each step of AE, those pixels belonging to top 20% of the largest value (a fraction suggested by ) in the heatmap are erased, which are then considered as foreground object regions. We use saliency maps from to produce the background localization cues. For those images belonging to indoor scenes (e.g. sofa or table), we adopt the normalized saliency value 0.06 as the threshold to obtain background localization cues (i.e. pixels whose saliency values are smaller than 0.06 are considered as background) in case some objects were wrongly assigned to background. For the images from other categories, the threshold is set as 0.12. For the testing phase of semantic segmentation, the prohibited threshold $p$ is empirically set as 0.1 and CRF is utilized for post processing.

2 Comparisons with State-of-the-arts

We make extensive comparisons with state-of-the-art weakly-supervised semantic segmentation solutions with different levels of annotations, including scribbles, bounding boxes, spots and image-level labels. Results of those methods as well as ours on PASCAL VOC val are summarized in Table 1. Among the baselines, MIL-* , STC and TransferNet use more images (700K, 50K and 70K) for training. All the other methods are based on 10K training images and built on top of the VGG16 model.

From the result, we can observe that our proposed approach outperforms all the other works using image-level labels and point annotation for weak supervision. In particular, AF-MCG achieves the second best performance among the baselines only using image-level labels. However, the MCG generator is trained in a fully-supervised way on PASCAL VOC, thus the corresponding result, i.e. AF-MCG , implicitly makes use of stronger supervision. Thus, with the Selective Search segments, the performance of AF-SS drops by 1.7%. Furthermore, GrabCut is also employed by AF-* to refine the segmentation masks for supervision, which is usually time consuming for training. In contrast, the proposed AE approach is very simple and convenient to carry out for object region mining. In addition, the online PSL is also effective and efficient for training the semantic segmentation network. Compared with those methods using image-level labels for supervision, the proposed AE-PSL improves upon the best performance by over 2.4%. Besides, our approach also outperforms those methods that implicitly use pixel-level supervision by over 0.7%. Additional comparison among these approaches on PASCAL VOC test is shown in Table 2. It can be seen that our method achieves the new state-of-the-art for this challenging task on a competitive benchmark.

Figure 4 shows some successful segmentations, indicating that our method can produce accurate results even for some complex images. One typical failure case is given in the bottom row of Figure 4. This case may be well addressed with a better erasing strategy such as using low level visual features (e.g. color and texture) to refine and extend erasing regions.

3 Ablation Analysis

With the AE approach, discriminative object regions are adversarially erased step by step. Therefore, it is expected that the loss values of the classification networks at the convergence of training across different AE steps would progressively increase as more discriminative regions are absent for training the classification networks. Figure 5 (a) shows the comparison of the classification training loss curves for different AE steps. It can be observed that the loss value at convergence of training with original images is around 0.05. By performing the AE for multiple steps, the converged loss value slightly increases (AE-step2: $\sim$0.08, AE-step3: $\sim$0.1) compared with that of the AE-step1. This demonstrates that AE removes regions with a descending discriminative ability. By continuing to perform the AE for more steps to remove more regions, the classification network only converges to one that provides a training loss as large as $\sim$0.15. This demonstrates no more useful regions are left for obtaining a good classification network, due to over erasing. over erasing may introduce many true negative regions into the mined foreground object regions and hampers learning segmentation. Some failure cases caused by over erasing are shown in Figure 5 (b). In the case where most object regions are removed from the training images, the classification network has to rely on some contextual regions to recognize the categories. These regions are true negative ones and detrimental for the segmentation network training. To prevent contamination from negative regions, we only integrate those discriminative regions mined from the first three steps into the final segmentation masks.

For quantitatively understanding the contribution of each AE step, Table 3 shows the comparison of mIoU scores using foreground regions merged from varying $k\ (k=1,2,3,4)$ AE steps for training the segmentation network based on DeepLab-CRF-LargeFOV. We can observe that the performance indeed increases as more foreground object regions are added since the segmentation network gets denser supervision. However, after performing four AE steps, the performance drops by 2.1% due to the over erasing as explained above. Some visualization examples are shown in Figure 6, including training images (top row), heatmaps produced by different AE steps and the finally erased regions (bottom row). We can observe that the AE approach effectively drives the classification network to localize different discriminative object regions. For example, regions covering the body of the right-most instance of “cow” shown in the last column are first localized. By erasing this instance, another two instances on the left side are then discovered. We also conduct experiments on VOC 2012 test set using object regions merged from the first three AE steps. The mIoU score is 52.8%, which outperforms all those methods (as indicated in Table 2) only using image-level labels for supervision.

3.2 Online PSL for Semantic Segmentation

We now proceed to evaluate the online PSL and investigate how it benefits the AE approach by discovering auxiliary information. We report the performance of online PSL in Table 4, where “w/o PSL” and “w/ PSL” denote the result of vanilla DeepLab-CRF-LargeFOV and the proposed PSL method for training, respectively. We can observe that PSL improves the performance by 3.2% compared with “w/o PSL”, , demonstrating the significant effectiveness of PSL providing additional useful segmentation supervision.

Besides, we perform one more iterative training step on PSL to improve the segmentation results. In particular, we first employ the trained segmentation model from AE and PSL to segment training images. Then, the predicted segmentation masks are used as supervision for training the segmentation network for another round. As shown in Table 4, the performance provided by this extra training (denoted as w/ PSL++) is further improved from 54.1% to 55.0%. The improvement benefits from the operation of performing CRF on the predicted segmentation masks of training images. After one round training on top of CRF results, the segmentation network has been trained well. We do not observe further performance increase by performing additional training, as no new supervision information is fed in.

Furthermore, we also examine the effectiveness of our testing strategy where the prohibited threshold is empirically set as 0.1. We utilize ground-truth image-level labels as classification confidences to weight the predicted segmentation score maps (note this is different from the prohibitive information imposed in the training stage). The result is 56.1% (“w/ PSL + GT”), which is only 1.1% better than “w/ PSL ++”. Note that “w/ PSL + GT” actually provides an upper bound on the achievable performance as the score maps are filtered by the ground-truth category annotations and “w/ PSL ++” performs very closely to this upper bound.

PSL adopts the on-the-fly output of the classification network to re-weight segmentation score maps. Another choice for such classification information is the ground-truth annotation. We also consider the case of using ground-truth image-level labels for prohibiting during the training stage and evaluate the performance. However, using ground-truth information leads to performance drop of 0.6% compared with our proposed PSL design. This is because PSL effectively exploits the information about object scale that is beneficial for generating more accurate segmentation masks (i.e. categories of large objects are preferred with high classification scores compared with those of small objects). Simply using 0-1 ground-truth annotation ignores the scale and performs worse. We also investigate how PSL performs without using image-level classification confidences and find that the performance drops 1%. This clearly validates the effectiveness of the proposed online PSL approach using image-level classification information.

Conclusion

We proposed an adversarial erasing approach to effectively adapt a classification network to progressively discovering and expanding object discriminative regions. The discovered regions are used as pixel-level supervision for training the segmentation network. This approach provides a simple and effective solution to the weakly-supervised segmentation problems. Moreover, we proposed an online prohibitive segmentation learning method, which shows to be effective for mining auxiliary information to AE. Indeed, the PSL method can aid any other weakly-supervised methods. This work paves a new direction of adversarial erasing for achieving weakly-supervised semantic segmentation. In the future, we plan to develop more effective strategies for improving adversarial erasing, such as erasing each training image with adaptive steps or integrating adversarial erasing and PSL into a more unified framework.

Acknowledgment

The work is partially supported by the National Key Research and Development of China (No. 2016YFB0800404), National University of Singapore startup grant R-263-000-C08-133, Ministry of Education of Singapore AcRF Tier One grant R-263-000-C21-112 and the National Natural Science Foundation of China (No. 61532005).

References