A Weakly Supervised Learning Framework for Salient Object Detection via Hybrid Labels
Runmin Cong, Qi Qin, Chen Zhang, Qiuping Jiang, Shiqi Wang, Yao Zhao, Sam Kwong
I Introduction
Salient object detection (SOD) task aims to locate the most attractive and interesting objects or regions from an image, which is consistent with the human visual attention mechanism, and has been applied to image segmentation , object tracking , image enhancement , and other vision tasks. In recent years, fully-supervised SOD models based on deep learning have made great breakthroughs in performance , but these models usually require a large number of pixel-level labels for training, while such labeling costs are obviously very expensive. Therefore, weakly-supervised or unsupervised SOD methods have received increasing attention from both academia and industry, aiming to reduce or get rid of the reliance on the elaborately labeled data. Some related areas such as weakly supervised semantic segmentation , light field SOD , remote sensing SOD , and visual grounding have also been developed.
According to the given labeled data, weakly-supervised/unsupervised SOD methods can be roughly divided into the following categories: (1) Weakly-supervised scribble label supervision as shown in Fig. 1(b), that is, the parts of the foreground and background regions of each training sample are outlined in a scribble way.
(2) Weakly-supervised point label supervision as shown in Fig. 1(c), that is, the foreground and background regions of each training sample are marked with only one point respectively. (3) Weakly-supervised image-level label supervision as shown in Fig. 1(e), that is, only the category of salient object is known during training. (4) Unsupervised coarse label supervision as shown in Fig. 1(f), that is, the saliency map generated by the existing unsupervised traditional methods is used as the label of the training sample. Generally speaking, the weaker the supervision information, the more limited the detection performance. Compared to image-level labels and sparse labels (i.e., scribble and point labels) as supervision information, the information given by coarse label is uncontrollable and may introduce some inevitable distracting noise, so it provides weaker supervision than others. Different from the above-mentioned forms of supervision, we construct a new supervision form for the first time to solve the weakly-supervised SOD task by releasing part of the real labels on the basis of unsupervised labels, called hybrid labels. The hybrid label supervision consists of two parts, that is, a small number of pixel-level real labels and a large number of coarse labels generated by the existing unsupervised SOD models (e.g., one tenth of the fully-supervised pixel-wise annotations). Such weakly supervision form is expected to achieve better detection performance at a smaller annotation cost. But with this kind of supervision, the weakly-supervised SOD task becomes more challenging due to the unreliability of coarse labels and the imbalance between real-labeled and coarse-labeled data. Specifically, on the one hand, the coarse labels are generated by traditional unsupervised methods and necessarily contain a lot of noise and mislabeling. If the network is trained with such labels all the time, the network will gradually become chaotic and disabled, seriously affecting the final performance. On the other hand, the proportion of real labels and coarse labels in the hybrid label setting is severely imbalanced (e.g., 1:9), and the network learning will collapse if the mixed training is directly performed. Therefore, in order to address these issues, we propose a new weakly-supervised SOD framework with hybrid labels from the perspective of pipeline structure and training strategy.
For such a new weakly-supervised learning task set in this paper, the key problems we need to solve are also different from the existing methods, and thus our model framework and technical implementation are also different. To address the problem of unreliable coarse label and imbalanced sample size under this new hybrid supervision, different from the previous single-stage SOD framework , we decouple the weakly-supervised SOD task into two sub-tasks of coarse label refinement and salient object detection, and construct a joint learning framework as shown in Fig. 2(b), consisting of a Refinement Network (R-Net) and a Saliency Network (S-Net). These two networks cooperate with each other and train alternately. To achieve the R-Net, a two-stream encoder-decoder model equipped with Blender with Guidance and Aggregation Mechanisms (BGA) is designed for coarse label refinement, including a saliency-refinement mainstream branch and an RGB-image guidance branch. On the one hand, considering the uncertainty and noise of coarse labels, a separate RGB-image guidance branch is introduced to form the two-stream structure and provide effective guidance information from the raw RGB data. On the other hand, we propose a BGA to achieve two-stage feature decoding, where the guidance stage aims to gain relatively robust baseline performance for mainstream branch by the guidance branch information, and the aggregation stage is to integrate the encoder features, previous decoder features, and global features by considering the roles of different features. The S-Net is a replaceable salient object detection network supervised by the pseudo labelFor clarity, the coarse label in our paper specifically refer to labels generated by unsupervised SOD model and used as input to R-Net. The pseudo label is saliency map obtained by testing with the trained R-Net or S-Net. generated by the current R-Net, with the original RGB image as input and output of the pseudo label for the subsequent round learning of R-Net. By using such a decoupled architecture, not only the negative impact of coarse labels on S-Net can be effectively reduced, but also the number of training samples can be expanded with the refined coarse labels generated by the trained R-Net, thereby enhancing the learning ability of the network.
Besides, in the face of imbalanced training on the hybrid-labeled data, some well-designed training strategies are crucial to guarantee the effectiveness and efficiency of network training. Specifically, we design three ingenious training strategies: (1) Alternate iteration mechanism. In order to ensure that sufficient and effective samples participate in training, we alternately perform iterative training of R-Net and S-Net. The two networks provide better labels for each other. (2) Group-wise incremental mechanism. In order to avoid the imbalance of inputting a large number of pseudo-labeled samples and a small number of real-labeled samples at the same time, we group the training set and gradually increase the amount of data with pseudo labels in each training iteration, thereby stepwise learning the effective feature representations. (3) Credibility verification mechanism. In order to ensure that the two networks can provide credible labels to each other during the iteration process, starting from the second iteration, we design a validation phase on the validation set containing 100 images, and only the best model that satisfies the validation conditions can be used to generate pseudo labels for the corresponding data to participate in the next step of training. In general, the three training strategies constrain the training process from three aspects, i.e., quantity allocation, training method and reliability judgment, so as to achieve the effective training of the network.
The main contributions of this paper mainly lie in three aspects, including task setting, technical framework and training strategy:
For the first time, we launch a new weakly-supervised SOD task based on hybrid labels, with a large number of coarse labels and a small number of real labels as supervision. To this end, we decouple this task into two sub-tasks of coarse label refinement and salient object detection, and design the corresponding R-Net and S-Net. Moreover, our method achieves competitive performance on five widely used benchmark datasets using only one-tenth of the real labels in fully-supervised setting.
We design a BGA in the R-Net to achieve two-stage feature decoding, where the guidance stage is used to introduce the guidance information from the RGB-image guidance branch to guarantee a relatively robust performance baseline, and the aggregation stage is to dynamically integrate different levels of features according to their modification or supplementation roles.
In order to guarantee the effectiveness and efficiency of network training, from the perspective of quantity allocation, training method and reliability judgment, we design the alternate iteration mechanism, group-wise incremental mechanism, and credibility verification mechanism.
II RELATED WORK
Inspired by image semantic segmentation, Zhao et al. proposed a fully supervised model based on CNN to integrate local and global features to predict the saliency map. Wang et al. adopted a recurrent CNN to refine the predicted saliency map step by step. Most of them follow an encoder-decoder architecture similar to FCN , on this basis, in order to obtain more accurate and convincing detection results, the researchers carried out a series of elaborate network designs. Several recent works integrated features in multiple layers of CNN to exploit the context information at different semantic levels. Among them, Hou et al. introduced short connection to the skip-layer structure for capturing fine details. Deng et al. proposed an iterative method to optimize the saliency map, leveraging features generated by deep and shallow layers. Zhang et al. designed an attention guided network that selectively integrates multi-level contextual information in a progressive manner. Wei et al. focused on the feature fusion strategies and proposed a SOD network that equipped with cross feature module and cascaded feedback decoder trained with a new pixel position aware loss. Pang et al. investigated the multi-scale issue in salient object detection and proposed an effective and efficient network with the transformation-interaction-fusion strategy. In the SOD task, for obtaining results with elaborate boundaries, edge-guided or boundary-guided methods have been proposed. Qin et al. proposed a boundary-aware model to segment salient object regions and predict the boundaries simultaneously. Li et al. proposed an edge-guided SOD network to learn the complementarity between salient edge information and salient object information in a single network. Feng et al. designed an attentive feedback network by integrating some feedback network modules to explore the structure of objects better and proposed a new boundary-enhanced loss for learning exquisite boundaries. Wang et al. introduced the salient edge detection module into an essential pyramid attention structure for salient object detection and achieved superior performance.
Although superior performance has been obtained, a fatal problem still exists is that they all require a mass of pixel-level labeled training data. Accordingly, how to obtain satisfactory detection results with fewer annotations has become a topic worth exploring, which also motivates our work.
II-B Weakly supervised salient object detection
Unlike fully-supervised SOD needs a complete pixel-level label for each training sample, weakly-supervised SOD model may utilize simpler labels, e.g., scribble/point/image-level label, as supervision signals to achieve comparable performance. Due to the low cost of labels and considerable prospects, it has received more and more attention. The WSS model is the first weakly-supervised SOD method by using image-level label, which employs a global smooth pooling layer and a foreground inference scheme to make the network generate good prediction results even for unseen categories. Zeng et al. utilized multiple labels (i.e., image-level labels and captions) to train a SOD model, and then a classification network and a caption generation network were designed to predict object class and generate captions, respectively. Zhang et al. used scribble labels as the supervision to train the network, including an auxiliary edge detection task to locate object edges explicitly and a gated structure-aware loss to place constraints on the scope of structure to be recovered. Apart from that, Zhang et al. first applied an unsupervised method to generate coarse labels, then obtained refined saliency maps by modelling the noise in the coarse labels. Zheng et al. first introduced saliency subitizing as the weak supervision and proposed a SOD model with saliency subitizing module and saliency updating module. Yu et al. proposed a local coherence loss to propagate the labels to unlabeled regions based on image features and pixel distance. Piao et al. introduced a new multiple-pseudo-label framework to integrate more comprehensive and accurate saliency cues from multiple labels, avoiding that the generated single label is inevitably affected by adopted refinement algorithms. Gao et al. proposed a point supervised saliency detection model, where an adaptive masked flood filling algorithm is designed to generate pseudo labels, and the transformer-based point-supervised SOD model and a Non-Salient Suppression (NSS) method are used to achieve two-stage saliency map generation and optimization. Yan et al. made the first attempt to achieve SOD by exploiting unsupervised domain adaption from synthetic data, and constructed a synthetic SOD dataset named UDASOD.
Besides, weakly supervised video object segmentation/SOD methods can also provide us with some enlightenment. Zhao et al. proposed the first weakly supervised video salient object detection model based on ”fixation guided scribble annotations”. And some methods used weakly-supervised approaches to video object segmentation by fusing information between different frames . In contrast, Zhou et al. relied only on the current frame image and the corresponding optical flow data to achieve the zero-shot video object segmentation. En et al. performed video object segmentation with the help of saliency information.
In this paper, we construct a new label form for the first time to solve the weakly-supervised SOD task, namely hybrid labels, which only contains one-tenth of the real pixel-wise labelled samples. With the help of the proposed learning framework and training strategies, our method finally achieves encouraging performance.
III PROPOSED METHOD
At first, the hybrid labels used in this paper can be divided into two parts, i.e., a small number of pixel-level real labels and a large number of coarse labels, where the coarse labels are generated by a traditional unsupervised method (e.g., MB ). The overall framework is shown in Fig. 2, consisting of a Refine Network (R-Net) and a Saliency Network (S-Net). The R-Net is designed as a two-stream encoder-decoder architecture that takes original RGB image and coarse label as inputs and outputs updated pseudo-labels (more details will be introduced in Section III-B). The S-Net is a replaceable SOD network that takes the original RGB image as input and the pseudo labels generated by R-Net as supervision signal.
The proposed framework is trained in alternating iterations. For the training process, the first thing to mention is that we divide the training samples into ten groups (note that only the samples in GROUP 1 include real labels) and incrementally load them into the training pool. On the one hand, the alternating iteration strategy makes the quality of pseudo labels continuously be optimized through the cooperation of these two networks. On the other hand, incremental loading of training samples enhances the guiding ability of real labels. As a result, the imbalance between the real label and pseudo label can be alleviated effectively. Specifically, taking the first iteration as an example, we first use the original RGB images and coarse labels of GROUP 1 as inputs to train the R-Net, then predict the corresponding pseudo labels of GROUP 2. Next, we input the GROUP 1 and GROUP 2 into S-Net for network training, then predict the corresponding pseudo labels of GROUP 3. At this point, this round of training is over. The samples in GROUP 1 and GROUP 3 will be used for the next iteration of R-Net training. The iteration is terminated until we run out of data. Ultimately, we only use the trained S-Net for testing and no longer need coarse label input. More details will be introduced in the Training Strategy with Hybrid Labels of Section III-C.
III-B Refinement Network (R-Net)
The R-Net is designed to refine the coarse labels and produce better pseudo labels that can be used for S-Net training. Intuitively, we only need to input coarse labels and corresponding RGB images into the network to achieve the label refinement. But in our setting, the coarse labels are generated by the traditional unsupervised method, which very noisy for some complex scenes, even inferior to the results obtained by the weakly-supervised deep learning methods. In this way, if only the Saliency-Refinement Mainstream Branch (mainstream branch for short) is used to directly refine the label, the difficulty can be imagined. Considering the uncertainty and noise of coarse labels, a separate RGB-Image Guidance Branch (referred to as the Guidance Branch) is introduced into the R-Net to form a two-stream encoding structure, which is used to provide some guidance information to the mainstream branch, such as object localization and integrity, thereby guaranteeing a relatively robust performance baseline. The whole framework of the R-Net is shown in Fig. 3. The encoders of both streams are based on the ResNet-50 to extract the corresponding multi-level features. Then, we propose a Blender with Guidance and Aggregation Mechanisms (BGA) to achieve two-stage feature decoding, as shown on the right side of Fig. 3. The role of the first stage is guidance, that is, to supplement the mainstream branch with the information of the guidance branch, ensuring that the mainstream branch has a relatively robust baseline performance. The role of the second stage is aggregation, that is, to integrate the encoder features, previous decoder features, and global features by considering the roles of different features.
We hope that in the first stage, the RGB branch can provide guidance information (e.g., object localization and integrity) for the mainstream branch, guaranteeing its effective learning and robust performance baseline. The detailed architecture is shown on the top right corner of Fig. 3.
First, in order to ensure that enough saliency information can be transferred to the mainstream branch and mitigate unreliable noise from the coarse label input, we supplement and filter the features in the channel dimension. Specifically, the encoder features of corresponding layers in the two branches are first concatenated for complementation, and then channel attention is used to highlight essential channel features for filtering. This process can be formulated as:
where denote the complementation features after channel attention, is the channel attention operation , and denote the encoder features of the layer in the mainstream branch and guidance branch, respectively, represents the concatenation operation along the channel dimension, and means element-wise multiplication with channel-wise broadcasting.
Secondly, in addition to the direct complement of channel dimensions, the RGB branch can also provide pixel-level spatial guidance information, which can both reinforce important regions and suppress irrelevant noise interference. Specifically, we use the spatial attention to generate the spatial location mask that need to be emphasized from the perspective of RGB information, and use this to update the features of the refinement branch.
where are the final output encoder features of the guidance stage after the spatial attention, is the spatial attention operation , is the element-wise multiplication, and denotes the convolutional layer with the kernel size of .
III-B2 The Aggregation Stage
As mentioned earlier, the second stage is mainly used to realize the fusion of multi-level features, including the encoder features of the corresponding layer generated in the first stage, the global features from the top encoder layer, and the decoder features of the previous layer. In order to implement the aggregation more effectively, we need to analyze the roles of various features. In general, both encoder features and global features should play an auxiliary role to obtain better decoder features in the feature decoding stage. The auxiliary functions can be divided into two aspects: one is to refine the decoder features under the guidance of global information; the other is to supplement the decoder features under the guidance of encoder features.
First, the high-level semantic features from the top encoder layer are crucial for distinguishing salient objects, but as the decoding process proceeds, the semantic constraint will be gradually diluted. Therefore, in order to enforce semantic information throughout the decoding process, we generate the corresponding semantic guidance mask to refine the decoder features of each level. Specifically, we firstly combine the semantic features from two branches and the encoder features generated in the first stage through an importance weighting strategy :
where denote the fused semantic features from two branches, is the learned importance weight that controls the fusion rate of the features of and (more details can be found in ). Then, the fusion features containing global semantic information are activated as a semantic mask, which is used to modify the upsampled decoder features:
where are the modified decoder features of the level, represent the original decoder features of the level, represents the up-sampling operation by bilinear interpolation, and is the sigmoid activation function.
Second, as demonstrated in , the encoder features contain many valuable information that can complement the decoder feature learning, such as the shallower features including rich spatial information to better recover details, etc. Therefore, we further supplement the modified decoder features with the filtered features via the spatial attention mechanism to obtain more comprehensive saliency-related decoder features. This process can be formulated as:
where are the modified decoder features of the level, and is the spatial attention operation .
III-C Training Strategy with Hybrid Labels
Training settings. The pixel-level real labels and coarse labels are given in our training set, in which the coarse labels are only used as the input to the S-Net instead of supervision. At the same time, the pseudo label will be generated as supervision information during the network training process. In the implementation, we randomly select 1,000 samples from the DUTS-TR dataset as the real-labeled training subset, and use the MB method to generate the corresponding coarse labels of all samples in the DUTS-TR dataset (including the 1,000 samples mentioned earlier). As thus, these 1,000 samples (including RGB images, coarse labels, and real labels) can support the first iteration of R-Net training. In order to guarantee the effectiveness and efficiency of network training, we propose three key training mechanisms, including alternate iteration mechanism, group-wise incremental mechanism, and credibility verification mechanism, as illustrated in Fig. 4.
Alternate iteration mechanism. As mentioned earlier, considering that coarse labels may contain a lot of noise, directly training the network under these supervisions will inevitably lead to poor performance. Therefore, we design a R-Net for label correction and a S-Net for salient object detection. In terms of network training, we train these two networks alternately and iteratively, thereby providing better pseudo labels for each other. In detail, the S-Net of the current iteration is trained using the pseudo labels generated by the trained R-Net of the current iteration, and the pseudo labels generated by the trained S-Net are further used for the next iteration of R-Net training. The two networks are trained in an alternating manner until all training samples are traversed, which is called alternate iteration mechanism.
Group-wise incremental mechanism. Another important problem in the weakly-supervised SOD framework with hybrid label is sample imbalance caused by the difference in the number of real-labeled samples and coarse-labeled samples. If the unbalanced training samples are directly used for network training, it will cause ambiguity and unavailability of network learning. Therefore, we propose a group-wise incremental mechanism to avoid network collapse caused by importing a large amount of pseudo-labeled data at a time. Specifically, we divide all training samples (i.e., 1,000 real-labeled samples and 9,000 coarse-labeled samples) into ten groups equally, of which 1,000 samples with real labels are grouped into GROUP 1. In the first iteration, we only load the samples of GROUP 1 to train the R-Net, and then we use the trained R-Net to test the samples in GROUP 2 and obtain the the corresponding pseudo labels. Subsequently, all the samples in GROUP 1 with real labels and GROUP 2 with pseudo labels are used for S-Net training. Finally, the trained S-Net is used to test the samples in GROUP 3, and the generated saliency maps are used as the pseudo labels for the next iteration of R-Net training. At this point, the first iteration is completed. Noteworthy, in order to prevent the model from overfitting to real-labeled data and improve the robustness of the model, we do not load all real-labeled samples into training in the first iteration, but also use an incremental strategy. To be specific, in the first iteration, we select 500 samples from GROUP 1 as the real-labeled sample batch, and the remaining 500 samples are degenerated into contaminated-labeled sample batch through the rotation, cropping, and occlusion operations. In subsequent training iterations, we gradually reduce the number of contaminated-labeled data and increase the amount of real-labeled data. In the second iteration, we still train R-Net first, followed by the S-Net. For the R-Net training, 2,000 samples in GROUP 1 and GROUP 3 are used, where the number of real-labeled samples is increased to 600 and the number of contaminated-labeled samples is decreased to 400. Note that, the corresponding pseudo labels in GROUP 3 are obtained by testing the samples in GROUP 3 using the S-Net trained in the previous iteration. Then, the new trained R-Net is utilized to test the GROUP 4. And the samples in GROUPs 1, 2, 4 are used to train the S-Net again. In order to traverse the entire training dataset (i.e., 10,000 samples), 5 iterations need to be performed, and the training process for other iterations can be analogized. In addition, since both R-Net and S-Net are trained with hybrid labels, in order to ensure the effectiveness and performability of training, we first train the network under the pseudo-labeled samples and then fine-tune the real-labeled samples in a training epoch.
Credibility verification mechanism. The purpose of alternating training of the two networks is to provide better pseudo labels for each other, so we introduce a credibility verification mechanism from the second iteration to ensure the validity of the provided labels. Only when the current model outperforms the previous best model on the validation set including 100 images, we use it to generate pseudo-labels for the corresponding group and participate in the next training step. Taking the validation process of S-Net as an example, in the second iteration, if the MAE score of the newly trained R-Net model using GROUP1 and GROUP3 data on the validation set is smaller than the MAE score MAE refers to the mean absolute error, which represents the error between the prediction and the ground truth, and the smaller the value means the better. of the previous best R-Net model (the R-Net model trained in the first iteration at this time), then we use the current R-Net model to test the GROUP 4 and generate the corresponding pseudo labels, otherwise we use the R-Net model trained in the first iteration to generate the pseudo labels. Validation of other iterations is similar to this process.
III-D Loss Function
Referring to the traditional SOD method, we also use the binary cross-entropy loss as the loss function for R-Net training. As mentioned earlier, in order to reduce noise pollution from pseudo labels and enhance the guidance of real labels, we first train the network under the pseudo-labeled samples and then fine-tune the real-labeled samples in a training epoch. Specifically, we treat these two types of labels differently and change the binary cross-entropy loss to the following form:
where and are all standard BCE losses, but the samples used in the calculation of the two loss functions are different. The loss calculates the BCE loss of samples with real labels, while calculates the BCE loss of samples with pseudo labels. and correspond to the training set with real labels and pseudo labels, respectively. denotes the training sample in the corresponding set, where are the inputs of the R-Net including the RGB image and the corresponding coarse label, and is the real label or pseudo label of the sample. denotes the R-Net, and represents the network parameters of R-Net.
In general, the whole loss of the R-Net consists of the dominant loss on the final prediction and three auxiliary losses on the side outputs generated by the middle three layers of the decoder, which is formulated as:
where indexes the real-labeled data or pseudo-labeled data, and are the hyper-parameters that control the weight of each auxiliary loss, which are set to in experiments.
Since the S-Net in this paper is replaceable, we still follow the loss function of the original paper in the training order of ‘first real labels, then pseudo labels’.
IV EXPERIMENT
Five widely-used salient object detection benchmark datasets are employed to evaluate the entire performance, including:
DUTS dataset contains 10,553 training images (DUTS-TR) and 5,019 testing images (DUTS-TE), with pixel-wise saliency ground truth.
ECSSD dataset consists of real images of complex scenes, containing 1,000 complex images with the corresponding the pixel-wise saliency ground truths.
HKU-IS dataset includes 4,447 challenging images, most of which are low-contrast or have multiple salient objects.
PASCAL-S dataset consists of 850 images from the PASCAL VOC 2010 validation set, with multiple salient objects in the scene.
THUR dataset collects 15,000 images from the Internet and annotates each image with the corresponding pixel-level saliency ground truth.
IV-A2 Evaluation Metrics
We adopt Precision-Recall (PR) curve , max F-measure score , S-measure score , and MAE score as the evaluation metrics. As the PR curve is closer to the upper right corner, the model performance is better. The larger the F-measure and S-measure values, the better the performance, while the MAE score is just the opposite.
IV-A3 Implementation Details
We select the first 1,000 samples in the DUTS-TR dataset as the real-labeled data, providing the pixel-wise real ground truth. Then, we use the MB method to generate the saliency maps for all the images in the DUTS-TR dataset , thereby forming the coarse-labeled set. The validation set includes a total of 100 images from the SOD dataset.
We use the Pytorch toolbox to implement the proposed network and accelerate training by an NVIDIA GeForce RTX 3090 GPU card. We also implement our network by using the MindSpore Lite toolhttps://www.mindspore.cn/. The ResNet-50 is used as the backbone of the R-Net to extract encoding features, with initial parameters loaded from the pre-trained model on ImageNet . The MINet is used as S-Net in our implementation. For the R-Net, the training images are first resized to by uniform resizing and random cropping. All training samples are then augmented using random horizontal flips and rotations. For the S-Net, we directly resize all images to during training and inference, and then apply the same augmentation strategy to R-Net. During training, the R-Net and S-Net are optimized by Adam optimizer with the batch size of 8, momentum of 0.9, and the weight decay of . The initial learning rate is set to , and divided by ten every ten epochs. The overall network is trained for a total of five iterations, and we design the same number of epochs (i.e., 30 epochs) for each iteration of the training process, whether it is R-Net or S-Net. Note that we only use the warm-up strategy in the first iteration. In the R-Net, we need to concatenate the RGB image and coarse label together into the backbone, for a total of 4 channels. Following the setting in , we duplicate the original three channels of the ResNet model and its parameter once to form 6 channels, and then take the first four channels as the input layer of the new model.
IV-B Comparison with State-of-the-arts
We compare the proposed method with other state-of-the-art models, including fully-supervised methods (i.e., PAGR , MLMSNet , CPD , AFNet , BASNet , GCPANet , DGRL , PiCANet , PFAN , and MINet ), weakly-supervised methods (i.e., MSW , WSSA , ASMO , WSS , MFNet , and WSSD ) and unsupervised methods (i.e., SVF and MNL). For a fair comparison, the saliency maps of the different methods are provided by the authors or obtained by running the released code with default parameters.
First of all, the PR curves are shown in Fig. 5. Our method (red dashed line) achieves the best performance in most cases compared to other weakly supervised and unsupervised methods on five common datasets, which is consistent with the quantitative scores reported in Table I. Compared with the unsupervised SVF method on the PASCAL-S dataset, the percentage gain of our method reaches for max F-measure, for S-measure and for MAE score. Compared with the WSSD method with subitizing supervision and MFNet method with image-level supervision on the HKU-IS dataset, the percentage gain reaches and for MAE score.
In addition, our method also achieves more competitive performance against the weakly-supervised SOD models with stronger supervision. For example, compared with the MSW method with a variety of combination supervision labels, the percentage gain of S-measure reaches on the DUTS-TE dataset, and the the percentage gain of max F-measure also wins . For the scribble based weakly-supervised SOD method (e.g., WSSA ), although the proportion of annotations is relatively small, each sample clearly defines the foreground and background regions. That is, the supervision information given by the scribble is perfectly accurate. By contrast, the hybrid labels we use contain coarse labels with a lot of uncertain noise, but our model outperforms the WSSA method overall on all metrics across all datasets. For example, compared with the WSSA on the DUTS-TE dataset, the percentage gains of max F-measure, S-measure, and MAE score reach , , and , respectively. It is worth mentioning that our method catches up or even surpasses some fully-supervised methods on certain datasets (e.g., THUR dataset). In our proposed framework, we choose the MINet as our S-Net for training, and achieve the original performance of using only of the original training set. Of course, there is still a lot of room for improvement in the performance.
IV-B2 Qualitative Comparison
Some visual comparisons are shown in Fig. 6. It can be seen that our method surpasses the unsupervised and weakly-supervised methods in terms of structural integrity and accuracy, and achieves comparable results to fully supervised methods. Our advantages are reflected in the following aspects:
Advantages in background suppression: Our model can effectively suppress noise and accurately locate the position of salient objects. For example, in the second image, the unsupervised SVF method and scribble-based WSSA method fail to accurately locate the boundary between the snow and the sheep from the complex backgrounds. Also, in the fourth image, some weakly-supervised methods (e.g., WSSD , WSSA ), as well as several fully-supervised methods (e.g., CPD , PAGR , PFAN ), are wrongly detect the hand as the salient object. In contrast, our model has better results in terms of location accuracy and background suppression.
Advantages in detail depiction: Our model has a better ability to capture detailed information such as sharp boundaries and complete structure. In the third image, other weakly-supervised methods either fail to detect the dog’s limbs completely, or fail to distinguish the boundary between the limbs and the grass background. Similarly, neither the cow in the fifth image nor the person in the sixth image can be detected completely. In contrast, our method is not only able to detect the relatively complete structure of these salient objects, but also has clearer and sharper boundaries. In the last image, our method has a clear advantage in characterizing the horns and limbs of the cow compared to other weakly-supervised methods, especially our method can accurately detect the cow feet at a distance, producing more complete result.
Advantages in low-contrast scene: Our model can identify the salient object although in low-contrast scenes. For example, The color of the eagle¡¯s wings and the mountain behind it are so close that it is difficult for even fully-supervised methods to fully detect this region, such as CPD and PAGR . As you can easily imagine, all other weakly-supervised methods also fail in this region. Fortunately, thanks to the entire network architecture and the multi-dimensional feature fusion, the method proposed in this paper successfully detects the left wing of the eagle. Furthermore, in the eighth image, not only is the color of the leopard very close to the tree trunk, but part of its legs are covered by the tree trunk, which increases the difficulty of detection. However, our model can still detect the entire leopard’s legs based on the relationship between objects, which even exceeds the detection ability of some strongly supervised models.
IV-C Ablation Study
To validate the effectiveness of our proposed network, we conduct comprehensive ablation experiments on the HKU-IS, DUTS-TE, and PASCAL-S datasets, including the overall framework, the design of R-Net, and the training strategy.
In the face of hybrid labels, Fig. 2 shows two SOD framework pipelines. One is direct hybrid training, as shown in Fig. 2(a), or even training with only real labels, and the other is our proposed framework in this paper, where label refinement and SOD are alternately trained under hybrid labels. To verify the effectiveness of our overall framework, we design three ablation experiments. (1) M1: we first train the S-Net with 9,000 coarse-labeled samples and then fine-tune it with 1,000 real-labeled samples. (2) M2: we only train the S-Net with 1,000 real-labeled samples. (3) M3: we first train the S-Net using 1,000 samples with real labels, then use the trained S-Net to predict and update the original coarse labels for the remaining 9,000 samples, and finally retrain the S-Net by using the updated coarse-labeled samples and real-labeled samples. The quantitative results on the PASCAL-S, DUTS-TE and HKU-IS datasets are reported in Table II, and some visual comparisons are shown in Fig. 7.
Comparing M1 and M2, we can see that directly introducing coarse labels leads to a significant drop in performance, mainly due to the unreliable noise of coarse labels. Compared with these two schemes, our proposed framework guarantees that better results can still be achieved when training with coarse labels. For example, on the DUTS-TE dataset, the percentage gain of max F-measure against the M2 model is 8.4%, and the percentage gain of max F-measure against the M1 model reaches 29.1%. All these results demonstrate the effectiveness of our overall framework. As can be seen from Fig. 7, experiment M1 can only detect the main part of the object, with obvious omissions (such as the chicken on the far right in the first image and the pomegranate on the left in the second image). Furthermore, the results of experiment M1 are inferior to those of experiment M2 trained with only 1,000 real-labeled data. However, the M2 still contains a lot of noise and has very limited ability to describe the details (such as the duck’s paws in the first image).
In addition, the performance of experiment M3 outperforms the experiment M2, but our model still has obvious advantages in performance. For example, on the DUTS-TE dataset, compared with experiment M3, the max F-measure score of the full model is improved from 0.755 to 0.803 with a percentage gain of 6.3%, and the MAE score is improved from 0.065 to 0.050 with a percentage gain of 23.1%. On the HKU-IS dataset, the max F-measure score of the full model is improved from 0.868 to 0.892 with a percentage gain of 2.8% compared with experiment M3, and the MAE score is improved from 0.048 to 0.038 with a percentage gain of 20.8%. For the experiment M3, although the noise is significantly reduced, the performance is still inferior to our framework in terms of details, such as chicken feet. All these experiments verify the effectiveness of our overall framework.
IV-C2 Effectiveness of the R-Net
In the design of R-Net, the BGA module is a crucial core module, including the guidance stage and aggregation stage. In order to demonstrate the effectiveness of the designed BGA module R-Net, we conduct the ablation experiments on the PASCAL-S, DUTS-TE and HKU-IS datasets, and the quantitative and quantitative results are reported in Table III and Fig. 8.
First, we replace the BGA module with a simple concatenation-convolution fusion, thereby forming the baseline model. Based on the baseline model, we separately add the guidance and aggregation stages in the verification experiments. In the guidance stage, the saliency-refinement mainstream branch is supplemented and enriched with the guidance information (e.g., object localization and completeness) of the RGB-image guidance branch. As shown in the Table III, compared with the baseline model on the DUTS-TE dataset, the max F-measure is improved from 0.766 to 0.786 by only introducing the guidance stage, with a percentage gain of 2.6%. The visualization results in Fig. 8 show that some irrelevant backgrounds are effectively suppressed (such as the left region in the first image), and some salient regions can also be recovered (such as the lower wing region in the third image), but there are also some cases where the detection is incomplete (such as top wing in the third image). In addition, the aggregation stage aims to more comprehensively integrate the corresponding encoder features, previous decoder features, and global context features. When only the aggregation stage is introduced, we can achieve better performance than the baseline model, and even slightly better than the model only with the guidance stage, which also illustrates the importance of effective multi-level fusion. For example, only with the aggregation stage on the DUTS-TE dataset, the max F-measure is improved from 0.766 to 0.791, with a percentage gain of 3.3%. From the visualization results, it can be seen that the aggregation stage can better complete the object structure (such as the upper wing in the third image), but still introduces some additional noise and interference. By contrast, the model that includes both stages achieves the best performance. On the PASCAL-S, DUTS-TE and HKU-IS datasets, the percentage gain of the max F-measure reaches 4.4%, 4.8% and 3.1% against the baseline model, respectively. Also, the structure of the final result is more complete, and irrelevant background regions are suppressed more thoroughly.
Besides, to verify the effectiveness of the introduction of the RGB-image guidance branch in the R-Net, we add an ablation experiment. As a comparison, we remove the RGB-image guidance branch from the full model, denoted as w/o RGB. From the Table IV, we can see that the performance of the network degrades after removing the RGB branch on three testing datasets. For example, on the DUTS-TE dataset, compared with the model without the RGB branch, the max F-measure score is improved from 0.761 to 0.803 with a percentage gain of 5.5%, and the MAE score is improved from 0.077 to 0.050 with a percentage gain of 35.1%.
IV-C3 Effectiveness of Training Strategy
To validate the effectiveness of our proposed training strategy, we conduct two ablation experiments: (1) No.1: we simplify the designed training strategy. The training of the two networks is no longer performed alternately, but directly trains R-Net on 1,000 real-labeled samples, then tests 9,000 samples with coarse labels to obtain the corresponding pseudo labels, and finally uses the pseudo-labeled and real-labeled samples to train the S-Net. This experiment is designed to verify the effectiveness of the overall training strategy. (2) No.2: we remove the credibility verification mechanism in each iteration to verify its effectiveness. (3) No.3: we use all refined coarse labels and real labels as supervision in the fifth iteration of S-Net training. (4) No.4: we remove contaminated data from each iteration and participate in each iteration using real-labeled samples, which is used to verify the effectiveness of the contamination mechanism on the real-labeled data.
As can be found in Table V, even with our designed framework pipeline of label refinement and SOD, without our proposed training strategy, the network cannot exert its maximum advantage. For example, on the DUTS-TE dataset, the max F-measure of experiment No.1 drops from 0.803 to 0.763 compared to the model with full training strategy, a decrease of 4%. In addition, we introduce a credibility verification mechanism from the second iteration to ensure the validity of the pseudo labels. It can also be seen from the table that after removing this mechanism for experiment No.2, the indicators on all datasets decreased. From experiment No.3, we can see that although the amount of data for training S-Net is increased, the balance of data is disrupted, resulting in a slight decrease in the performance of the model instead of increasing. For example, compared with original training strategy, the max F-measure score of experiment No.3 drops from 0.827 to 0.824 on PASCAL-S dataset, and from 0.803 to 0.800 on the DUTS-TE dataset. Moreover, the training time of experiment No.3 is much longer than the full model. In addition, the role of the contamination labels is to prevent the network from overfitting the 1,000 samples with real labels, since these samples are involved in each round of training. From Table V, it can be seen that the participation of contaminated real-labeled data in training improves the robustness and performance of the network. Compared to the version without the contamination mechanism on the real-labeled data, the MAE score is improved from 0.080 to 0.076 with a percentage gain of 5.2% on the PASCAL-S dataset, and from 0.042 to 0.038 with a percentage gain of 10.5% on the HKU-IS dataset. In summary, our model framework equipped with the designed training strategy achieves apparent advantages in the detection performance on the PASCAL-S, HKU-IS and DUTS-TE datasets.
IV-C4 Impact of the Group Settings
Considering the sample imbalance issue caused by the difference in the number of real-labeled samples and coarse-labeled samples, we propose a group-wise incremental mechanism to avoid network collapse caused by importing a large amount of pseudo-labeled data at a time. In implementation, we divide all training samples into some groups, and gradually increase the amount of data with pseudo labels in each training iteration. The number of groups simply reflects the number of samples embedded in each iteration and does not have a significant impact on performance theoretically, but the larger the number of groups, the more iterations required and the longer the training time. To this end, we design ablation experiments with different number of groups (i.e., 5 and 15), as reported in Table VI. When the number of groups is set to 5, the 1,000 samples with real labels are still grouped into GROUP 1. The remaining 9,000 samples with coarse labels are divided into four groups for training, each containing 2,250 samples. Since the number of training iterations is related to the number of groups, we add one group of data for each iteration, and these five groups are iterated three times in total. Similarly, when the number of groups is set to 15, the remaining 9,000 samples with coarse labels are divided into 14 groups of 642 or 643 images each, and the whole training process requires 8 iterations. It can be seen from Table VI that the performance of different grouping numbers is slightly different, which is also consistent with our theoretical analysis.
V Conclusion
In this paper, we propose a weakly-supervised learning framework for SOD tasks with hybrid labels, which is decoupled into a R-Net and a S-Net. In order to make full use of the limited annotation information, the R-Net equipped with Blender with Guidance and Aggregation Mechanisms is designed to refine the coarse label and generate the pseudo label for S-Net training. In addition, we design three training mechanisms to guarantee the effectiveness and efficiency of network training, including alternate iteration mechanism, group-wise incremental mechanism, and credibility verification mechanism. Evaluations of five benchmark datasets demonstrate the effectiveness of our approach.