PCL: Proposal Cluster Learning for Weakly Supervised Object Detection

Introduction

Object detection is one of the most important problems in computer vision with many applications. Recently, due to the development of Convolutional Neural Network (CNN) and the availability of large scale datasets with detailed boundingbox-level annotations , there have been great leap forwards in object detection . However, it is very labor-intensive and time-consuming to collect detailed annotations, whereas acquiring images with only image-level annotations (i.e., image tags) indicating whether an object class exists in an image or not is much easier. For example, we can use image search queries to search on the Internet (e.g., Google and Flickr) to obtain a mass of images with such image-level annotations. This fact inspires us to explore methods for the Weakly Supervised Object Detection (WSOD) problem, i.e., training object detectors with only image tag supervisions.

Many previous methods follow the Multiple Instance Learning (MIL) pipeline for WSOD . They treat images as bags and proposals as instances; then instance classifiers (object detectors) are trained under MIL constraints (i.e., a positive bag contains at least one positive instance and all instances in negative bags are negative). In addition, inspired by the great success of CNN, recent efforts often combine MIL and CNN to obtain better WSOD performance. Some researches have shown that treating CNNs pre-trained on large scale datasets as off-the-shelf proposal feature extractors can obtain much better performance than traditional hand-designed features . Moreover, many recent works have achieved even better results for WSOD by an MIL network using standard end-to-end training or a variant of end-to-end training . See Section 2.3 for this variant of end-to-end and how it differs from the standard one. We use the same strategy of training a variant of end-to-end MIL network inspired by .

Although some promising results have been obtained by MIL networks for WSOD, they do not perform as well as fully supervised ones . As shown in Fig. 3 (a), previous MIL networks integrate the MIL constraints into the network training by transferring the instance classification (object detection) problem to a bag classification (image classification) problem, where the final image scores are the aggregation of the proposal scores. However, there is a big gap between image classification and object detection. For classification, even parts of objects can contribute to correct results (e.g., the red boxes in Fig. 1), because important parts include many characteristics of the objects. Many proposals only cover parts of objects, and “seeing” proposals only of parts may be enough to roughly localize the objects. But this may not localize objects well enough considering the performance requirement of high Intersection-over-Union (IoU) between the resulting boxes and groundtruth boundingboxes: the top ranking proposals may only localize parts of objects instead of whole objects. Recall that for detection, the resulting boxes should not only give correct classification, but also localize objects and have enough overlap with groundtruth boundingboxes (e.g., the green boxes in Fig. 1).

Before presenting our solution of the problem referred above, we first introduce the concept of proposal cluster. Object detection requires algorithms to generate multiple overlapping proposals closely surrounding objects to ensure high proposal recall (e.g., for each object, there are tens of proposals on average from Selective Search which have IoU$>$0.5 with the groundtruth boundingbox on the PASCAL VOC dataset). Object proposals in an image can be grouped into different spatial clusters. Except for one cluster for background proposals, each object cluster is associated with a single object and proposals in each cluster are spatially adjacent, as shown in Fig. 2. For fully supervised object detection (i.e., training object detectors using boundingbox-level annotations), proposal clusters can be generated by treating the groundtruth boundingboxes as cluster centers. Then object detectors are trained according to the proposal clusters (e.g., assigning all proposals the label of the corresponding object class for each cluster). This alleviates the problem that detectors may only focus on parts.

But in the weakly supervised scenario, it is difficult to generate proposal clusters because groundtruth boundingboxes that can be used as cluster centers are not provided. To cope with this difficulty, we suggest to find proposal clusters as follows. First we generate proposal cluster centers from those proposals which have high classification scores during training, because these top ranking proposals can always detect at least parts of objects. That is, for each image, after obtaining proposal scores, we select some proposals with high scores as cluster centers, and then proposal clusters are generated based on spatial overlaps with the cluster centers. Then the problem reduces to how to select proposals as centers, because many high scoring proposals may correspond to the same object. The most straightforward way is to choose the proposal with the highest score for each positive object class (i.e., the object class exists in the image) as the center. But such a method ignores the fact that there may exist more than one object with the same object category in natural images (e.g., the two motorbikes in Fig. 2). Therefore, we propose a graph-based method to find cluster centers. More specifically, we build a graph of top ranking proposals according to the spatial similarity for each positive object class. In the graph, two proposals are connected if they have enough spatial overlaps. Then we greedily and iteratively choose the proposals which have most connections with others to estimate the centers. Although a cluster center proposal may only capture an object partially, its adjacent proposals (i.e., other proposals in the cluster) can cover the whole object, or at worst contain larger parts of the object.

Based on these proposal clusters, we propose two methods to refine instance classifiers (object detectors) during training. We first propose to assign proposals object labels directly. That is, for each cluster, we assign its proposals the label of its corresponding object class, as in Fig. 3 (b). Compared with the conventional MIL network in Fig. 3 (a), this strategy forces network to “see” larger parts of objects by assigning object labels to proposals that cover larger parts of objects directly, which fills the gap between classification and detection to some extent. While effective, this strategy still has potential ambiguities, because assigning the same object label to proposals that cover different parts of objects simultaneously may confuse the network and will hurt the discriminative power of the detector. To address this problem, we propose to treat each proposal cluster as a small new bag to train refined instance classifiers, as in Fig. 3 (c). Most of the proposals in these new bags should have relatively high classification scores because the cluster centers covers at least parts of objects and proposals in the same cluster are spatially adjacent (except for the background cluster). In the same time, not all proposals in the bags should have high classification scores. Thus compared with the directly assigning label strategy, this strategy is more flexible and can reduce the ambiguities to some extent. We name our method Proposal Cluster Learning (PCL) because it learns refined instance classifiers based on proposal clusters.

To implement our idea effectively and efficiently, we further propose an online training approach. Our network has multiple output streams as in Fig. 4. The first stream is a basic MIL network which aggregates proposal scores into final image scores to train basic instance classifiers, and the other streams refine the instance classifiers iteratively. During the forward process of training, proposal classification scores are obtained and proposal clusters are generated consequently for each stream. Then based on these proposal clusters, supervisions are generated to compute losses for the next stream. According to the losses, these refined classifiers are trained during back-propagation. Except for the first stream that is supervised by image labels, the other streams are supervised by the image labels as well as outputs from their preceding streams. As our method forces the network to “see” larger parts of objects, the detector can discover the whole object instead of parts gradually by performing refinement multiple times (i.e., multiple output streams). But at the start of training, all classifiers are almost untrained, which will result in very noisy proposal clusters, and so the training will deviate from the correct solutions a lot. Thus we design a weighted loss further by associating different proposals with different weights in different training iterations. After that, all training procedures can thus be integrated into a single end-to-end network. This can improve the performance benefiting from our PCL-based classifier refinement procedure. It is also very computational efficient in both training and testing. In addition, performance can be improved by sharing proposal features among different output streams.

We elaborately conduct many experiments on the challenging PASCAL VOC, ImageNet detection, and MS-COCO datasets to confirm the effectiveness of our method. Our method achieves $48.8\%$ mAP and $66.6\%$ CorLoc on VOC 2007 which is more than $5\%$ absolute improvement compared with previous best performed methods.

This paper is an extended version of our previous work . In particular, we give more analyses of our method and enrich literatures of most recent related works, making the manuscript more complete. In addition, we make two methodological improvements: the first one is to generate proposal clusters using graphs of top ranking proposals instead of using the highest scoring proposal, and the second one is to treat each proposal cluster as a small new bag. In addition, we provide more discussions of experimental results, and show the effectiveness of our method on the challenging ImageNet detection and MS-COCO datasets.

The rest of our paper is organized as follows. In Section 2, some related works are introduced. In Section 3, the details of our method are described. Elaborate experiments and analyses are conducted in Section 4. Finally, conclusions and future directions are presented in Section 5.

Related work

MIL, first proposed for drug activity prediction , is a classical weakly supervised learning problem. Many variants have been proposed for MIL . In MIL, a set of bags are given, and each bag is associated with a collection of instances. It is natural to treat WSOD as an MIL problem. Then the problem turns into finding instance classifiers only given bag labels. Our method also follows the MIL strategy and makes several improvements to WSOD. In particular, we learn refined instance classifiers based on proposal clusters according to both instance scores and spatial relations in an online manner. “Instance” and “proposal” are used interchangeably in this paper.

MIL has many applications to computer vision, such as image classification , weakly supervised semantic segmentation , object detection , object tracking , etc. The strategy of treating proposal clusters as bags was partly inspired by , where proposes to train MIL for patches around groundtruth locations and proposes to train MIL for patches around predicted object locations. However, they require groundtruth locations for either all training samples or the beginning time frames , whereas WSOD does not have such annotations. Therefore, it is much harder to generate proposal clusters only guided by image-level supervisions for WSOD. In addition, we incorporate the strategy of treating proposal clusters as bags into the network training whereas do not. Oquab et al. also train a CNN network using the max pooing MIL strategy to localize objects. But their methods can only coarsely localize objects regardless of their sizes and aspect ratios, whereas our method can detect objects more accurately.

2 Weakly supervised object detection

WSOD has attracted great interests nowadays because the amount of data with image-level annotations is much bigger and is growing much faster than that with boundingbox-level annotations. Many methods are emerging for the WSOD problem . For example, Chum and Zisserman first initialize object locations by discriminative visual words and then introduce an exemplar model to measure similarity between image pairs for updating locations. Deselaers et al. propose to initialize boxes by objectness and use a CRF-based model to iteratively localize objects. Pandey and Lazebnik train a DPM model under weak supervisions for WSOD. Shi et al. use Bayesian latent topic models to jointly model different object classes and background. Song et al. develop a technology to discover frequent discriminative configurations of visual patterns for robust WSOD. Cinbis et al. iteratively train a multi-fold MIL to avoid the detector being locked onto inaccurate local optima. Wang et al. relax the MIL constraints into a derivable loss function to train detectors more efficient.

Recently, with the revolution of CNNs in computer vision, many works also try to combine the WSOD with CNNs. Early works treat CNN models pre-trained on ImageNet as off-the-shelf feature extractors . They extract CNN features for each candidate regions, and then train their own detectors on top of these features. These methods have shown that CNN descriptors can boost performance against traditional hand-designed features. More recent efforts tend to train end-to-end networks for WSOD . They integrate the MIL constraints into the network training by aggregating proposal classification scores into final image classification scores, and then image-level supervision can be directly added to image classification scores. For example, Tang et al. propose to use max pooling for aggregation. Bilen and Vedaldi develop a weighted sum pooing strategy. Building on , Kantorov et al. argue that context information can improve the performance. Diba et al. show that weakly supervised segmentation map can be used as guidance to filter proposals, and jointly train the weakly supervised segmentation network and WSOD end-to-end. Our method is built on these networks and any of them can be chosen as our basic network. Our strategy proposes to learn refined instance classifiers based on proposal clusters, and propose a novel online approach to train our network effectively and efficiently. Experimental results show our strategies can boost the results significantly.

In addition to the weighted sum pooing, also proposes a “spatial regulariser” that forces features of the highest scoring proposal and its spatially adjacent proposals to be the same. Unlike this, we show that finding proposal cluster centers using graph and treating proposal clusters as bags are more effective. The contemporary work uses a graph model to generate seed proposals. Their network training has many steps: first, an MIL network is trained; second, seed proposals are generated using the graph; third, based on these seed proposals, a Fast R-CNN like detector is trained. Our method differs from in many aspects: first, we propose to generate proposal clusters for each training iteration and thus our network is trained end-to-end instead of step-by-step, which is more efficient and can benefit from sharing proposal features among different streams; second, we propose to treat proposal clusters as bags for training better classifiers. As evidenced by experiments, our method obtains much better and more robust results.

3 End-to-end and its variants

In standard end-to-end training, the update requires optimizing losses w.r.t. all functions of network parameters. For example, the Fast R-CNN optimizes their classification loss and boundingbox regression loss w.r.t. proposal classification and feature extraction for fully supervised object detection. The MIL networks in optimize their MIL loss w.r.t. proposal classification and feature extraction for WSOD.

Unlike the standard end-to-end training, there exists a variant of end-to-end training. The variant contains functions which depend on network parameters, but losses are not optimized w.r.t. all these functions . As we described in Section 2.2, the “spatial regulariser” in forces features of the highest scoring proposal and its spatially adjacent proposals to be the same. They use a function of network parameters to compute the highest scoring proposal, and do not optimize their losses w.r.t. this function. Diba et al. filter out background proposals using a function of network parameters and use these filtered proposals in their latter network computations. They also do not optimize their losses w.r.t. this function. Inspired by , we use this variant of end-to-end training. More precisely, we do not optimize our losses w.r.t. the generated supervisions for instance classifier refinement.

4 Others

There are many other important related works that do not focus on weakly supervised learning but should be discussed. Similar to other end-to-end MIL networks, our method is built on top of the Region of Interest (RoI) pooling layer or Spatial Pyramid Pooling (SPP) layer to share convolutional computations among different proposals for model acceleration. But both and require boundingbox-level annotations to train their detectors. The sharing proposal feature strategy in our network is similar to multi-task learning . Unlike the multi-task learning that each output stream has their own relatively independent external supervisions for different tasks, in our method, all streams have the same task and supervisions of later streams depend on the outputs from their preceding streams.

Method

The overall architecture of our method is shown in Fig. 4. Given an image, about $2,000$ object proposals from Selective Search or EdgeBox are generated. During the forward process of training, the image and these proposals are fed into some convolutional (conv) layers with an SPP layer to produce a fixed-size conv feature map per-proposal. After that, proposal feature maps are fed into two fully connected (fc) layers to produce proposal features. These features are branched into different streams: the first one is an MIL network to train basic instance classifiers and the others refine the classifiers iteratively. For each stream, proposal classification scores are obtained and proposal clusters are generated consequently. Then based on these proposal clusters, supervisions are generated to compute losses for the next stream. During the back-propagation process of training, the network losses are optimized to train proposal features and classifiers. As shown in the figure, supervisions of the $1$-st refined classifier depend on the output from the basic classifier, and supervisions of $k$-th refined classifier depend on outputs from $\{k-1\}$-th refined classifier. In this section, we will introduce our method of learning refined instance classifiers based on proposal clusters in detail.

We compute $N^{k}$ proposal cluster centers ${\cal S}^{k}=\{S^{k}_{n}\}_{n=1}^{N^{k}}$ for the $k$-th refinement. The $n$-th cluster center $S^{k}_{n}=(b^{k}_{n},y^{k}_{n},s^{k}_{n})$ consists of a proposal box $b^{k}_{n}\in{\cal B}$, an object label $y^{k}_{n}$ ($y^{k}_{n}=c,c\in\{1,...,C\}$ indicates the $c$-th object class), and a confidence score $s^{k}_{i}$ indicating the confidence that $b^{k}_{n}$ covers at least part of an object of class $y^{k}_{n}$. We have $N^{k}+1$ proposal clusters ${\cal C}^{k}=\{{\cal C}^{k}_{n}\}_{n=1}^{N^{k}+1}$ according to ${\cal S}^{k}$ (${\cal C}^{k}_{N^{k}+1}$ for background and others for objects). For object clusters, the $n$-th cluster ${\cal C}^{k}_{n}=({\cal B}^{k}_{n},y^{k}_{n},s^{k}_{n}),n\neq N^{k}+1$ consists of $M^{k}_{n}$ proposal boxes ${\cal B}^{k}_{n}=\{b^{k}_{nm}\}_{m=1}^{M^{k}_{n}}\subseteq{\cal B}$, an object label $y^{k}_{n}$ that is the same as the cluster center label, and a confidence score $s^{k}_{n}$ that is the same as the cluster center score, where $s^{k}_{n}$ indicates the confidence that ${\cal C}^{k}_{n}$ corresponds to an object of class $y^{k}_{n}$. Unlike object clusters, the background cluster ${\cal C}^{k}_{n}=({\cal P}^{k}_{n},y^{k}_{n}),n=N^{k}+1$ consists of $M^{k}_{n}$ proposals ${\cal P}^{k}_{n}=\{P^{k}_{nm}\}_{m=1}^{M^{k}_{n}}$ and a label $y^{k}_{n}=C+1$ indicating the background. The $m$-th proposal $P^{k}_{nm}=(b^{k}_{nm},s^{k}_{nm})$ consists of a proposal box $b^{k}_{nm}\in{\cal B}$ and a confidence score $s^{k}_{nm}$ indicating the confidence that $b^{k}_{nm}$ is the background.

2 Basic MIL network

It is necessary to generate proposal scores and clusters to supervise refined instance classifiers. More specifically, the first refined classifier requires basic instance classifiers to generate proposal scores and clusters. Therefore, we first introduce our basic MIL network as the basic instance classifier. Our overall network is independent of the specific MIL methods, and thus any method that can be trained end-to-end could be used. There are many possible choices . Here we choose the method by Bilen and Vedaldi which proposes a weighted sum pooling strategy to obtain the instance classifier, because of its effectiveness and implementation convenience. To make our paper self-contained, we briefly introduce as follows.

A simple interpretation of the two branches framework is as follows. $[\bm{\sigma}(\mathbf{X}^{\textup{cls}})]_{cr}$ is the probability of the $r$-th proposal belonging to class $c$. $[\bm{\sigma}(\mathbf{X}^{\textup{det}})]_{cr}$ is the normalized weight that indicates the contribution of the $r$-th proposal to image being classified to class $i$. So $[\bm{\phi}(\mathbf{F},\mathbf{W}^{0})]_{c}$ is obtained by weighted sum pooling and falls in the range of $(0,1)$. Given the image label vector $\mathbf{y}=[y_{1},...,y_{C}]^{T}$. We train the basic instance classifier by optimizing the multi-class cross entropy loss Eq. (1) w.r.t. $\mathbf{F},\mathbf{W}^{0}$.

3 The overall training strategy

As we stated before, supervisions to train the $k$-th instance classifier are generated based on proposal scores $\bm{\varphi}^{k-1}$ and image label $\mathbf{y}$. Thus we denote the supervisions by ${\cal H}^{k}(\bm{\varphi}^{k-1},\mathbf{y})$. Then we train our overall network by optimizing the loss Eq. (2) w.r.t. $\mathbf{F},\mathbf{W}^{k}$. We do not optimize the loss w.r.t. ${\cal H}^{k}(\bm{\varphi}^{k-1},\mathbf{y})$, which means that the supervisions ${\cal H}^{k}(\bm{\varphi}^{k-1},\mathbf{y})$ are only computed in the forward process and we do not compute their gradients to train our network.

During the forward process of each Stochastic Gradient Descent (SGD) training iteration, we obtain a set of proposal scores of an input image. Accordingly, we generate the supervisions ${\cal H}^{k}(\bm{\varphi}^{k-1},\mathbf{y})$ for the iteration to compute the loss Eq. (2). During the back-propagation process of each SGD training iteration, we optimize the loss Eq. (2) w.r.t. proposal features $\mathbf{F}$ and classifiers $\mathbf{W}^{k}$. We summarize this procedure in Algorithm 1. Note that we do not use an alternating training strategy, i.e., fixing supervisions and training a complete model, fixing the model and updating supervisions. The reasons are that: 1) it is very time-consuming because it requires training models multiple times; 2) training different models in different refinement steps separately may harm the performance because it hinders the process to benefit from the shared proposal features (i.e., $\mathbf{F}$).

4 Proposal cluster learning

Here we will introduce our methods to learn refined instance classifiers based on proposal clusters (i.e., proposal cluster learning).

For the first step, we compute proposal cluster centers ${\cal S}^{k}=\{S^{k}_{n}\}_{n=1}^{N^{k}}$ based on $\bm{\varphi}^{k-1}$ and $\mathbf{y}$. The $n$-th cluster center $S^{k}_{n}=(b^{k}_{n},y^{k}_{n},s^{k}_{n})$ is defined in Section 3.1. We propose two algorithms to find ${\cal S}^{k}$ in Section 3.4.1 (1) and (2) (also Algorithm 2 and Algorithm 3), where the first one was proposed in the conference version paper and the second one is proposed in this paper.

For the second step, according to the proposal cluster centers ${\cal S}^{k}$, proposal clusters ${\cal C}^{k}=\{{\cal C}^{k}_{n}\}_{n=1}^{N^{k}+1}$ are generated (${\cal C}^{k}_{N^{k}+1}$ for background and others for objects). The $n$-th object cluster ${\cal C}^{k}_{n}=({\cal B}^{k}_{n},y^{k}_{n},s^{k}_{n}),n\neq N^{k}+1$ and the background cluster ${\cal C}^{k}_{n}=({\cal P}^{k}_{n},y^{k}_{n}),n=N^{k}+1$ are defined in Section 3.1. We use the different notation for the background cluster because background proposals are scattered in each image, and thus it is hard to determine a cluster center and accordingly a cluster score. The method to generate ${\cal C}^{k}$ was proposed in the conference version paper and is described in Section 3.4.2 (also Algorithm 4).

In the following we introduce two algorithms to find proposal cluster centers.

(1) Finding proposal cluster centers using the highest scoring proposal. A solution for finding proposal cluster centers is to choose the highest scoring proposal, as in our conference version paper . As in Algorithm 2, suppose an image has object class label $c$ (i.e., $y_{c}=1$). For the $k$-th refinement, we first select the $r^{k}_{c}$-th proposal which has the highest score by Eq. (3), where $\varphi^{k-1}_{cr}$ is the predicted score of the $r$-th proposal, as defined in Section 3.1.

Then this proposal is chosen as the cluster center, i.e., $S^{k}_{n}=(b^{k}_{n},y^{k}_{n},s^{k}_{n})=(b_{r^{k}_{c}},c,\varphi^{k-1}_{cr^{k}_{c}})$, where $b_{r^{k}_{c}}$ is the box of the $r^{k}_{c}$-th proposal. $\varphi^{k-1}_{cr}$ is chosen as the confidence score that the $r$-th proposal covers at least part of an object of class $c$, because $\varphi^{k-1}_{cr}$ is the predicted score of the $r$-th proposal been categorized to class $c$. Therefore, the highest scoring proposal can probably cover at least part of the object and thus be chosen as the cluster center.

There is a potential problem that one proposal may be chosen as the cluster centers for multiple object classes. To avoid this problem, if one proposal corresponds to the cluster centers for multiple object classes, this proposal would be chosen as the cluster center only by the class with the highest predicted score and we re-choose cluster centers for other classes.

(2) Finding proposal cluster centers using graphs of top ranking proposals. As stated in Section 1, although we can find good proposal cluster centers using the highest scoring proposal, this ignores that in natural images there are often more than one object for each category. Therefore, we propose a new method to find cluster centers using graphs of top ranking proposals.

More specifically, suppose an image has object class label $c$. We first select the top ranking proposals with indexes ${\cal R}^{k}_{c}=\{r^{k}_{c1},...,r^{k}_{cN^{k}_{c}}\}$ for the $k$-th refinement. Then we build an undirected unweighted graph $G^{k}_{c}=(V^{k}_{c},E^{k}_{c})$ of these proposals based on spatial similarity, where vertexes $V^{k}_{c}$ correspond to these top ranking proposals, and edges $E^{k}_{c}=\{e^{k}_{crr^{\prime}}\}=\{e(v^{k}_{cr},v^{k}_{cr^{\prime}})\},r,r^{\prime}\in{\cal R}^{k}_{c}$ correspond to the connections between the vertexes. $e^{k}_{crr^{\prime}}$ is determined according to the spatial similarity between two vertexes (i.e., proposals) as in Eq. (4), where $I_{rr^{\prime}}$ is the IoU between the $r$-th and $r^{\prime}$-th proposals and $I_{t}$ is a threshold (e.g., $0.4$).

Therefore, two vertexes are connected if they are spatially adjacent. After that, we greedily generate some cluster centers for class $c$ using this graph. That is, we iteratively select vertexes which have most connections to be the cluster centers, as in Algorithm 3. The number of cluster centers (i.e., $N^{k}$) changes for each image in each training iteration because the top ranking proposals ${\cal R}^{k}_{c}$ change. See Section 4.2.9 for some typical values of $N^{k}$. We use the same method as in Section 3.4.1 (1) to avoid one proposal been chosen as the cluster centers for multiple object classes.

The reasons for this strategy are as follows. First, according to our observation, the top ranking proposals can always cover at least parts of objects, thus generating centers from these proposals encourages the selected centers to meet our requirements. Second, because these proposals cover objects well, better proposals (covering more parts of objects) should have more spatially overlapped proposals (i.e., have more connections). Third, these centers are spatially far apart, and thus different centers can correspond to different objects. This method also has the attractive characteristic that it can generate adaptive number of proposals for each object class, which is desirable because in natural images there are arbitrary number of objects per-class. We set the score of the $n$-th proposal cluster center $s^{k}_{n}$ by

(see the $8$-th line in Algorithm 3) because if the adjacent proposals of a center proposal have high confidence to cover at least part of an object (i.e., have high classification scores) the center proposal should also have such high confidence.

There is an important issue for the graph-based method: how to select the top ranking proposals? A simple method is to select proposals whose scores exceed a threshold. But in our case, proposal scores change in each training iteration, and thus it is hard to determine a threshold. Instead, for each positive object class, we use the $k$-means algorithm to divide proposal scores of an image into some clusters, and choose proposals in the cluster which has the highest score center to form the top ranking proposals. This method ensures that we can select the top ranking proposals although proposal scores change during training. Other choices are possible, but this method works well in experiments.

4.2 Generating proposal clusters

After the cluster centers are found, we generate the proposal clusters as in our conference version paper . Except for the cluster for background, good proposal clusters require that proposals in the same cluster are associated with the same object, and thus proposals in the same cluster should be spatially adjacent. Specially, given the $r$-th proposal, we compute a set of IoUs $\{I^{k}_{r1},...,I^{k}_{rN^{k}}\}$, where $I^{k}_{rn}$ is the IoU between the $r$-th proposal and the box $b^{k}_{n}$ of the $n$-th cluster center. Then we assign the $r$-th proposal to the $n^{k}_{r}$-th object cluster if $I^{k}_{rn^{k}_{r}}$ is larger than a threshold $I^{\prime}_{t}$ (e.g., $0.5$) and to the background cluster otherwise, where $n^{k}_{r}$ is the index of the most spatially adjacent cluster center as Eq. (5).

The overall procedures to generate proposal clusters are summarized in Algorithm 4. We set the proposal scores for the background cluster to the scores of their most spatially adjacent centers as the 10-the line in Algorithm 4, because if the cluster center $S^{k}_{n}$ has confidence $s^{k}_{n}$ that it covers an object, the proposal far away from $S^{k}_{n}$ should have confidence $s^{k}_{n}$ to be background.

4.3 Learning refined instance classifiers

(1) Assigning proposals object labels. The most straightforward way to refine classifiers is to directly assign object labels to all proposals in object clusters because these proposals potentially correspond to whole objects, as in our conference version paper . As the cluster centers covers at least parts of objects, their adjacent proposals (i.e., proposals in the cluster) can contain larger parts of objects. Accordingly, we can assign the cluster label $y^{k}_{n}$ to all proposals in the $n$-th cluster.

Through iterative instance classifier refinement (i.e., multiple times of refinement as $k$ increase), the detector detects larger parts of objects gradually by forcing the network to “see” larger parts of objects.

Actually, the so learnt supervisions ${\cal H}^{k}$ are very noisy, especially in the beginning of training. This results in unstable solutions. To solve this problem, we change the loss in Eq. (6) to a weighted version, as in Eq. (7).

$\lambda^{k}_{r}$ is the loss weight that is the same as the cluster confidence score $s^{k}_{n}$ for object clusters or proposal confidence score $s^{k}_{nm}$ for the background cluster if the $r$-th proposal belongs to the $n$-th cluster. From Algorithm 4, we can observe that $\lambda^{k}_{r}$ is the same as the cluster center confidence score $s^{k}_{n}$. The reasons for this strategy are as follows. In the beginning of training, although we cannot obtain good proposal clusters, each $s^{k}_{n}$ is small, hence each $\lambda^{k}_{r}$ is small and the loss is also small. As a consequence, the performance of the network will not decrease a lot. During the training, the top ranking proposals will cover objects well, and thus we can generate good proposal clusters. Then we can train satisfactory instance classifiers.

(2) Treating clusters as bags. As we stressed before, although directly assigning proposals object labels can boost the results, it may confuse the network because we simultaneously assign the same label to different parts of objects. Focusing on this, we further propose to treat each proposal cluster as a small new bag and use the cluster label as the bag label. Thus the supervisions ${\cal H}^{k}$ for the $k$-th refinement are bag-level (cluster-level) labels, i.e., ${\cal H}^{k}=\{y^{k}_{n}\}_{n=1}^{N^{k}+1}$. $y^{k}_{n}$ is the label of the $n$-th bag, i.e., the label of the $n$-th proposal cluster, as defined in Section 3.1.

Specially, for object clusters, we choose average MIL pooling, because these proposals should cover at least parts of objects and thus should have relatively high prediction scores. For the background cluster, we assign the background label to all proposals in the cluster according to the MIL constraints (all instances in negative bags are negative). Then the loss function for refinement will be Eq. (8).

$s^{k}_{n}$, $M_{n}^{k}$, and $\varphi^{k}_{cr}$ are the cluster confidence score of the $n$-th object cluster, the number of proposals in the $n$-th cluster, and the predicted score of the $r$-th proposal, respectively, as defined in Section 3.1. $b_{r}\in{\cal B}^{k}_{n}$ and $r\in{\cal C}^{k}_{N^{k}+1}$ indicate that the $r$-th proposal belongs to the $n$-th object cluster and the background cluster respectively.

Compared with the directly assigning label approach, this method tolerates some proposals to have low scores, which can reduce the ambiguities to some extent.

5 Testing

During testing, the proposal scores of refined instance classifiers are used as the final detection scores, as the blue arrows in Fig. 4. Here the mean output of all refined classifiers is chosen. The Non-Maxima Suppression (NMS) is used to filter out redundant detections.

Experiments

In this section, we first introduce our experimental setup including datasets, evaluation metrics, and implementation details. Then we conduct elaborate experiments to discuss the influence of different settings. Next, we compare our results with others to show the effectiveness of our method. After that, we show some qualitative results for further analyses. Finally, we give some runtime analyses of our method. Codes for reproducing our results are available at https://github.com/ppengtang/oicr/tree/pcl.

We evaluate our method on four challenging datasets: the PASCAL VOC 2007 and 2012 datasets , the ImageNet detection dataset , and the MS-COCO dataset . Only image-level annotations are used to train our models.

The PASCAL VOC 2007 and 2012 datasets have $9,962$ and $22,531$ images respectively for $20$ object classes. These two datasets are divided into train, val, and test sets. Here we choose the trainval set ($5,011$ images for 2007 and $11,540$ images for 2012) to train our network. For testing, there are two metrics for evaluation: mAP and CorLoc. Following the standard PASCAL VOC protocol , Average Precision (AP) and the mean of AP (mAP) is the evaluation metric to test our model on the testing set. Correct Localization (CorLoc) is to test our model on the training set measuring the localization accuracy . All these two metrics are based on the PASCAL criterion, i.e., IoU$>$0.5 between groundtruth boundingboxes and predicted boxes.

The ImageNet detection dataset has hundreds of thousands of images with $200$ object classes. It is also divided into train, val, and test sets. Following , we split the val set into val1 and val2, and randomly choose at most $1$K images in the train set for each object class (we call it train${}_{\textup{1K}}$). We train our model on the mixture of train${}_{\textup{1K}}$ and val1 sets, and test it on the val2 set, which will lead to $160,651$ images for training and $9,916$ images for testing. We also use the mAP for evaluation on the ImageNet.

The MS-COCO dataset has $80$ object classes and is divided into train, val, and test sets. Since the groundtruths on the test set are not released, we train our model on the MS-COCO 2014 train set (about $80$K images) and test it on the val set (about $40$K images). For evaluation, we use two metrics mAP@0.5 and mAP@[.5, .95] which are the standard PASCAL criterion (i.e., IoU$>$0.5) and the standard MS-COCO criterion (i.e., computing the average of mAP for IoU$\in$[0.5 : 0.05 : 0.95]) respectively.

1.2 Implementation details

Our method is built on two pre-trained ImageNet networks VGG$\_$M and VGG16 , each of which has some conv layers with max-pooling layers and three fc layers. We replace the last max-pooling layer by the SPP layer, and the last fc layer as well as the softmax loss layer by the layers described in Section 3. To increase the feature map size from the last conv layer, we replace the penultimate max-pooling layer and its subsequent conv layers by the dilated conv layers . The newly added layers are initialized using Gaussian distributions with -mean and standard deviations $0.01$. Biases are initialized to .

During training, the mini-batch size for SGD is set to be $2$, $32$, and $4$ for PASCAL VOC, ImageNet, and MS-COCO, respectively. The learning rate is set to $0.001$ for the first $40$K, $60$K, $15$K, and $85$K iterations for the PASCAL VOC 2007, PASCAL VOC 2012, ImageNet, and MS-COCO datasets, respectively. Then we decrease the learning rate to $0.0001$ in the following $10$K, $20$K, $5$K, and $20$K iterations for the PASCAL VOC 2007, PASCAL VOC 2012, ImageNet, and MS-COCO datasets, respectively. The momentum and weight decay are set to be $0.9$ and $0.0005$ respectively.

Selective Search , EdgeBox , and MCG are adopted to generate about $2,000$ proposals per-image for the PASCAL VOC, ImageNet, and MS-COCO datasets, respectively. For data augmentation, we use five image scales $\{480,576,688,864,1200\}$ (resize the shortest side to one of these scales) with horizontal flips for both training and testing. If not specified, the instance classifiers are refined three times, i.e., $K=3$ in Section 3.3, so there are four output streams; the IoU threshold $I_{t}$ in Section 3.4.1 (2) (also Eq. (4)) is set to $0.4$; the number of $k$-means clusters in the last paragraph of Section 3.4.1 (2) is set to $3$; $I^{\prime}_{t}$ in Section 3.4.2 (also the $5$-th line of Algorithm 4) is set to $0.5$.

Similar to other works , we train a supervised object detector through choosing the top-scoring proposals given by our method as pseudo groundtruths to further improve our results. Here we train a Fast R-CNN (FRCNN) using the VGG16 model and the same five image scales (horizontal flips only in training). The same proposals are chosen to train and test the FRCNN. NMS (with $30\%$ IoU threshold) is applied to compute AP.

Our experiments are implemented based on the Caffe deep learning framework, using Python and C++. The $k$-means algorithm to produce top ranking proposals is implemented by scikit-learn . All of our experiments are running on an NVIDIA GTX TitanX Pascal GPU and Intel(R) i7-6850K CPU (3.60GHz).

2 Discussions

We first conduct some experiments to discuss the influence of different components of our method (including instance classifier refinement, different proposal generation methods, different refinement strategies, and weighted loss) and different parameter settings (including the IoU threshold $I_{t}$ defined in Section 3.4.1 (2), the number of $k$-means clusters described in Section 3.4.1 (2), the IoU threshold $I^{\prime}_{t}$ defined in Section 3.4.2, and multi-scale training and testing.) We also discuss the number of proposal cluster centers. Without loss of generality, we only perform experiments on the VOC 2007 dataset and use the VGG$\_$M model.

As the five curves in Fig. 5 show, we observe that compared with the basic MIL network, for both refinement methods, even refining instance classifier a single time boosts the performance a lot. This confirms the necessity of refinement. If we refine the classifier multiple times, the results are improved further. But when refinement is implemented too many times, the performance gets saturated (there are no obvious improvements from $3$ times to $4$ times). This is because the network tends to converge so that the supervision of the $4$-th time is similar to the $3$-rd time. In the rest of this paper we only refine classifiers $3$ times. Notice that in Fig. 5, the “0 time” is similar to the WSDDN using Selective Search as proposals.

2.2 The influence of different proposal cluster generation methods

We discuss the influence of different proposal cluster generation methods. As shown in the Fig. 5 (green and purple solid curves for the highest scoring proposal based method, blue and red solid curves for the graph-based method), for all refinement times, the graph-based method obtains better performance, because it can generate better cluster centers. Thus we choose the graph-based method in the rest of our paper.

2.3 The influence of different refinement strategies

We then show the influence of different refinement strategies. The directly assigning label method is replaced by treating clusters as bags (blue and green solid curves). From Fig. 5, it is obvious that the results by treating clusters as bags are better. In addition, compared with the alternating training strategy (blue dashed curve), our online training boosts the performance consistently and significantly, which confirms the necessity of sharing proposal features. Online training also reduces the training time a lot, because it only requires training a single model instead of training $K+1$ models for $K$ times refinement in the alternating strategy. In the rest of our paper, we only report results by the “PCL-OB-G” method in Fig. 5 because it achieves the best performance.

2.4 The influence of weighted loss

We also study the influence of our weighted loss in Eq. (8). Note that Eq. (8) can be easily changed to the unweighted version by simply setting $\lambda^{k}_{r}$ and $s^{k}_{n}$ to be $1$. Here we train a network using the unweighted loss. The results of the unweighted loss are mAP $33.6\%$ and CorLoc $51.2\%$. We see that if we use the unweighted loss, the improvement from refinement is very scant and the performance is even worse than the alternating strategy. Using the weighted loss achieves much better performance (mAP $40.8\%$ and CorLoc $59.6\%$), which confirms our theory in Section 3.4.3.

Here we discuss the influence of the IoU threshold $I_{t}$ defined in Section 3.4.1 (2) and Eq. (4). From Fig. 7, we see that setting $I_{t}$ to $0.4$ obtains the best performance. Therefore, we set $I_{t}$ to $0.4$ for the other experiments.

2.6 The influence of the number of k𝑘k-means clusters

In previous experiments we set the number of $k$-means clusters described in the last paragraph of Section 3.4.1 (2) to be $3$. Here we set it to other numbers to explore its influence. The results from other numbers of $k$-means clusters are mAP $40.2\%$ and CorLoc $59.3\%$ for $2$ clusters, and mAP $40.7\%$ and CorLoc $59.6\%$ for $4$ clusters, which are a little worse than the results from $3$ cluster. Therefore, we set the number of $k$-means clusters to $3$ for the other experiments.

We also analyse the influence of $I^{\prime}_{t}$ defined in Section 3.4.2 and the $5$-th line of Algorithm 4. As shown in Fig. 7, $I^{\prime}_{t}=0.5$ outperforms other choices. Therefore, we set $I^{\prime}_{t}$ to $0.5$ for the other experiments.

2.8 The influence of multi-scale training and testing

Previously our experiments are conducted based on five image scales for training and testing. Here we show the influence of this multi-scale setting. We train and test our method using a single image scale $600$ as the default scale setting of FRCNN . The single-scale results are mAP $37.4\%$ and CorLoc $55.5\%$ which are much worse than our multi-scale results (mAP $40.8\%$ and CorLoc $59.6\%$). Therefore, we use five image scales as many WSOD networks .

2.9 The number of proposal cluster centers

As we stated in Section 3.4.1 (2), the number of proposal cluster centers (i.e., $N^{k}$) changes for each image in each training iteration. Here we give some typical values of $N^{k}$. In the beginning of training, the proposal scores are very noisy and thus the selected top ranking proposals to form graphs are scattered in images, which results in dozens of proposal cluster centers for each image. After some (about 3K) training iterations, the proposal scores are more reliable and our method finds 1$\sim$3 proposal cluster centers for each positive object class. To make the training more stable in the beginning, for each positive object class we empirically select at most five proposal cluster centers which have higher scores, and the number of selected proposal cluster centers does not influence the performance much.

3 Comparison with other methods

Here we compare our best performed strategy PCL-OB-G, i.e., using graph-based method and treating clusters as bags to train the network online, with other methods.

We first report our results for each class on VOC 2007 and 2012 in Table I, Table II, Table III, and Table IV. It is obvious that our method outperforms other methods using single model VGG$\_$M or VGG16 (PCL-OB-G+VGG$\_$M and PCL-OB-G+VGG16 in tables.) Our single model results even better than others by combining multiple different models (e.g., ensemble of models) . Specially, our method obtains much better results compared with other two methods also using the same basic MIL network . Importantly, also equips the weighted sum pooling with objectness measure of EdgeBox and the spatial regulariser, and adds context information into the network, both of which are more complicated than our basic MIL network. We believe that our performance can be improved by choosing better basic MIL networks, like the complete network in and using context information . As reimplementing their method completely is non-trivial, here we only choose the simplest architecture in . Even in this simplified case, our method achieves very promising results.

Our results can also be improved by combing multiple models. As shown in the tables, there are little improvements from the ensemble of the VGG$\_$M and VGG16 models (PCL-OB-G-Ens. in tables). Here we do the ensemble by summing up the scores produced by the two models. Also, as mentioned in Section 4.1, similar to , we train a FRCNN detector using top-scoring proposals produced by PCL-OB-G-Ens. as groundtruths (PCL-OB-G-Ens.+FRCNN in tables). As we can see, the performance is improved further.

We then show results of our method on the large scale ImageNet detection dataset in Table V. We observe similar phenomenon that our method outperforms other methods by a large margin.

We finally report results of our method on MS-COCO in Table VI. Our method obtains better performance than the recent work . In particular, Ge et al. use the method proposed in our conference version paper as a basic component. We can expect to obtain better detection performance through replacing our conference version method in by our newly proposed method here, which we would like to explore in the future.

4 Qualitative results

We first show some proposal clusters generated by our method in Fig. 8. As we can see, the cluster centers contain at least parts of objects and are able to cover adaptive number of objects for each class.

We then show qualitative comparisons among the WSDDN , the WSDDN+context , and our PCL method, both of which use the same basic MIL network. As shown in Fig. 9, we can observe that for classes such as bike, car, cat, etc., our method tends to provide more accurate detections, whereas other two methods sometimes fails by producing boxes that are overlarge or only contain parts of objects (the first four rows in Fig. 9). But for some classes such as person, our method sometimes fails by only detecting parts of objects such as the head of person (the fifth row in Fig. 9). Exploiting context information sometimes help the detection (as in WSDDN+context ), we believe our method can be further improved by incorporating context information into our framework. All these three methods (actually almost all weakly supervised object detection methods) suffers from two problems: producing boxes that not only contain the target object but also include their adjacent similar objects, or only detecting parts of object for objects with deformation (the last row in Fig. 9).

We finally visualize some success and failure detection results on VOC 2007 trainval by PCL-Ens.+FRCNN, as in Fig. 10. We observe similar phenomena as in Fig. 9. Our method is robust to the size and aspect of objects, especially for rigid objects. The main failures for these rigid objects are always due to overlarge boxes that not only contain objects, but also include adjacent similar objects. For non-rigid objects like “cat”, “dog”, and “person”, they often have great deformations, but their parts (e.g., head of person) have much less deformation, so our detector is still inclined to find these parts. An ideal solution is yet wanted because there is still room for improvement.

5 Runtime

The runtime comparisons between our method and our basic MIL network are shown in Table VII, where the runtime of proposal generation is not considered. As we can see, although our method has more components than our basic MIL network , our method takes almost the same testing time as it. This is because all our output streams share the same proposal feature computations. The small extra training computations of our method mainly come from the procedures to find proposal cluster centers and generate proposal clusters. Although with small extra training computations, our method obtains much better detection results than the basic MIL network.

Conclusion

In this paper, we propose to generate proposal clusters to learn refined instance classifiers for weakly supervised object detection. We propose two strategies for proposal cluster generation and classifier refinement, both of which can boost the performance significantly. The classifier refinement is implemented by multiple output streams corresponding to some instance classifiers in multiple instance learning networks. An online training algorithm is introduced to train the proposed network end-to-end for effectiveness and efficiency. Experiments show substantial and consistent improvements by our method. We observe that the most common failure cases of our algorithm are connected with the deformation of non-rigid objects. In the future, we will concentrate on this problem. In addition, we believe our learning algorithm has the potential to be applied in other weakly supervised visual learning tasks such as weakly supervised semantic segmentation. We will also explore how to apply our method to these related applications.

Acknowledgements

This work was supported by NSFC (No. 61733007, No. 61572207, No. 61876212, No. 61672336, No. 61573160), ONR with grant N00014-15-1-2356, Hubei Scientific and Technical Innovation Key Project, and the Program for HUST Academic Frontier Youth Team. The corresponding author of this paper is Xinggang Wang.

References