Scale-Aware Trident Networks for Object Detection

Yanghao Li, Yuntao Chen, Naiyan Wang, Zhaoxiang Zhang

Introduction

In recent years, deep convolutional neural networks (CNNs) have achieved great success in object detection. Typically, these CNN-based methods can be roughly divided into two types: one-stage methods such as YOLO or SSD which directly utilizes feed-forward CNN to predict the bounding boxes of interest, while two-stage methods such as Faster R-CNN or R-FCN first generate proposals, and then exploit the extracted region features from CNN for further refinement. However, a central issue for both methods is the handling of scale variation. The scales of object instances could vary in a wide range, which impedes the detectors, especially those very small or very large ones.

To remedy the large scale variation, an intuitive way is to leverage multi-scale image pyramids , which is popular in both hand-crafted feature based methods and current deep CNN based methods (Figure 1(a)). Strong evidence shows that deep detectors could benefit from multi-scale training and testing. To avoid training objects with extreme scales (small/large objects in smaller/larger scales), SNIP proposes a scale normalization method that selectively trains the objects of appropriate sizes in each image scale. Nevertheless, the increase of inference time makes the image pyramid methods less favorable for practical applications.

Another line of efforts aims to employ in-network feature pyramids to approximate image pyramids with less computation cost. The idea is first demonstrated in , where a fast feature pyramid is constructed for object detection by interpolating some feature channels from nearby scale levels. In the deep learning era, the approximation is even easier. SSD utilizes multi-scale feature maps from different layers and detects objects of different scales at each feature layer. To compensate for the absence of semantics in low-level features, FPN (Figure 1(b)) further augments a top-down pathway and lateral connections to incorporate strong semantic information in high-level features. However, region features of objects with different scales are extracted from different levels of FPN backbone, which in turn are generated with different sets of parameters. This makes feature pyramids an unsatisfactory alternative for image pyramids.

Both the image pyramid and the feature pyramid methods share the same motivation that models should have different receptive fields for objects of different scales. Despite the inefficiency, the image pyramid fully utilizes the representational power of the model to transform objects of all scales equally. In contrast, the feature pyramid generates multi-level features thus sacrificing the feature consistency across different scales. This leads to a decrease in effective training data and a higher risk of overfitting for each scale. The goal of this work is to get the best of two worlds by creating features with a uniform representational power for all scales efficiently.

In this paper, instead of feeding in multi-scale inputs like the image pyramid, we propose a novel network structure to adapt the network for different scales. In particular, we create multiple scale-specific feature maps with the proposed trident blocks as shown in Figure 1(c). With the help of dilated convolutions , different branches of trident blocks have the same network structure and share the same parameters yet have different receptive fields. Furthermore, to avoid training objects with extreme scales, we leverage a scale-aware training scheme to make each branch specific to a given scale range matching its receptive field. Finally, thanks to weight sharing through the whole multi-branch network, we could approximate the full TridentNet with only one major branch during inference. This approximation only brings marginal performance degradation. As a result, it could achieve significant improvement over the single-scale baseline without any compromise of inference speed. This property makes TridentNet more desirable over other methods for practical uses.

To summarize, our contributions are listed as follows:

We present our investigation results about the effect of the receptive field in scale variation. To our best knowledge, we are the first to design controlled experiments to explore the receptive field on the object detection task.

We propose a novel Trident Network to deal with scale variation problem for object detection. Through multi-branch structure and scale-aware training, TridentNet could generate scale-specific feature maps with a uniform representational power.

We propose a fast approximation, TridentNet Fast, with only one major branch via our weight-sharing trident-block design, thus introducing no additional parameters and computational cost during inference.

We validate the effectiveness of our approach on the standard COCO benchmark with thorough ablation studies. Compared with the state-of-the-art methods, our proposed method achieves an mAP of 48.4 using a single model with ResNet-101 backbone.

Related Work

Deep learning based object detection methods have shown dramatic improvements in both accuracy and speed recently. As one of the predominant detectors, two-stage detection methods first generate a set of region proposals and then refine them by CNN networks. In , R-CNN generates region proposals by Selective Search and then classifies and refines the cropped proposal regions from the original image by a standard CNN independently and sequentially. To reduce the redundant computation of feature extraction in R-CNN, SPPNet and Fast R-CNN extract the feature of the whole image once, and then generate region features through spatial pyramid pooling and RoIPooling layers, respectively. The RoIPooling layer is further improved by RoIAlign layer to address the coarse spatial quantization problem. Faster R-CNN first proposes a unified end-to-end framework for object detection. It introduces a region proposal network (RPN) which shares the same backbone network with the detection network to replace the original standalone time-consuming region proposal methods. To further improve the efficiency of Faster R-CNN, R-FCN constructs position-sensitive score maps through a fully convolutional network to avoid the RoI-wise head network. To avoid additional large score maps in R-FCN, Light-Head R-CNN uses a thin feature map and a cheap R-CNN subnet to build a two-stage detector more efficiently.

On the other hand, one-stage methods which are popularized by YOLO and SSD , aim to be more efficient by directly classifying the pre-defined anchors and further refining them using CNNs without the proposal generation step. Based on the multi-layer prediction module in SSD, DSSD introduces additional context information with deconvolutional operators to improve the accuracy. RetinaNet proposes a new focal loss to address the extreme foreground-background class imbalance which stands out as a central issue in one-stage detectors. Inheriting the merits of two-stage approaches, RefineDet proposes an anchor refinement module to first filter the negative anchor boxes and coarsely adjust the anchor boxes for the next detection module.

Methods for handling scale variation.

As the most challenging problem in object detection, large scale variation across object instances hampers the accuracy of detectors. The multi-scale image pyramid is a common scheme to improve the detection methods, especially for objects of small and large scales. Based on the image pyramid strategy, SNIP proposes a scale normalization method to train objects that fall into the desired scale range for each resolution during multi-scale training. To perform multi-scale training more efficiently, SNIPER only selects context regions around the ground-truth instances and sampled background regions for each scale during training. However, SNIP and SNIPER still suffer from the inevitable increase of inference time.

Instead of taking multiple images as input, some methods utilize multi-level features of different spatial resolutions to alleviate scale variation. Methods like HyperNet and ION concatenate low-level and high-level features from different layers to generate better feature maps for prediction. Since the features from different layers usually have different resolutions, specific normalization or transformation operators need to be designed before fusing multi-level features. Instead, SSD and MS-CNN perform object detection at multiple layers for objects of different scales without feature fusion. TDM and FPN further introduce a top-down pathway and lateral connections to enhance the semantic representation of low-level features at bottom layers. PANet enhances the feature hierarchies in FPN by additional bottom-up path augmentation and proposes adaptive feature pooling to aggregate features from all levels for better prediction. Rather than using features from different layers, our proposed TridentNet generates scale-specific features through multiple parallel branches, thus endowing our network the same representational power for objects of different scales.

Dilated convolution.

Dilated convolution (aka Atrous convolution ) enlarges the convolution kernel with original weights by performing convolution at sparsely sampled locations, thus increasing the receptive field size without additional cost. Dilated convolution has been widely used in semantic segmentation to incorporate large-scale context information . In the object detection field, DetNet designs a specific detection backbone network to maintain the spatial resolution and enlarge the receptive field using dilated convolution. Deformable convolution further generalizes dilated convolution by learning the sampling location adaptively. In our work, we employ dilated convolution in our multi-branch architecture with different dilation rates to adapt the receptive fields for objects of different scales.

Investigation of Receptive Field

There are several design factors of the backbone network that may affect the performance of object detectors including downsample rate, network depth, and the receptive field. Several works have already discussed their impacts. The effects of the first two factors are straightforward: deeper network with low downsample rate may increase the complexity, but benefit the detection task in general. Nevertheless, as far as we know, there is no previous work that studies the impact of the receptive field in isolation.

To investigate the effect of the receptive field on the detection of objects with different scales, we replace some convolutions in the backbone network with their dilated variants . We use different dilation rates to control the receptive field of a network.

Dilated convolution with a dilation rate $d_{s}$ inserts $d_{s}-1$ zeros between consecutive filter values, enlarging the kernel size without increasing the number of parameters and computations. Specifically, a dilated 3 $\times$ 3 convolution could have the same receptive field as the convolution with the kernel size of $3+2(d_{s}-1)$ . Suppose the total stride of current feature map is $s$ , then a dilated convolution of rate $d_{s}$ could increase the receptive field of the network by $2(d_{s}-1)s$ . Thus if we modify $n$ conv layers with $d_{s}$ dilation rate, the receptive field could be increased by $2(d_{s}-1)sn$ .

We conduct our pilot experiment using a Faster R-CNN detector with the ResNet-C4 backbone on the COCO dataset. The results are reported in the COCO-style mmAP on all objects and objects of small, medium and large sizes, respectively. We use ResNet-50 and ResNet-101 as the backbone networks and vary the dilation rate $d_{s}$ of the 3 $\times$ 3 convolutions from 1 to 3 for the residual blocks in the conv4 stage.

Table 1 summarizes the results. We can find that as the receptive field increases (larger dilation rate), the performance of the detector on small objects drops consistently on both ResNet-50 and ResNet-101. While for large objects, the detector benefits from the increasing receptive fields. The above findings suggest that:

The performance on objects of different scales are influenced by the receptive field of a network. The most suitable receptive field is strongly correlated with the scale of objects.

Although ResNet-101 has a large enough theoretical receptive field to cover large objects (greater than 96 $\times$ 96 resolution) in COCO, the performance of large objects could still be improved when enlarging the dilation rate. This finding shares the same spirit in that the effective receptive field is smaller than the theoretical receptive field. We hypothesize that the effective receptive field of detection networks needs to balance between small and large objects. Increasing dilation rates enlarges the effective receptive field by emphasizing large objects, thus compromising the performance of small objects.

The aforementioned experiments motivate us to adapt the receptive field for objects of different scales as detailed in the next section.

Trident Network

In this section, we describe our scale-aware Trident Network (TridentNet) for object detection. The proposed TridentNet consists of weight sharing trident blocks and a deliberately designed scale-aware training scheme. Finally, We also present the inference details of TridentNet, including a fast inference approximation method.

Our goal is to inherit the merits of different receptive field sizes and avoid their drawbacks for detection networks. We propose a novel Trident architecture for this goal as shown in Figure 2. In particular, our method takes a single-scale image as input, and then creates scale-specific feature maps through parallel branches where convolutions share the same parameters but with different dilation rates.

We construct TridentNets by replacing some convolution blocks with the proposed trident blocks in the backbone network of a detector. A trident block consists of multiple parallel branches in which each shares the same structure with the original convolution block except the dilation rate.

Taking ResNet as an example, for a single residual block in the bottleneck style , which consists of three convolutions with kernel size 1 $\times$ 1, 3 $\times$ 3 and 1 $\times$ 1, a corresponding trident block is constructed as multiple parallel residual blocks with different dilation rates for the 3 $\times$ 3 convs, as shown in Figure 3. Stacking trident blocks allows us to control receptive fields of different branches in an efficient way similar to the pilot experiment in Section 3. Typically, we replace the blocks in the last stage of the backbone network with trident blocks since larger strides lead to a larger difference in receptive fields as needed. Detailed design choices could be referred in Section 5.2.

Weight sharing among branches.

An immediate problem of our multi-branch trident block is that it introduces several times parameters which may potentially incur overfitting. Fortunately, different branches share the same structure (except dilation rates) and thus make weight sharing straightforward. In this work, we share the weights of all branches and their associated RPN and R-CNN heads, and only vary the dilation rate of each branch.

The advantages of weight sharing are three-fold. It reduces the number of parameters and makes TridentNet need no extra parameters compared with the original detector. It also echoes with our motivation that objects of different scales should go through a uniform transformation with the same representational power. A final point is that transformation parameters could be trained on more object samples from all branches. In other words, the same parameters are trained for different scale ranges under different receptive fields.

2 Scale-aware Training Scheme

The proposed TridentNet architecture generates scale-specific feature maps according to the pre-defined dilation rates. However, the degradation in Table 1 caused by scale mismatching (e.g. small objects on the branch with too large dilation) still exists for every single branch. Thus, it is natural to detect objects of different scales on different branches. Here, we propose a scale-aware training scheme to improve the scale awareness of every branch and avoid training objects of extreme scales on mismatched branches.

Similar to SNIP , we define a valid range $[l_{i},u_{i}]$ for each branch $i$ . During training, we only select the proposals and ground truth boxes whose scales fall in the corresponding valid range of each branch. Specifically, for an Region-of-Interest (RoI) with width $w$ and height $h$ on the input image(before resize), it is valid for branch $i$ when:

This scale-aware training scheme could be applied on both RPN and R-CNN. For RPN, we select ground truth boxes which are valid for each branch according to Eq. 1 during anchor label assignment. Similarly, we remove all invalid proposals for each branch during the training of R-CNN.

3 Inference and Approximation

During inference, we generate detection results for all branches and then filter out the boxes which fall outside the valid range of each branch. We then use NMS or soft-NMS to combine the detection outputs of multiple branches and obtain the final results.

A major drawback of TridentNet is the slow inference speed due to its branching nature. Here we propose TridentNet Fast, a fast approximation of TridentNet with only one branch during inference. For a three-branch network as in Figure 2, we use the middle branch for inference since its valid range covers both large and small objects. In this way, TridentNet Fast incurs no additional time cost compared with a standard Faster R-CNN detector. Surprisingly, we find that this approximation only exhibits a slight performance drop compared with the original TridentNet. This may due to our weight-sharing strategy, through which multi-branch training is equivalent to within-network scale augmentation. Detailed ablation of TridentNet Fast could be found in Section 5.3.

Experiments

In this section, we conduct experiments on the COCO dataset . Following , we train models on the union of 80k training images and 35k subset of validation images (trainval35k), and evaluate on a set of 5k validation images (minival). We also report the final results on a set of 20k test images (test-dev). We first describe the implementation details of TridentNets and training settings in Section 5.1. We then conduct thorough ablation experiments to validate the proposed method in Section 5.2. Finally, Section 5.4 compares the results of TridentNets with state-of-the-art methods on the test-dev set.

We re-implement Faster R-CNN as our baseline method in MXNet . Following other standard detectors , the network backbones are pre-trained on the ImageNet . The stem, the first residual stage, and all BN parameters are freezeed. The input images are resized to a short side of 800. Random horizontal flip is adopted during training. By default, models are trained in a batch size of 16 on 8 GPUs. Models are trained in 12 epochs by default, with the learning rate starting from 0.02 and decreased by a factor of 0.1 after the 8th and 10th epoch. The $2\times$ or $3\times$ training schemes means doubling or tripling the total training epochs and learning rate schedules accordingly. We adopt the output of conv4 stage in ResNet as the backbone feature map and the conv5 stage as the R-CNN head in both baseline and TridentNet. If not otherwise specified, we adopt three branches as our default TridentNet structure. For each branch in TridentNet, the top 12000/500 proposals are kept before/after NMS and we sample 128 ROIs for training. The dilation rates are set to 1, 2 and 3 in three branches, respectively. When adopting scale-aware training scheme for TridentNet, we set the valid ranges of three branches as $,$ and $[90,\infty]$ , respectively.

For the evaluation, we report the standard COCO evaluation metric of Average Precision (AP) as well as $\text{AP}_{50}$ / $\text{AP}_{75}$ . We also report COCO-style $\text{AP}_{s}$ , $\text{AP}_{m}$ and $\text{AP}_{l}$ on objects of small (less than 32 $\times$ 32), medium (from 32 $\times$ 32 to 96 $\times$ 96) and large (greater than 96 $\times$ 96) sizes.

2 Ablation Studies

First, we analyze the importance of each component in TridentNet. The baseline methods (Table 2(a)) are evaluated on ResNet-101 and ResNet-101-Deformable backbones. Then we gradually apply our multi-branch architecture, weight sharing design, and scale-aware training scheme.

Multi-branch. Based on the pilot experiment, Table 2(b) evaluates a straightforward way to get the best of multiple receptive fields. This multi-branch variant improves over the baselines on AP for both ResNet-101 (from 37.9 to 39.0) and ResNet-101-Deformable (from 39.9 to 40.5), especially for large objects (2.3/1.2 increase). This indicates that even the simplest multi-branch design could benefit from different receptive fields.

Scale-aware. Table 2(d) shows the ablation results of adding scale-aware training based on multi-branch (Table 2(b)). It brings additional improvements (0.8/1.0 increase on ResNet-101/ResNet-101-Deformable) for small objects but drops in performance for large objects. We conjecture that the scale-aware training design prevents each branch from training objects of extreme scales, but may also bring about the over-fitting problem in each branch caused by reduced effective samples.

Weight-sharing. By applying weight sharing on multi-branch (Table 2(c)) and TridentNet (Table 2(e)), we could achieve consistent improvements on both two base networks. This demonstrates the effectiveness of weight-sharing. It reduces the number of parameters and improves the performance of detectors. With the help of weight-sharing (Table 2(e)), all branches share the same parameters which are fully trained on objects of all scales, thus alleviating the over-fitting issue in scale-aware training (Table 2(d)).

Finally, TridentNets achieve significant improvements (2.7/1.9 AP increase) on the two base networks. It also reveals that the proposed TridentNet structure is compatible with methods like deformable convolution which could adjust receptive field adaptively.

Number of branches.

We study the choice of the number of branches in TridentNets. Table 3 shows the results using one to four branches. Note that we do not add scale-aware training scheme here to avoid elaborately tuning valid ranges for different numbers of branches. The results in Table 3 demonstrate that TridentNets consistently improve over the single-branch method (baseline) with 2.7 to 3.4 AP increase. As can be noticed, four branches do not bring further improvement over three branches. Thus, considering the complexity and performance, we choose three branches as our default setting.

Stage of Trident blocks.

We conduct ablation study on TridentNet to find the best stage to place trident blocks in ResNet. Table 4 shows the results of applying trident blocks in conv2, conv3 and conv4 stages, respectively. The corresponding total strides are 4, 8 and 16. Comparing with conv4 stage, TridentNets on conv2 and conv3 stages achieve minor increase over the baseline. This is because the strides in conv2 and conv3 feature maps are not large enough to widen the discrepancy of receptive fields between three branches.

Number of trident blocks.

As the conv4 stage in ResNet has multiple residual blocks, we also conduct ablation study to explore how many trident blocks are needed for TridentNet. Here we replace different numbers of residual blocks with trident blocks on conv4 on ResNet-101. The results in Figure 4 show that when the number of trident blocks grows beyond 10, the performance of TridentNet becomes stable. This indicates the robustness of TridentNet with respect to the number of trident blocks, when the discrepancy of receptive fields between branches is large enough.

Performance of each branch.

In this section, we investigate the performance of each branch of our multi-branch TridentNet. We evaluate the performance of each branch independently without removing the detections out of the scale-aware range. Table 5 shows the results of every single branch and three branches combined. As expected, through scale-aware training, branch-1 with the smallest receptive field achieves good results on small objects, branch-2 works well on objects of the medium scale while branch-3 with the largest receptive field is good at large objects. Finally, the three-branch method inherits the merits from three single branches and achieves the best results.

3 Fast Inference Approximation

To reduce the inference time of TridentNet, we propose TridentNet Fast which uses a single major branch to approximate the three-branch results during inference. As indicated in Table 5, branch-2 emerges as a natural candidate for the major branch as its scale-aware range covers most objects. We investigate the effect of scale-aware ranges for scale-aware training in Table 6. As shown in Table 6(c), by enlarging the scale-aware range of the major branch to incorporate objects of all scales, the performance of Trident Fast improves by 1.5 AP over the default scale-aware range setting. Furthermore, extending scale-aware ranges for all three branches achieves the best performance of 40.0 AP, which is close to the original TridentNet result of 40.6 AP. We hypothesize this may due to the weight-sharing strategy. Since the weights of the major branch are shared on other branches, training all branches in the scale-agnostic scheme is equivalent to perform within-network multi-scale augmentation.

4 Comparison with State-of-the-Arts

In this section, we evaluate TridentNet on COCO test-dev set and compare with other state-of-the-art methods. Here we report the results of our methods under different settings in Table 7.

TridentNet, which is to directly apply our method on Faster R-CNN with ResNet-101 backbone in the 2 $\times$ training scheme, achieves 42.7 AP without bells and whistles.

To fairly compare with SNIP and SNIPER, we apply multi-scale training, soft-NMS, deformable convolutions, large-batch BN, and the 3 $\times$ training scheme on TridentNet and get TridentNet*. It gives an AP of 46.8, already surpassing image pyramid based SNIP and SNIPER in the single-scale testing setting. If we adopt the image pyramid for testing, it further improves the result of TridentNet* to 48.4 AP. To our best knowledge, for single models with ResNet-101 backbone, our result is the best entry among state-of-the-art methods. Furthermore, TridentNet* Fast + Image Pyramid achieves 47.6 AP.

In this section, we compare TridentNet with other popular scale handling methods like FPN and ASPP . FPN is the de facto model for handling scale variation in detection. ASPP is a special case of TridentNet with only one trident block and dilation rates of three branches set to (6, 12, 18), followed by a feature fusion operator. To fairly compare with FPN, we adopt a 2fc head instead of a conv5 head for models in this section. Table 8 compares these methods under the same training setting. TridentNet improves significantly over other methods on all scales. It shows the effectiveness of scale specific feature maps generated by TridentNet with the same set of parameters. Furthermore, TridentNet Fast achieves 41.0 AP which improves 1.2 AP over the baseline with no computation cost.

Conclusion

In this paper, we present a simple object detection method called Trident Network to build in-network scale-specific feature maps with the uniform representational power. A scale-aware training scheme is adopted for our multi-branch architecture to equip each branch with the specialized ability for corresponding scales. The fast inference method with the major branch makes TridentNet achieve significant improvements over baseline methods without any extra parameters and computations.

Acknowledge

This work was supported in part by the National Key R&D Program of China (No.2018YFB1402605), the Beijing Municipal Natural Science Foundation (No.Z181100008918010), the National Natural Science Foundation of China (No.61836014, No.61761146004, No.61773375, No.61602481). The authors would like to thanks NVAIL for the support.