OPANAS: One-Shot Path Aggregation Network Architecture Search for Object Detection

Introduction

Recognizing objects at vastly different scales is one of the major challenges in computer vision. To address this issue, great progress has been made in designing deep convolutional networks in the past few years. Intuitively, directly extracting feature pyramid from CNN at different stages provides an efficient solution. Each level of the feature pyramid corresponds to a specific scale in the original image. However, high-level features are with more semantics while the low-level ones are more content descriptive . Such a semantic gap is unable to deliver strong features for multi-scale visual recognition tasks (e.g., object detection, and segmentation). To alleviate the discrepancy, different feature fusion strategies have been proposed. Feature Pyramid Network (FPN) is arguably the most popular basic architecture and inspires many important variants. It adopts a backbone model, typically designed for image classification, and builds a top-down information flow by sequentially combining two adjacent layers in feature hierarchy in the backbone. By such design, low-level features are complemented by semantic information from high-level features. Despite simple and effective, FPN may not be the optimal architecture design.

Two lines of research have been conducted to advance FPN-based detection algorithms. On one hand, various approaches (e.g., PANet , BiFPN , Libra R-CNN and SEPC ) enrich FPN by aggregating multiple heterogeneous information paths and achieve impressive results. However, as shown in Fig. 1 (a-e), they only explore aggregations of up to three types of information paths, (\ie, top-down and bottom-up , top-down and fusing-splitting , and top-down and scale-equalizing ). Moreover, most of these methods follow a straightforward chain-style aggregation structure, except BiFPN that adds additional skip-connect on PANet with several repetitions, but remains in a simple topology. On the other hand, Neural Architecture Search (NAS)-based FPN architectures have achieved remarkable performance gain beyond manually designed architectures, but with following limitations: (1) inefficiency, the searching processes are often computationally expensive (e.g., 300 TPU days ) due to the extremely large search space, and (2) weak adaptability, their searched architectures are specialized for certain detector with special training skills (\eg, large batch size or longer training schedule).

Inspired by these studies and meanwhile to address aforementioned issues, we propose a new efficient and effective NAS framework, named OPANAS (One-Shot Path Aggregation Neural Architecture Search, see Fig. 2) to automatically search a better FPN for object detection. Firstly, we carefully design four parameterized information paths (top-down, bottom-up, scale-equalizing and fusing-splitting, see Fig. 3 (a-d)) and two parameter-free ones (skip-connect and none, see Fig. 3 (e-f)) to build our search space. Clearly, these six modules introduce different information flows, different connections between backbone and detection head, and lead to complementary and highly interpretable aggregation modules. Note that the four parameterized ones are relatively heavy and the two parameter-free ones are light-weighted, and they work together to achieve a promising accuracy-efficiency trade-off.

Secondly, to achieve the optimal aggregation of the six information paths, we propose a novel FPN search space, in which each FPN candidate is represented by a densely-connected directed acyclic graph (each node is a feature pyramid and each edge is a specific one of the six heterogeneous information paths as shown in Fig. 3). Notably, our search space contains richer aggregation topological structures of FPNs than existing methods as in Fig. 1, and hence enables richer cross-level and cross-module interactions.

Thirdly, we propose an efficient one-shot search method to search the optimal FPN architecture, that is, we first train a super-net and then search the optimal sub-net from the super-net with an evolutionary algorithm that has strong global optimum search capability. Experiments show that our method is efficient as the differentiable NAS methods, \ie, DARTS and Fair DARTS , while the searched FPN architecture can achieve better detection accuracy with less parameters and FLOPs. Moreover, following the simple vanilla training protocol, our searched FPN architecture can consistently improve the detection accuracy of the main-stream detectors including RetinaNet, Faster R-CNN and Cascade R-CNN by 2.3$\sim$3.2 mAP, with less parameters and FLOPs. These results demonstrate the efficacy of the proposed OPANAS for object detection.

We carefully design 6 information paths that can aggregate multi-level information, and thus enable the effective and complementary combination of low-, medium- and high-level information. To our knowledge, we are the first to investigate the aggregations of multiple ($>$3) information paths.

We propose a novel one-shot method, OPANAS, to efficiently and effectively search the optimal aggregation of the 6 kinds of information paths.

Working as a plug and play module, our searched architecture can easily be adapted to main-stream detectors including RetinaNet, Faster R-CNN and Cascade R-CNN, and significantly improve their detection accuracy by 2.3$\sim$3.2 $\%$ mAP. Notably, we achieve a new state-of-the-art accuracy-speed trade-off (52.2 $\%$ mAP at 7.6 FPS).

Related Work

Existing deep learning-based detectors can be briefly categorized into two streams: one-stage detectors such as SSD and RetinaNet , which utilize CNN directly to predict the bounding boxes; and two-stage methods such as Faster R-CNN and Mask R-CNN , which generate the the final detection results after extracting region proposals upon a region proposal network (RPN). Although encouraging signs of progress have been made, existing detectors are still suffering from the problems caused by the scale variation across object instances. The feature pyramid is popularly used to deal with scale variation , which introduces a top-down information flow.

Beyond FPN, some recent extensions employ two or three types of information paths. For example, PANet introduces an extra bottom-up path after the top-down path of classic FPN , and Libra R-CNN adopts Non-Local module to fuse the features produced by the classic FPN and then transfers the fused feature into multi-scale pyramid features. Multi-level FPN first fuses the backbone features as the base feature and then introduces multiple U-shape modules to extract multi-level pyramid features and builds a powerful one-stage detector. SEPC stacks 4 scale equalizing modules behind classic FPN to enhance cross-scale correlation. More recently, BiFPN exploits a simplified architecture of PANet and stacks it repeatedly with skip-connect to build a more powerful one-stage detector named EfficientDet. Though promising results are achieved by EfficientDet, its training cost is extremely expensive, i.e., large batch-size (128 on 32TPU) with a long training schedule (300 or 500 epochs). Generally speaking, these FPNs suffer from intrinsic architecture limitations since they only aggregate at most three types of information paths with naive topological structure.

2 Neural Architecture Search

More recently, neural architecture search (NAS) is applied to automatically search an FPN architecture for a specific detector. NAS-FPN , NAS-FCOS and SpineNet use reinforcement learning to control the architecture sampling and obtain promising results. SM-NAS uses evolutionary algorithm and partial order pruning method to search the optimal combination of different parts of the detectors. The above NAS methods are effective though can be time-consuming. Auto-FPN , Hit-Detector uses gradient-based method to search the optimal detector, which can significantly reduce searching time. However, gradient-based methods tend to trap into local minima in certain nodes of super-net during the progress of optimization and introduce further complexity . Recently, researchers propose one-shot method to decouple the super-net training and architecture search in two sequential steps. DetNAS follows this idea to search an efficient backbone for object detection. One limitation of the single-path approach is that the search space is restricted to a sequential structure as shown in Fig. 2.1.

The aforementioned methods take the layer-wise operations as transform blocks (i.e, single-scale feature as nodes), which are completely separated from manual design. Such design forms a large search space which contains architectures beyond human design, while also includes many poor-performing architectures, leading to low search efficiency. To reduce the post-processing overhead, we propose multi-level information path aggregation as our search space. With the help of carefully designed information paths, our search can be efficient and robust.

Methodology

In this work, we first propose six types of information paths, which capture diverse multi-level information. Second, to search the optimal aggregations of these information paths, we introduce an efficient One-Shot Path Aggregation Network Architecture Search (OPANAS) algorithm. Last, we detail the optimization and searching process.

To effectively aggregate different levels of pyramidal features, we propose 6 information paths, which can capture low-, medium- and high-level information. Similar to classic FPN , these information paths map the input pyramidal features $\{P_{2},P_{3},P_{4},P_{5}\}$ (see Fig. 3) to $\{F_{2},F_{3},F_{4},F_{5}\}$. However, the proposed information paths can capture much richer and diverse information than FPN, which will be described as following.

The top-down information path is modified from the classic FPN (Fig. 3 (a)). For this path, the output pyramidal features (denoted as $F_{2}^{t},F_{3}^{t},F_{4}^{t},F_{5}^{t}$) are sequentially constructed in a top-down manner, i.e., the smaller scale (high-level, e.g., $F_{5}^{t}$) feature map is constructed first. Specifically, each feature map ($F_{i}^{t}$) is iteratively built by combining input pyramid feature map of the same level ($P_{i}$) and the higher-level output feature ($F_{i+1}^{t}$):

where $\textit{U}(\cdot)$ denotes upsampling with factor of 2. For high-level features $(i=3,4,5)$, $\mathbf{W}_{i}^{t}$ is the $3\times 3$ deformable convolution filter to alleviate discrepancy of a feature pyramid , and $\mathbf{W}_{2}^{t}$ is a normal $3\times 3$ convolution filter.

For the bottom-up information path, the output pyramidal features (denoted as $F_{2}^{b},F_{3}^{b},F_{4}^{b},F_{5}^{b}$) are sequentially constructed in a bottom-up manner, i.e., the large scale (low-level, e.g., $F_{2}^{b}$) feature map is constructed first as shown in Fig. 3 (b). Each feature map ($F_{i}^{b}$) is obtained by merging the input feature maps ($P_{i}$) of the same level, and the output feature map below it ($F_{i-1}^{b}$):

where $\textit{D}(\cdot)$ denotes downsampling with factor of 2 and $\mathbf{W}_{i}^{b}$ is the convolution filter with the same setting as above.

The scale-equalizing information path is motivated by SEPC , which stacks scale-equalizing pyramid convolutions after the classic FPN to capture inter-scale correlation. Here we take a single pyramid convolution operation as an information path. As shown in Fig. 3 (c), each feature map ($F_{i}^{s}$) is obtained by merging the adjacent-level input feature maps ($P_{i}$):

where $\mathbf{W}^{s}_{1},\mathbf{W}^{s}_{0},\mathbf{W}^{s}_{-1}$ are $3\times 3$ deformable convolution filters and the stride of $\mathbf{W}^{s}_{-1}$ is set to 2 to down-sample.

We design a two-step fusing-splitting information path, which first combines the high- and low-level input pyramidal features, and then splits the combined features to multi-scale output features in Fig. 3 (d). In practice, the highest two input feature maps are merged into $\alpha_{s}$, and the lowest two are merged into $\alpha_{l}$ through an element-wise sum:

After obtaining the combined features, we fuse them through concatenation,

Specially, we add a skip-connect path to perform identity mapping. Moreover, a ‘none’ information path is exploited to remove redundant information paths. These two parameter-free information paths are designed to reduce the complexity of the model, leading to a better accuracy-efficiency trade-off.

2 One-Shot Search

We propose a one-shot search method to efficiently and effectively search the optimal aggregation of the above six types of information paths. Specifically, we first construct a super-net $\mathcal{A}$, which is a fully-connected Multigraph DAG (directed acyclic graph). The node of DAG stands for feature maps (in the way of a feature pyramid), and there are six edges of different types between two nodes, and each edge represents one information path. The whole optimization includes two steps: (i) super-net training and (ii) optimal sub-net search, as shown in Fig. 2. For (i), we train the super-net until convergence (optimization of the weights of the super-net) using a fair sampling strategy detailed in Section 3.2.1. The weights of super-net are fixed once this training is done (one-shot optimization). For (ii), we use evolutionary algorithm (EA) to search for the optimal sub-net $a^{*}$, which is a DAG with only one optimal edge between two nodes. Obviously, the optimal sub-net represents the desired optimal FPN aggregating multiple information paths. Note that (ii) is very efficient because each sampled sub-net $a$ just goes through the inference process by using the weights of the super-net trained in (i). This is the main reason why one-shot optimization is very efficient.

To detail the optimization process, we first introduce the components of the super-net. The super-net is a DAG consists of $N+2$ nodes ($N$ is a predefined constant value), where the input node $P$ represents the extracted feature from the backbone, and the output node $O$ is the final output feature pyramid. Similarly, intermediate nodes ${x}_{i}(i=1,2,...,N)$ are also feature pyramids. Each directed edge $(i,j)$ is associated with some information path $\mathbf{IP}(i,j)$ that transforms ${x}_{i}$ to $x_{j}$. We assume the intermediate nodes are fully connected with former nodes, and identity mapped to the output node through summation. In such DAG model, each node $i\in\{1,2,\ldots,N\}$ aggregates inputs from previous nodes, where

In this way, OPANAS allows $6^{{N(N+1)}/2}$ possible DAGs without considering graph isomorphism with $N$ intermediate nodes. In particular, to maximize the search space without affecting the convergence, we set $N=5$, and the total number of sub-nets is approximately $6^{15}\approx 10^{12}$.

Second, we formulate the super-net training as following. The architecture space $\mathcal{A}$ is encoded in a super-net, denoted as $\mathcal{N}(\mathcal{A},W)$, where $W$ stands for the weights of super-net. Thus, the super-net training can be formulated as:

This super-net training is detailed in Section 3.2.1.

Third, we discuss the optimal sub-net search in (ii). We aim to search the optimal sub-net $a^{*}\in\mathcal{A}$ that maximizes the validation accuracy, which can be formulated as:

We use an evolutionary algorithm to conduct this optimal sub-net search detailed in Section 3.2.2.

Unlike the existing One-Shot method SPOS only having edges between adjacent nodes, our OPANAS is densely connected to explore richer topological structures for aggregation. To adapt to our multiple paths (edges) optimization, we associate an edge importance weight to each edge. To guarantee the consistency between training and test, we set these weights to be continuous. Consequently, each node $i\in\{1,2,\ldots,N\}$ aggregates weighted inputs from the previous nodes, then Eq. (7) can be formulated as:

where $\gamma_{i,j}$ denotes the edge importance weight between node $i$ and $j$. Concomitantly, the optimization in Eq. (8) is modified as:

To assist the convergence of model, we add $L_{1}$ regularization to these edge importance weights with a hyper-parameter $\mu$ to balance with the original bounding box loss. Thus the total loss function is:

$\mathcal{L}_{\rm cls},\mathcal{L}_{\rm loc}$ are objective functions corresponding to recognition and localization task respectively.

Instead of training the whole super-net directly, we sample $K$ sub-nets per training iteration to reduce the GPU memory cost. Note that ‘skip-connection’ and ‘none’ are parameter-free and do not require any optimization. Hence, they are only considered during the searching process. Consequently, in super-net training, only $K=4$ types of information paths are involved. To alleviate training unfairness between the $K$ parameterized information paths, we adopt strict fair sampling strategy in our super-net training. To be more specific, in the $n$-th super-net training step, $K$ sub-nets are sampled with no intersection. That is, each edge of them is associated with different information path, and the weights of the super-net are updated after accumulating gradients from the $K$ sampled sub-nets. By this sampling strategy, all information paths are ensured to be equally sampled and trained within each training step, and each edge is activated only once within each training step. Consequently, the expectation and variance of edge $E_{i}$ with information path $i$ ($i=0,1,2$ and $3$ correspond to top-down, bottom-up, scale-equalizing and fusing-splitting, respectively) are given by,

The variance does not change with $n$, thus fairness is assured at every training step.

2.2 Sub-net Search with Evolutionary Algorithm

We conduct the sub-net search with an evolutionary algorithm. Specifically, during the optimal sub-net search in Fig. 2 (b), we first randomly sample $N_{S}$ sub-nets, each passes the coarse search, from the super-net and rank their performance. Note that evaluating a sub-net requires only inference without training, which makes the search very efficient. Then we repeatedly generate new sub-nets through crossover and mutation on top $k$ performing sub-nets. Following an evolutionary algorithm , crossover denotes that two randomly selected sub-nets are crossed to produce a new one, mutation means a randomly selected sub-net mutates its every edge with probability 0.1 to produce a new sub-net. In this work, we set population size $N_{S}=50$, max iterations $T=12$ and $k=10$.

Experiments

We conduct experiments on the COCO and PASCAL VOC benchmarks. For COCO, the training is conducted on the 118k training images, and ablation studies are evaluated on the 5k minival images. We also report the results on the 20k images in test-dev for comparison with state-of-the-art (SOTA). For evaluation, we adopt the metrics from the COCO detection evaluation criteria, including the mean Average Precisions (mAP) across IoU thresholds ranging from 0.5 to 0.95 at different scales. For PASCAL VOC, training is performed on the union of VOC 2007 trainval and VOC 2012 trainval (10K images) and evaluation is performed on VOC 2007 test (4.9K images), mAP with an IoU threshold of 0.5 is used for evaluation.

1.2 Super-net Training and Sub-net Searching Phase

We consider a total of $N=5$ intermediate nodes for super-net training and optimal sub-net searching. We choose Faster R-CNN (ResNet50 ) as the baseline. During super-net training, we use input-size $800\times 500$ and sample $1/5$ images from training-set of COCO to further reduce the search cost. As for PASCAL VOC, the input-size is set to $384\times 384$. We use SGD optimizer with initial learning rate 0.02, momentum 0.9, and $10^{-4}$ as weight decay. We train super-net for 12 epochs with batch-size of 16. Edge importance weight $\gamma$ is initialized as 1, and hyper-parameter $\mu$ is $10^{-4}$. In total, the whole search phase is completed in 4 days (1 day for super-net training and 3 days for optimal sub-net search) using 1 V100 GPU.

1.3 Full Training Phase

In this phase, we fully train the searched model.SGD is performed to train the full model with batch-size of 16. The initial learning rate is 0.02; $10^{-4}$ as weight decay; 0.9 as momentum. Single-scale training with input $1333\times 800$ size is trained for 12 epochs, and the learning rate is decreased by 0.1 at epoch 8 and 11. While multi-scale training (pixel size=$400\sim 1400$) is trained for 24 epochs with learning rate decreased by 0.1 at epoch 16 and 22. We use single-scale training for ablation studies if not specified, and we compare with SOTA with multi-scale training.

2 Results

The searched optimal architecture by OPANAS, denoted as OPA-FPN, is illustrated in Fig. 2 (c). It is evaluated together with state-of-the-art detectors including hand-crafted and NAS-based ones . Note that these methods search different components of detectors. Specifically, SM-NAS searches the overall architecture of Cascade R-CNN, NAS-FPN searches the neck architecture, and Auto-FPN searches the architectures of both neck (FPN) and detection head. As shown in Tab. 1, compared with representative results achieved by these SOTA methods, our method achieves better or very competitive results in terms of amount of parameters, computation complexity, accuracy, and inference speed. Notably, our method can search for the best neck architecture more efficiently, e.g., 4 GPU days on COCO. Specially, our searched OPA-FPN equipped with Cascade R-CNN Res2Net101-DCN achieves a new state-of-the-art accuracy-speed trade-off (52.2 $\%$ mAP at 7.6 FPS), outperforming SpineNet (the SOTA NAS based method) and EfficientDet (based on the SOTA NAS searched backbone). These results demonstrate the effectiveness of our carefully designed search space and the efficiency of the search algorithm.

2.2 Model Adaptability for Main-stream Detectors

To further verify the performance of the OPA-FPN on main-stream detectors, we adapt it to RetinaNet , Faster R-CNN and Cascade R-CNN in Tab. 2. Under the same training strategy with baseline, we obtain a lighter model with better performance on each detector: improving RetinaNet by 2.3$\%$ mAP with 13$\%$ FLOPs decreasing, and improving Cascade R-CNN by 2.5$\%$ mAP with 27$\%$ parameter amount decreasing. Moreover, comparing with other hand-craft FPNs, our architecture achieves clearly better results in terms of amount of parameters, computation complexity, accuracy.

2.3 Model Transferability Between COCO and VOC

To evaluate the transferability of our architecture on different datasets, we transfer the searched architecture between COCO and VOC with multi-scale training, as shown in Tab. 3. With the architecture searched on COCO, our method boosts the performance by $3.0\%$ mAP for COCO, $2.8\%$ mAP for VOC, respectively. When searched on VOC, our method boosts the performance by $3.0\%$ mAP for VOC, $2.9\%$ mAP for COCO, respectively. No matter which dataset we search on, our architecture performs better than Auto-FPN (e.g., $41.6\%$ vs.$40.5\%$ in terms of mAP) with fewer computation cost (e.g., FLOPs 128G vs. 260G). These results further demonstrate the effectiveness of our method.

3 Ablation Study

To demonstrate the effectiveness of aggregating different information paths in our searched OPA-FPN, we first compare it with the architectures using single information path in Tab. 4. We choose the original Faster R-CNN (ResNet50) + vanilla FPN as the baseline, and we adjust the channel dimensions of our searched OPA-FPN and detection head, aiming to align the complexity with the baseline. OPA-FPN significantly surpasses the baseline and the single-information-path architectures, with fewer FLOPs/Params (e.g, FLOPs 197G vs. 207G, Params 35.5M vs. 41.5M), achieving an effective aggregation and exploration of information paths.

3.2 Edge Importance Weighting

To verify the effectiveness of the edge importance weight $\gamma$ in Eq. (10), we illustrate the intermediate results of training super-net with fair sampling in Fig. 4 (a,b), and observe that the edge importance weighting brings clear benefits for the training of super-net. These results prove that distinguishing the importance of different edges is effective for densely connected super-net training.

3.3 Correlation Analysis

Recently, the effectiveness of weight sharing-based NAS methods is questioned because of the lack of (1) fair comparison on the same search space and (2) adequate analysis on the correlation between the super-net performance and the stand-alone sub-net model performance . Here we adopt Kendall Tau to measure the correlation of model ranking obtained from super-net. Specifically, we randomly sample 15 sub-nets from the trained super-net and conduct full train to evaluate their performance. In Tab. 5, when changing from single-path super-net used by SPOS to our densely connected super-net, there is a drop in correlation but the detection accuracy increases. However, by adopting fair sampling and edge importance weight, we achieve a much higher correlation value, showing that our method can achieve a higher correlation between super-net and sub-net with the fair sampling and the proposed edge importance weighting.

3.4 Comparisons with More NAS Baselines

We further compare our method with more existing NAS methods, including a) Random Search: we randomly sample 15 architectures from the proposed search space and conduct full training under the same training setting in our experiments; b) SPOS: we train the single-path one-shot FPN super-net and perform EA search strategy following ; c) DARTS: a very popular differentiable NAS method ; and d) Fair DARTS: an improved version of DARTS with softmax relaxation and zero-one loss . As the results reported in Tab. 6, compared with other NAS methods, our method can find a better architecture with less or comparable time. These results prove the effectiveness of our super-net training and optimal sub-net search strategy.

Conclusion

In this paper, we propose One-Shot Path Aggregation Network Architecture Search (OPANAS), which consists of a novel search space and an efficient searching algorithm, to automatically find an effective FPN architecture for visual object detection. In particular, we introduce six types (i.e., top-down, bottom-up and scale-equalizing, fusing-splitting, balanced, skip-connect, and none) of information paths as candidate operations, and exploit densely connected DAG to represent FPN to aggregate them. An efficient one-shot search method is further invented to search the optimal FPN, based on the super-net training with fair sampling and edge importance weighting. Extensive experimental results demonstrate the superiority of the proposed OPANAS in both efficiency and effectiveness.

References