Accelerating Very Deep Convolutional Networks for Classification and Detection

Xiangyu Zhang, Jianhua Zou, Kaiming He, Jian Sun

Introduction

The accuracy of convolutional neural networks (CNNs) has been continuously improving , but the computational cost of these networks also increases significantly. For example, the very deep VGG models , which have witnessed great success in a wide range of recognition tasks , are substantially slower than earlier models . Real-world systems may suffer from the low speed of these networks. For example, a cloud service needs to process thousands of new requests per seconds; portable devices such as phones and tablets may not afford slow models; some recognition tasks like object detection and semantic segmentation need to apply these models on higher-resolution images. It is thus of practical importance to accelerate test-time performance of CNNs.

There have been a series of studies on accelerating deep CNNs . A common focus of these methods is on the decomposition of one or a few layers. These methods have shown promising speedup ratios and accuracy on one or two layers and whole (but shallower) models. However, few results are available for accelerating very deep models (e.g., $\geq$ 10 layers). Experiments on complex datasets such as ImageNet are also limited - e.g., the results in are about accelerating a single layer of the shallower AlexNet . Moreover, performance of the accelerated networks as generic feature extractors for other recognition tasks remain unclear.

It is nontrivial to speed up whole, very deep models for complex tasks like ImageNet classification. Acceleration algorithms involve not only the decomposition of layers, but also the optimization solutions to the decomposition. Data (response) reconstruction solvers based on stochastic gradient descent (SGD) and backpropagation work well for simpler tasks such as character classification , but are less effective for complex ImageNet models (as we will discussed in Sec. 4). These SGD-based solvers are sensitive to initialization and learning rates, and might be trapped into poorer local optima for regressing responses. Moreover, even when a solver manages to accelerate a single layer, the accumulated error of approximating multiple layers grow rapidly, especially for very deep models. Besides, the layers of a very deep model may exhibit a great diversity in filter numbers, feature map sizes, sparsity, and redundancy. It may not be beneficial to uniformly accelerate all layers.

In this paper, we present an accelerating method that is effective for very deep models. We first propose a response reconstruction method that takes into account the nonlinear neurons and a low-rank constraint. A solution based on Generalized Singular Value Decomposition (GSVD) is developed for this nonlinear problem, without the need of SGD. Our explicit treatment of the nonlinearity better models a nonlinear layer, and more importantly, enables an asymmetric reconstruction that accounts for the error from previous approximated layers. This method effectively reduces the accumulated error when multiple layers are approximated sequentially. We also present a rank selection method for adaptively determining the acceleration of each layer for a whole model, based on their redundancy.

In experiments, we demonstrate the effects of the nonlinear solution, asymmetric reconstruction, and whole-model acceleration by controlled experiments of a 10-layer model on ImageNet classification . Furthermore, we apply our method on the publicly available VGG-16 model , and achieve a 4 $\times$ speedup with merely a 0.3% increase of top-5 center-view error.

The impact of the ImageNet dataset is not merely on the specific 1000-class classification task; deep models pre-trained on ImageNet have been actively used to replace hand-engineered features, and have showcased excellent accuracy for challenging tasks such as object detection and semantic segmentation . We exploit our method to accelerate the very deep VGG-16 model for Fast R-CNN object detection. With a 4 $\times$ speedup of all convolutions, our method has a graceful degradation of 0.8% mAP (from 66.9% to 66.1%) on the PASCAL VOC 2007 detection benchmark .

A preliminary version of this manuscript has been presented in a conference . This manuscript extends the initial version from several aspects to strengthen our method. (1) We demonstrate compelling acceleration results on very deep VGG models, and are among the first few works accelerating very deep models. (2) We investigate the accelerated models for transfer-learning-based object detection , which is one of the most important applications of ImageNet pre-trained networks. (3) We provide evidence showing that a model trained from scratch and sharing the same structure as the accelerated model is inferior. This discovery suggests that a very deep model can be accelerated not simply because the decomposed network architecture is more powerful, but because the acceleration optimization algorithm is able to digest information.

Related Work

Methods for accelerating test-time computation of CNNs in general have two components: (i) a layer decomposition design that reduces time complexity, and (ii) an optimization scheme for the decomposition design. Although the former (“decomposition”) attracts more attention because it directly addresses the time complexity, the latter (“optimization”) is also essential because not all decompositions are similarly easy to fine good local optima.

The method of Denton et al. is one of the first to exploit low-rank decompositions of filters. Several decomposition designs along different dimensions have been investigated. This method does not explicitly minimize the error of the activations after the nonlinearity, which is influential to the accuracy as we will show. This method presents experiments of accelerating a single layer of an OverFeat network , but no whole-model results are available.

Jaderberg et al. present efficient decompositions by separating $k\times k$ filters into $k\times 1$ and $1\times k$ filters, which was earlier developed for accelerating generic image filters . Channel-wise dimension reduction is also considered. Two optimization schemes are proposed: (i) “filter reconstruction” that minimizes the error of filter weights, and (ii) “data reconstruction” that minimizes the error of responses. In , conjugate gradient descent is used to solve filter reconstruction, and SGD with backpropagation is used to solve data reconstruction. Data reconstruction in demonstrates excellent performance on a character classification task using a 4-layer network. For ImageNet classification, their paper evaluates a single layer of an OverFeat network by “filter reconstruction”. But the performance of whole, very deep models in ImageNet remains unclear.

Concurrent with our work, Lebedev et al. adopt “CP-decomposition” to decompose a layer into five layers of lower complexity. For ImageNet classification, only a single-layer acceleration of AlexNet is reported in . Moreover, Lebedev et al. report that they “failed to find a good SGD learning rate” in their fine-tuning, suggesting that it is nontrivial to optimize the factorization for even a single layer in ImageNet models.

Despite some promising preliminary results that have been obtained in the above works , the whole-model acceleration of very deep networks for ImageNet is still an open problem.

Besides the research on decomposing layers, there have been other streams on improving train/test-time performance of CNNs. FFT-based algorithms are applicable for both training and testing, and are particularly effective for large spatial kernels. On the other hand, it is also proposed to train “thin” and deep networks for good trade-off between speed and accuracy. Besides reducing running time, a related issue involving memory conservation has also attracted attention.

Approaches

Our method exploits a low-rank assumption for decomposition, following the stream of . We show that this decomposition has a closed-form solution (SVD) for linear neurons, and a slightly more complicated solution (GSVD ) for nonlinear neurons. The simplicity of our solver enables an asymmetric reconstruction method for reducing accumulated error of very deep models.

Our assumption is that the filter response at a pixel of a layer approximately lies on a low-rank subspace. A resulting low-rank decomposition reduces time complexity. To find the approximate low-rank subspace, we minimize the reconstruction error of the responses.

The complexity of using Eqn.(3) is $O(d^{\prime}k^{2}c)+O(dd^{\prime})$ , while the complexity of using Eqn.(1) is $O(dk^{2}c)$ . For many typical models/layers, we usually have $O(dd^{\prime})\ll O(d^{\prime}k^{2}c)$ , so the computation in Eqn.(3) will reduce the complexity to about $d^{\prime}/d$ .

In practice the low-rank assumption does not strictly hold, and the computation in Eqn.(3) is approximate. To find an approximate low-rank subspace, we optimize the following problem:

How good is the low-rank assumption? We sample the responses from a CNN model (with 7 convolutional layers, detailed in Sec. 4) trained on ImageNet. For the responses of each layer, we compute the eigenvalues of their covariance matrix and then plot the sum of the largest eigenvalues (Fig. 2). We see that substantial energy is in a small portion of the largest eigenvectors. For example, in the Conv2 layer ( $d=256$ ) the first 128 eigenvectors contribute over 99.9% energy; in the Conv7 layer ( $d=512$ ), the first 256 eigenvectors contribute over 95% energy. This indicates that we can use a fraction of the filters to precisely approximate the original filters.

2 Nonlinear Case

Next we investigate the case of using nonlinear units. We use $r(\cdot)$ to denote the nonlinear operator. In this paper we focus on the Rectified Linear Unit (ReLU) : $r(\cdot)=\max(\cdot,0)$ .

Driven by Eqn.(4), we minimize the reconstruction error of the nonlinear responses:

The above optimization problem is challenging due to the nonlinearity and the low-rank constraint. To find a feasible solution, we relax it as:

This problem appears similar to Eqn.(4) except that there are two sets of responses.

then $z_{ij}=\arg\min_{z_{0},z_{1}}(r(y_{ij})-r(z_{ij}))^{2}+\lambda(z_{ij}-y^{\prime}_{ij})^{2}$ . Our method is also applicable for other types of nonlinearities. The subproblem in (9) is a 1-dimensional nonlinear least squares problem, so can be solved by gradient descent for other $r(\cdot)$ .

In experiments, we find that it is sufficient to randomly sample 3,000 images to solve Eqn.(5). It only takes our method 2-5 minutes in MATLAB solving a layer. This is much faster than SGD-based solvers.

3 Asymmetric Reconstruction for Multi-Layer

When each layer is approximated independently, the error of shallower layers will be rapidly accumulated and affect deeper layers. We propose an asymmetric reconstruction method to alleviate this problem.

4 Rank Selection for Whole-Model Acceleration

In the above, the optimization is based on a target $d^{\prime}$ of each layer. $d^{\prime}$ is the only parameter that determines the complexity of an accelerated layer. But given a desired speedup ratio of the whole model, we need to determine the proper rank $d^{\prime}$ used for each layer. One may adopt a uniform speedup ratio for each layer. But this is not an optimal solution, because the layers are not equally redundant.

We empirically observe that the PCA energy after approximations is roughly related to the classification accuracy. To verify this observation, in Fig. 3 we show the classification accuracy (represented as the difference to no approximation) vs. the PCA energy. Each point in this figure is empirically evaluated using a reduced rank $d^{\prime}$ . 100% energy means no approximation and thus no degradation of classification accuracy. Fig. 3 shows that the classification accuracy is roughly linear on the PCA energy.

To simultaneously determine the reduced ranks of all layers, we further assume that the whole-model classification accuracy is roughly related to the product of the PCA energy of all layers. More formally, we consider this objective function:

Here $\sigma_{l,a}$ is the $a$ -th largest eigenvalue of the layer $l$ , and $\sum_{a=1}^{d^{\prime}_{l}}{\sigma_{l,a}}$ is the PCA energy of the largest $d^{\prime}_{l}$ eigenvalues in the layer $l$ . The product $\prod_{l}$ is over all layers to be approximated. The objective $\mathcal{E}$ is assumed to be related to the accuracy of the approximated whole network. Then we optimize this problem:

Here $d_{l}$ is the original number of filters in the layer $l$ , and $C_{l}$ is the original time complexity of the layer $l$ . So $\frac{d^{\prime}_{l}}{d_{l}}C_{l}$ is the complexity after the approximation. $C$ is the total complexity after the approximation, which is given by the desired speedup ratio. This optimization problem means that we want to maximize the accumulated energy subject to the time complexity constraint.

The problem in (14) is a combinatorial problem . So we adopt a greedy strategy to solve it. We initialize $d^{\prime}_{l}$ as $d_{l}$ , and consider the set $\{\sigma_{l,a}\}$ . In each step we remove an eigenvalue $\sigma_{l,d^{\prime}_{l}}$ from this set, chosen from a certain layer $l$ . The relative reduction of the objective is $\triangle\mathcal{E}/\mathcal{E}=\sigma_{l,d^{\prime}}/{\sum_{a=1}^{d^{\prime}_{l}}\sigma_{l,a}}$ , and the reduction of complexity is $\triangle C={\frac{1}{d_{l}}C_{l}}$ . Then we define a measure as $\frac{\triangle\mathcal{E}/\mathcal{E}}{\triangle C}$ . The eigenvalue $\sigma_{l,d^{\prime}_{l}}$ that has the smallest value of this measure is removed. Intuitively, this measure favors a small reduction of $\triangle\mathcal{E}/\mathcal{E}$ and a large reduction of complexity $\triangle C$ . This step is greedily iterated, until the constraint of the total complexity is achieved.

5 Higher-Dimensional Decomposition

On the other hand, compared with decomposition methods that operate on multiple dimensions (spatial and channel) , our method has to use a smaller $d^{\prime}$ to approach a given speedup ratio, which might limit the accuracy of our method. To avoid $d^{\prime}$ being too small, we further propose to combine our solver with Jaderberg et al.’s spatial decomposition. Thanks to our asymmetric reconstruction, our method can effectively alleviate the accumulated error for the multi-decomposition.

To determined the decomposed architecture (but not yet the weights), we first use our method to decompose all conv layers of a model. This involves the rank selection of $d^{\prime}$ for all layers. Then we apply Jaderberg et al.’s method to further decompose the resulting $k\times k$ layers ( $k>1$ ) into $k\times 1$ and $1\times k$ filters. The first $k\times 1$ layer has $d^{\prime\prime}$ output channels depending on the speedup ratio. In this way, an original layer of ( $k\times k$ , $d$ ) is decomposed into three layers of ( $k\times 1$ , $d^{\prime\prime}$ ), ( $1\times k$ , $d^{\prime}$ ), and ( $1\times 1$ , $d$ ). For a speedup ratio $r$ , we let each method contribute a speedup of $\sqrt{r}$ .

6 Fine-tuning

With any approximated whole model, we may “fine-tune” this model end-to-end in the ImageNet training data. This process is similar to training a classification network with the approximated model as the initialization.

However, we empirically find that fine-tuning is very sensitive to the initialization (given by the approximated model) and the learning rate. If the initialization is poor and the learning rate is small, the fine-tuning is easily trapped in a poor local optimum and makes little progress. If the learning rate is large, the fine-tuning process behaves very similar to training the decomposed architecture “from scratch” (as we will discuss later). A large learning rate may jump out of the initialized local optimum, and the initialization appears to be “forgotten”.

Fortunately, our method has achieved very good accuracy even without fine-tuning as we will show by experiments. With our approximated model as the initialization, the fine-tuning with a sufficiently small learning rate is able to further improve the results. In our experiments, we use a learning rate of 1e-5 and a mini-batch size of 128, and fine-tune the models for 5 epochs in the ImageNet training data.

We note that in the following the results are without fine-tuning unless specified.

Experiments

We comprehensively evaluate our method on two models. The first model is a 10-layer model of “SPPnet (OverFeat-7)” in , which we denote as “SPP-10”. This model (detailed in Table I) has a similar architecture to the OverFeat model but is deeper. It has 7 conv layers and 3 fc layers. The second model is the publicly available VGG-16 model www.robots.ox.ac.uk/~vgg/research/very_deep/ that has 13 conv layers and 3 fc layers. SPP-10 won the 3-rd place and VGG-16 won the 2-nd place in ILSVRC 2014 .

We evaluate the “top-5 error” using single-view testing. The view is the center $224\times 224$ region cropped from the resized image whose shorter side is 256. The single-view error rate of SPP-10 is 12.51% on the ImageNet validation set, and VGG-16 is 10.09% in our testing (which is consistent with the number reported by http://www.vlfeat.org/matconvnet/pretrained/). These numbers serve as the references for the increased error rates of our approximated models.

We first evaluate the effect of our each step on the SPP-10 model by a series of controlled experiments. Unless specified, we do not use the 3-d decomposition.

In this subsection we evaluate the single-layer performance. When evaluating a single approximated layer, the remaining layers are unchanged and not approximated. The speedup ratio (involving that single layer only) is shown as the theoretical ratio computed by the complexity.

In Fig. 4 we compare the performance of our linear solution (4) and nonlinear solution (6). The performance is displayed as increase of error rates (decrease of accuracy) vs. the speedup ratio of that layer. Fig. 4 shows that the nonlinear solution consistently performs better than the linear solution. In Table I, we show the sparsity (the portion of zero activations after ReLU) of each layer. A zero activation is due to the truncation of ReLU. The sparsity is over 60% for Conv2-7, indicating that the ReLU takes effect on a substantial portion of activations. This explains the discrepancy between the linear and nonlinear solutions. Especially, the Conv7 layer has a sparsity of 95%, so the advantage of the nonlinear solution is more obvious.

Fig. 4 also shows that when accelerating only a single layer by 2 $\times$ , the increased error rates of our solutions are rather marginal or negligible. For the Conv2 layer, the error rate is increased by $<0.1\%$ ; for the Conv3-7 layers, the error rate is increased by $\approx 0.2\%$ .

We also notice that for Conv1, the degradation is negligible near $2\times$ speedup ( $1.8\times$ corresponds to $d^{\prime}=32$ ). This can be explained by Fig. 2(a): the PCA energy has little loss when $d^{\prime}\geq 32$ . But the degradation can grow quickly for larger speedup ratios, because in this layer the channel number $c=3$ is small and $d^{\prime}$ needs to be reduced drastically to achieve the speedup ratio. So in the following whole-model experiments of SPP-10, we will use $d^{\prime}=32$ for Conv1.

Next we evaluate the performance of asymmetric reconstruction as in the problem (12). We demonstrate approximating 2 layers or 3 layers. In the case of 2 layers, we show the results of approximating Conv6 and 7; and in the case of 3 layers, we show the results of approximating Conv5-7 or Conv2-4. The comparisons are consistently observed for other cases of multi-layer.

We sequentially approximate the layers involved, from a shallower one to a deeper one. In the asymmetric version (12), $\mathbf{\hat{x}}$ is from the output of the previous approximated layer (if any), and $\mathbf{x}$ is from the output of the previous non-approximate layer. In the symmetric version (5), we use $\mathbf{x}$ for both terms. We have also tried another symmetric version of using $\mathbf{\hat{x}}$ for both terms, and found this symmetric version is even worse.

Fig. 5 shows the comparisons between the symmetric and asymmetric versions. The asymmetric solution has significant improvement over the symmetric solution. For example, when only 3 layers are approximated simultaneously (like Fig. 5 (c)), the improvement is over 1.0% when the speedup is 4 $\times$ . This indicates that the accumulative error rate due to multi-layer approximation can be effectively reduced by the asymmetric version.

When more and all layers are approximated simultaneously (as below), if without the asymmetric solution, the error rates will increase more drastically.

In Table II we show the results of whole-model acceleration. The solver is the asymmetric version. For Conv1, we fix $d^{\prime}=32$ . For other layers, when the rank selection is not used, we adopt the same speedup ratio on each layer and determine its desired rank $d^{\prime}$ accordingly. When the rank selection is used, we apply it to select $d^{\prime}$ for Conv2-7. Table II shows that the rank selection consistently outperforms the counterpart without rank selection. The advantage of rank selection is observed in both linear and nonlinear solutions.

In Table II we notice that the rank selection often chooses a higher rank $d^{\prime}$ (than the no rank selection) in Conv5-7. For example, when the speedup is 3 $\times$ , the rank selection assigns $d^{\prime}=167$ to Conv7, while this layer only requires $d^{\prime}=153$ to achieve 3 $\times$ single-layer speedup of itself. This can be explained by Fig. 2(c). The energy of Conv5-7 is less concentrated, so these layers require higher ranks to achieve good approximations.

As we will show, the rank selection is more prominent for VGG-16 because of its diversity of layers.

Comparisons with Jaderberg et al.’s method

We compare with Jaderberg et al.’s method , which is a recent state-of-the-art solution to efficient evaluation. Although our decomposition shares some high-level motivations as , we point out that our optimization strategy is different with and is important for accuracy, especially for very deep models that previous acceleration methods rarely addressed.

Jaderberg et al.’s method decomposes a $k\times k$ spatial support into a cascade of $k\times 1$ and $1\times k$ spatial supports. A channel-dimension reduction is also considered. Their optimization method focuses on the linear reconstruction error. In the paper of , their method is only evaluated on a single layer of an OverFeat network for ImageNet.

Our comparisons are based on our implementation of . We use the Scheme 2 decomposition in and its “filter reconstruction” version (as we explain below), which is used for ImageNet as in . Our reproduction of the filter reconstruction in gives a 2 $\times$ single-layer speedup on Conv2 of SPP-10 with $0.2\%$ increase of error. As a reference, in it reports $0.5\%$ increase of error on Conv2 under a 2 $\times$ single-layer speedup, evaluated on another OverFeat network similar to SPP-10.

It is worth discussing our implementation of Jaderberg et al.’s “data reconstruction” scheme, which was suggested to use SGD and backpropagation for optimization. In our reproduction, we find that data reconstruction works well for the character classification task as studied in . However, we find it nontrivial to make data reconstruction work for large models trained for ImageNet. We observe that the learning rate needs to be carefully chosen for the SGD-based data reconstruction to converge (as also reported independently in for another decomposition), and when the training starts to converge, the results are still sensitive to the initialization (for which we have tried Gaussian distributions of a wide range of variances). We conjecture that this is because the ImageNet dataset and models are more complicated, and using SGD to regress a single layer may be sensitive to multiple local optima. In fact, Jaderberg et al.’s only report “filter reconstruction” results of a single layer on ImageNet. For these reasons, our implementation of Jaderberg et al.’s method on ImageNet models is based on filter reconstruction. We believe that these issues have not be settled and need to be investigated further, and accelerating deep networks does not just involve decomposition but also the way of optimization.

In Fig. 6 we compare our method with Jaderberg et al.’s for whole-model speedup. For whole-model speedup of , we implement their method sequentially on Conv2-7 using the same speedup ratio.We do not apply Jaderberg et al.’s method on Conv1, because this layer has a small number of input channels (3), and the first $k\times 1$ decomposed layer can only have a very small number of filters (e.g., 5) to approach a speedup ratio (e.g., 4 $\times$ ). Also note that the speedup ratio is about all conv layers, and because Conv1 is not accelerated, other layers will have a slightly larger speedup. The speedup ratios are the theoretical complexity ratios involving all convolutional layers. Our method is the asymmetric version and with rank selection. Fig. 6 shows that when the speedup ratios are large (4 $\times$ and 5 $\times$ ), our method outperforms Jaderberg et al.’s method significantly. For example, when the speedup ratio is 4 $\times$ , the increased error rate of our method is 4.2%, while Jaderberg et al.’s is 6.0%. Jaderberg et al.’s result degrades quickly when the speedup ratio is getting large, while ours degrades slowly. This suggests the effects of our method for reducing accumulative error.

We further compare with our asymmetric version using 3d decomposition (Sec. 3.5). In Fig. 6 we show the results “asymmetric (3d)”. Fig. 6 shows that this strategy leads to significantly smaller increase of error. For example, when the speedup is 5 $\times$ , the error is increased by only 2.5%. Our asymmetric solver effectively controls the accumulative error even if the multiple layers are decomposed extensively, and the 3d decomposition is easier to achieve a certain speedup ratio.

For completeness, we also evaluate our approximation method on the character classification model released by . Our asymmetric (3d) solution achieves 4.5 $\times$ speedup with only a drop of 0.7% in classification accuracy, which is better than the 1% drop for the same speedup reported by .

The architecture of the approximated model can also be trained “from scratch” on the ImageNet dataset. One hypothesis is that the underlying architecture is sufficiently powerful, and the acceleration algorithm might be not necessary. We show that this hypothesis is premature.

We directly train the model of the same architecture as the decomposed model. The decomposed model is much deeper than the original model (each layer replaced by three layers), so we adopt the initialization method in otherwise it is not easy to converge. We train the model for 100 epochs. We follow the common practice in of training ImageNet models.

The comparisons are in Table IV. The accuracy of the model trained from scratch is worse than that of our accelerated model by a considerable margin (2.8%). These results indicate that the accelerating algorithms can effectively digest information from the trained models. They also suggest that the models trained from scratch have much redundancy.

Table III shows the comparisons of the absolute performance of the accelerated models. We also evaluate the AlexNet which is similarly fast as our accelerated 4 $\times$ models. The comparison is based on our re-implementation of AlexNet. Our AlexNet is the same as in except that the GPU splitting is ignored. Our re-implementation of this model has top-5 single-view error rate as 18.8% (10-view top-5 16.0% and top-1 37.6%). This is better than the one reported in In the 10-view error is top-5 18.2% and top-1 40.7%..

The models accelerated by our asymmetric (3d) version have 14.1% and 13.8% top-5 error, without and with fine-tuning. This means that the accelerated model has 5.0% lower error than AlexNet, while its speed is nearly the same as AlexNet.

Table III also shows the actual running time per view, on a C++ implementation and Intel i7 CPU (2.9GHz) or Nvidia K40 GPU. In our CPU version, our method has actual speedup ratios (3.5 $\times$ ) close to theoretical speedup ratios (4.0 $\times$ ). This overhead mainly comes from the fc and other layers. In our GPU version, the actual speedup ratio is about 3.3 $\times$ . An accelerated model is less easy for parallelism in a GPU, so the actual ratio is lower.

2 Experiments with VGG-16

The very deep VGG models have substantially improved a wide range of visual recognition tasks, including object detection , semantic segmentation , image captioning , video/action recognition , image question answering , texture recognition , etc. Considering the big impact yet slow speed of this model, we believe it is of practical significance to accelerate this model.

Accelerating VGG-16 for ImageNet Classification

Firstly we discover that our whole-model rank selection is particularly important for accelerating VGG-16. In Table VI we show the results without/with rank selection. No 3d decomposition is used in this comparison. For a 4 $\times$ speedup, the rank selection reduces the increased error from 6.38% to 3.84%. This is because of the greater diversity of layers in VGG-16 (Table V). Unlike SPP-10 (or other shallower models ) that repeatedly applies 3 $\times$ 3 filters on the same feature map size, the VGG-16 model applies them more evenly on five feature map sizes (224, 112, 56, 28, and 14). Besides, as the filter numbers in Conv51-53 are not increased, the time complexity of Conv51-53 is smaller than others. The selected ranks $d^{\prime}$ in Table VI show their adaptivity - e.g., the layers Conv51 to Conv53 keep more filters, because they have small time complexity and it is not a good trade-off to compactly reduce them. The whole-model rank selection is a key to maintain a high accuracy for accelerating VGG-16.

In Table VII we evaluate our method on VGG-16 for ImageNet classification. Here we evaluate our asymmetric 3d version (without or with fine-tuning). We evaluate challenging speedup ratios of 3 $\times$ , 4 $\times$ and 5 $\times$ . The ratios are those of the theoretical speedups of all 13 conv layers.

Somewhat surprisingly, our method has demonstrated compelling results for this very deep model, even without fine-tuning. Our no-fine-tuning model has a 0.9% increase of 1-view top-5 error for a speedup ratio of 4 $\times$ . On the contrary, the previous method suffers greatly from the increased depth because of the rapidly accumulated error of multiple approximated layers. After fine-tuning, our model has a 0.3% increase of 1-view top-5 error for a 4 $\times$ speedup. This degradation is even lower than that of the shallower model of SPP-10. This suggests that the information in the very deep VGG-16 model is highly redundant, and our method is able to effectively digest it.

Fig. 7 shows the actual vs. theoretical speedup ratios of VGG-16 using CPU and GPU implementations. The CPU speedup ratios are very close to the theoretical ratios. The GPU implementation, which is based on the standard Caffe library , exhibits a gap between actual vs. theoretical ratios (as is also witnessed in ). GPU speedup ratios are more sensitive to specialized implementation, and the generic Caffe kernels are not optimized for some layers (e.g., 1 $\times$ 1, 1 $\times$ 3, and 3 $\times$ 1 convolutions). We believe that a more specially engineered implementation will increase the actual GPU speedup ratio.

Figurnov et al.’s work is one of few existing works that present results of accelerating the whole model of VGG-16. They report increased top-5 1-view error rates of 3.4% and 7.1% for actual CPU speedups of 3 $\times$ and 4 $\times$ (for 4 $\times$ theoretical speedup they report a 3.8 $\times$ actual CPU speedup). Thus our method is substantially more accurate than theirs. Note that results in are after fine-tuning. This suggests that fine-tuning is not sufficient for whole-model acceleration; a good optimization solver for the decomposition is needed.

Current state-of-the-art object detection results mostly rely on the VGG-16 model. We evaluate our accelerated VGG-16 models for object detection. Our method is based on the recent Fast R-CNN .

We evaluate on the PASCAL VOC 2007 object detection benchmark . This dataset contains 5k trainval images and 5k test images. We follow the default setting of Fast R-CNN using the publicly released codehttps://github.com/rbgirshick/fast-rcnn. We train Fast R-CNN on the trainval set and evaluate on the test set. The accuracy is evaluated by mean Average Precision (mAP).

In our experiments, we first approximate the VGG-16 model on the ImageNet classification task. Then we use the approximated model as the pre-trained model for Fast R-CNN. We use our asymmetric 3d version with fine-tuning. Note that unlike image classification where the conv layers dominate running time, for Fast R-CNN detection the conv layers consume about 70% actual running time . The reported speedup ratios are the theoretical speedups about the conv layers only.

Table IX shows the results of the accelerated models in PASCAL VOC 2007 detection. Our method with a 4 $\times$ convolution speedup has a graceful degradation of 0.8% in mAP. We believe this trade-off between accuracy and speed is of practical importance, because even with the recent advance of fast object detection , the feature extraction running time is still considerable.

Conclusion

We have presented an acceleration method for very deep networks. Our method is evaluated under whole-model speedup ratios. It can effectively reduce the accumulated error of multiple layers thanks to the nonlinear asymmetric reconstruction. Competitive speedups and accuracy are demonstrated in the complex ImageNet classification task and PASCAL VOC object detection task.