InterActive: Inter-Layer Activeness Propagation

Introduction

We have witnessed a big revolution in computer vision brought by the deep Convolutional Neural Networks (CNN). With powerful computational resources and a large amount of labeled training data , a differentiable function for classification is trained to capture different levels of visual concepts organized by a hierarchical structure. A pre-trained deep network is also capable of generating deep features for various tasks, such as image classification , image retrieval and object detection .

Although deep features outperform conventional image representation models such as Bag-of-Visual-Words (BoVW), we note that the deep feature extraction process only involves forward propagation: an image is rescaled into a fixed size, input into a pre-trained network, and the intermediate neuron responses are summarized as visual features. As we shall see in Section 3.1, such a method ignores important high-level visual context, causing both a “big” problem and a “small” problem (see Figure 1). These problems harm the quality of the deep features, and, consequently, visual recognition accuracy.

In this paper, we present InterActive, a novel deep feature extraction algorithm which integrates high-level visual context with low-level neuron responses. For this, we measure the activeness of neuron connections for each specified image, based on the idea that a connection is more important if the network output is more sensitive to it. We define an unsupervised probabilistic distribution function over the high-level neuron responses, and compute the score function (a concept in statistics) with respect to each connection. Each neuron obtains its activeness by collecting the activeness of the related connections. InterActive increases the receptive field size of low-level neurons by allowing the supervision of the high-level neurons. We interpret neuron activeness in terms of spatial-weighted neuron responses, and the visualization of neuron weights demonstrates that visually salient regions are detected in an unsupervised manner. More quantitatively, using the improved InterActive features, we achieve state-of-the-art image classification performance on several popular benchmarks.

The remainder of this paper is organized as follows. Section 2 briefly introduces related works. The InterActive algorithm is presented in Section 3. Experiments are shown in Section 4, and we conclude this work in Section 5.

Related Works

Image classification is a fundamental problem in computer vision. In recent years, researchers have extended the conventional tasks to fine-grained , and large-scale cases.

The Bag-of-Visual-Words (BoVW) model represents each images with a high-dimensional vector. It typically consists of three stages, i.e., descriptor extraction, feature encoding and feature summarization. Due to the limited descriptive power of raw pixels, local descriptors such as SIFT and HOG are extracted. A visual vocabulary is then built to capture the data distribution in feature space. Descriptors are thereafter quantized onto the vocabulary as compact feature vectors , and summarized as an image-level representation . These feature vectors are post-processed , and then fed into a machine learning tool for evaluation.

The Convolutional Neural Network (CNN) serves as a hierarchical model for large-scale visual recognition. It is based on that a network with enough neurons is able to fit any complicated data distribution. In past years, neural networks were shown to be effective for simple recognition tasks . More recently, the availability of large-scale training data (e.g., ImageNet ) and powerful GPUs makes it possible to train deep CNNs which significantly outperform BoVW models. A CNN is composed of several stacked layers, in each of which responses from the previous layer are convoluted and activated by a differentiable function. Hence, a CNN can be considered as a composite function, and is trained by back-propagating error signals defined by the difference between supervised and predicted labels at the top level. Recently, efficient methods were proposed to help CNNs converge faster and prevent over-fitting . It is believed that deeper networks produce better recognition results .

The intermediate responses of CNN, or the so-called deep features, serve as efficient image description , or a set of latent visual attributes. They can be used for various vision applications, including image classification , image retrieval , object detection and object parsing . A discussion of how different CNN configurations impact deep feature performance is available in .

Visualization is an effective method of understanding CNNs. In , a de-convolutional operation was designed to capture visual patterns on different layers of a pre-trained network. and show that different sets of neurons are activated when a network is used for detecting different visual concepts. The above works are based on a supervised signal on the output layer. In this paper, we define an unsupervised probabilistic distribution function on the high-level neuron responses, and back-propagate it to obtain the activeness of low-level neurons. Neuron activeness can also be visualized as spatial weighting maps. Computing neuron activeness involves finding the relevant contents on each network layer , and is related to recovering low-level details from high-level visual context .

Inter-Layer Activeness Propagation

We start with deep features extracted from a pre-trained CNN. Throughout this paper, we will use the very deep VGGNet with $19$ convolutional layers. This produces competitive performance to GoogLeNet , and outperforms AlexNet significantly. We also adopt the same notation for layers used in VGGNet, e.g., conv-3-3, pool-5 and fc-7. All the referred neuron responses are ReLU-processed, i.e., negative values are replaced by .

One of the popular deep feature extraction approaches works as follows: an image is warped (resized) to the same size as the input of a pre-trained network (e.g. $224\times 224$ in VGGNet), then fed into the network, and the responses at an intermediate layer (e.g., fc-6) are used for image representation. A key observation of is that recognition accuracy is significantly boosted if the input images are not warped. In what follows, we resize an image, so that the number of pixels is approximately $512^{2}$, both width and height are divisible by $32$ (the down-sampling ratio of VGGNet), and the aspect ratio is maximally preserved. Using this setting, we obtain a 3D data cube at each layer (even for fc-6 and fc-7), and perform average-pooling or max-pooling to aggregate it as image representation. We emphasize that such a simple resizing modification gives significant improvement in recognition accuracy. For example, with features extracted from the fc-6 layer, the classification accuracy is $83.51\%$, $61.30\%$ and $93.54\%$ on the Caltech256, SUN-397 and Flower-102 datasets, whereas features extracted from warped images only report $80.41\%$, $53.06\%$ and $84.89\%$, respectively. On the pool-5 layer, the numbers are $81.40\%$, $55.22\%$ and $94.70\%$ for un-warped input images, and $77.46\%$, $48.19\%$ and $86.87\%$ for warped ones, also showing significant improvement.

Compared to the large input image size (approximately $512^{2}$ pixels), the receptive field of a neuron on an intermediate layer is much smaller. For example, a neuron on the pool-4, pool-5 and fc-6 layers can see $124\times 124$, $268\times 268$ and $460\times 460$ pixels on the input image, respectively, while its effective receptive field is often much smaller . We argue that small receptive fields cause the following problems: (1) a low-level neuron may not see enough visual context to make prediction, and (2) there may be many irrelevant neurons which contaminate the image representation. We name them the “small” problem and the “big” problem, respectively, as illustrated in Figure 1.

Both the above problems can be solved if low-level neurons receive more visual information from higher levels. In the network training process, this is achieved by error back-propagation, in which low-level neurons are supervised by high-level neurons to update network weights. In this section, we present InterActive, which is an unsupervised method allowing back-propagating high-level context on the testing stage. InterActive involves defining a probabilistic distribution function (PDF) on the high-level neuron responses, and computing the score function which corresponds to the activeness of network connections. As we will see in Section 3.4, this is equivalent to adding spatial weights on low-level neuron responses.

2 The Activeness of Network Connections

Let a deep CNN be a mathematical function ${\mathbf{h}\!\left(\mathbf{X}^{\left(0\right)};\boldsymbol{\Theta}\right)}$, in which $\mathbf{X}^{\left(0\right)}$ denotes the input image and $\boldsymbol{\Theta}$ the weights over neuron connections. There are in total $L$ layers, and the response on the $t$-th layer is $\mathbf{X}^{\left(t\right)}$ (${t}={0}$ indicates the input layer). In our approach, $\mathbf{X}^{\left(t\right)}$ is a vector of length $W_{t}\times H_{t}\times D_{t}$, where $W_{t}$, $H_{t}$ and $D_{t}$ denote the width, height and depth (number of channels), respectively. $x_{w,h,d}^{\left(t\right)}$ is a neuron on the $t$-th layer. The connections on the $t$-th layer, $\boldsymbol{\theta}^{\left(t\right)}$, are a matrix of $\left(W_{t}\times H_{t}\times D_{t}\right)\times\left(W_{t+1}\times H_{t+1}\times D_{t+1}\right)$ elements, where $\theta_{w,h,d,w^{\prime},h^{\prime},d^{\prime}}^{\left(t\right)}$ connects neurons $x_{w,h,d}^{\left(t\right)}$ and $x_{w^{\prime},h^{\prime},d^{\prime}}^{\left(t+1\right)}$. Let $\mathcal{U}_{w,h,d}^{\left(t\right)}$ be the set of neurons on the $\left(t+1\right)$-st layer that are connected to $x_{w,h,d}^{\left(t\right)}$, and $\mathcal{V}_{w^{\prime},h^{\prime},d^{\prime}}^{\left(t+1\right)}$ be the set of neurons on the $t$-th layer that are connected to $x_{w^{\prime},h^{\prime},d^{\prime}}^{\left(t+1\right)}$. Hence, the convolution operation can be written as:

where ${b}={b_{w^{\prime},h^{\prime},d^{\prime}}^{\left(t+1\right)}}$ is the bias term, and $\sigma\!\left[\cdot\right]$ is the ReLU activation: ${\sigma\!\left[\cdot\right]}={\max\left(\cdot,0\right)}$.

We study the PDF on the $T$-th layer $f\!\left(\mathbf{x}^{\left(T\right)}\right)$ by sampling, where ${\mathbf{x}^{\left(T\right)}}={\left(x_{1}^{\left(T\right)},\ldots,x_{D_{T}}^{\left(T\right)}\right)^{\top}}$ is the averaged neuron response vector over all spatial positions:

We use the Caltech256 dataset which contains $30607$ natural images to simulate the distribution. We simply assume that all the $D_{T}$ elements in $\mathbf{x}^{\left(T\right)}$ are nearly independent, and summarize all the $30607\times D_{T}$ elements by 1D histograms shown in Figure 2. We can observe that there are typically fewer neurons with large responses. Therefore, we can assume that the PDF of high-level neurons has the following form: ${f\!\left(\mathbf{x}^{\left(T\right)}\right)}={C_{p}\cdot\exp\!\left\{-\left\|\mathbf{x}^{\left(T\right)}\right\|_{p}^{p}\right\}}$, where $p$ is the norm and $C_{p}$ is the normalization coefficient.

In statistics, the score function indicates how a likelihood function depends on its parameters. The score function has been used to produce discriminative features from generative models , e.g., as of in Fisher vectors . It is obtained by computing the gradient of the log-likelihood with respect to the parameters. Given an image $\mathbf{X}^{\left(0\right)}$, we compute the intermediate network output $\mathbf{X}^{\left(T\right)}$, the response vector $\mathbf{x}^{\left(T\right)}$ using (2), and the likelihood ${f^{\left(T\right)}}\doteq{f\!\left(\mathbf{x}^{\left(T\right)}\right)}$. Then we compute the score function with respect to $\boldsymbol{\theta}^{\left(t\right)}$ to measure the activeness of each network connection in $\boldsymbol{\theta}^{\left(t\right)}$:

where $\mathbf{X}^{\left(t+1\right)}$ is taken as the intermediate term since it directly depends on $\boldsymbol{\theta}^{\left(t\right)}$. The two terms on the right-handed side are named the layer-score and the inter-layer activeness, respectively.

We first compute the layer score $\frac{\partial\ln f^{\left(T\right)}}{\partial\mathbf{X}^{\left(t+1\right)}}$. From the chain rule of differentiation we have:

The second term on the right-handed side, i.e., $\frac{\partial\mathbf{X}^{\left(T\right)}}{\partial\mathbf{X}^{\left(t+1\right)}}$, can be easily derived by network back-propagation as in the training process. The only difference is that the gradient on the top ($T$-th) layer is defined by $\frac{\partial\ln f^{\left(T\right)}}{\partial\mathbf{X}^{\left(T\right)}}$. From $\mathbf{x}^{\left(T\right)}$ defined in (2) and ${f^{\left(T\right)}}={C_{p}\cdot\exp\!\left\{-\left\|\mathbf{x}^{\left(T\right)}\right\|_{p}^{p}\right\}}$, we have:

where $\left(\mathbf{X}^{\left(T\right)}\right)^{p-1}$ is the element-wise $\left(p-1\right)$-st power of the vector. In particular, when ${p}={1}$, the layer score is proportional to an all-one vector $\mathbf{1}^{W_{T}\times H_{T}\times D_{T}}$; when ${p}={2}$, each of the $W_{T}\times H_{T}$ sections is proportional $\mathbf{x}^{\left(T\right)}$.

2.2 The Inter-Layer Activeness

Next we compute the inter-layer activeness $\frac{\partial\mathbf{X}^{\left(t+1\right)}}{\partial\boldsymbol{\theta}^{\left(t\right)}}$. Consider a single term $\frac{\partial x_{w^{\prime},h^{\prime},d^{\prime}}^{\left(t+1\right)}}{\partial\theta_{w,h,d,w^{\prime},h^{\prime},d^{\prime}}^{\left(t\right)}}$, direct differentiation of (1) gives:

3 The Activeness of Neurons

With the layer score (5) and the inter-layer gradient (6), the score function with respect to $\boldsymbol{\theta}^{\left(t\right)}$ is derived to be:

We summarize ${\widetilde{\mathbf{X}}^{\left(t\right)}}={\left\{\widetilde{x}_{w,h,d}^{\left(t\right)}\right\}^{W_{t}\times H_{t}\times D_{t}}}$ with max-pooling (2), resulting in a $D_{t}$-dimensional InterActive feature vector $\widetilde{\mathbf{x}}^{\left(t\right)}$. As we will see in Section 4.2, $\widetilde{\mathbf{x}}^{\left(t\right)}$ is a discriminative representation of the input image $\mathbf{X}^{\left(0\right)}$.

The relationship between $T$ and $t$ can be arbitrary, provided it satisfies ${T}\geqslant{t+1}$. In this paper, we consider two typical settings, i.e., ${T}={L}$ ($L$ is the number of layers) and ${T}={t+1}$, which means that the supervision comes from the final layer (i.e., fc-7) or its direct successor. We name them the last and the next configurations, respectively.

4 Visualization

Before using the InterActive features for experiments (Section 4), we note that $\widetilde{x}_{w,h,d}^{\left(t\right)}$ is a weighted version of the original neuron response $x_{w,h,d}^{\left(t\right)}$. The weighting term is:

It counts the activated (i.e., $x_{w^{\prime},h^{\prime},d^{\prime}}^{\left(t+1\right)}>0$) neurons on the $\left(t+1\right)$-st layer, with the importance $\frac{\partial\ln f^{\left(T\right)}}{\partial x_{w^{\prime},h^{\prime},d^{\prime}}^{\left(t+1\right)}}$, which is supervised by a higher level (the $T$-th layer).

We visualize the weighting term $\gamma_{w,h,d}^{\left(t\right)}$ on the 2D image plane by defining ${\widehat{\gamma}_{w,h}^{\left(t\right)}}={{\sum_{d}}\gamma_{w,h,d}^{\left(t\right)}}$. The weighting map is then resized to the original image size. Representative results are shown in Figure 3. We observe that spatial weighting weakly captures the interest regions, although the network is pre-trained using an independent set (i.e., ImageNet ). Here, we discuss how different parameters affect the weighting terms.

First, activeness measures the contribution of each neuron to higher-level visual outputs. For a low-level neuron, if the supervision comes from the next layer, its receptive field is not significantly enlarged (e.g., a neuron on the pool-1 receives information from the next layer to increase the receptive field from $6\times 6$ to $18\times 18$). Therefore, it is more likely that local high-contrast regions becomes more activated, and the weighting map looks like boundary detection results. As $t$ increases, neurons have larger receptive fields and capture less local details, thus the weighting map is more similar to saliency detection results.

Second, the last and next configurations make a big difference in activeness, especially for the low-level and mid-level neurons. Supervised by the top layer, the last configuration generates stable weighting maps, with the high-weight regions corresponding to the salient objects on the image. However, the output of the next configuration is quite sensitive to small noises, and sometimes the background regions even receive more attention than the semantic objects. As we will see in experiments (Section 4.2), the last configuration consistently produces higher recognition accuracy on the low-level and mid-level features.

We also compare different norms, i.e., ${p}={1}$ vs. ${p}={2}$. When ${p}={2}$, spatial weighting rewards neurons with high responses more heavily, and the high-activeness regions become more concentrated. In general, $p$ reflects the extent that we assume high-response neurons are more important. Although other $p$ values can be used, we believe that ${p}={1}$ and ${p}={2}$ are sufficient to illustrate the difference and produce good performance. We also test ${p}\rightarrow{+\infty}$, which only considers the neuron with the maximal response, but the performance is inferior to that using ${p}={1}$ and ${p}={2}$.

5 Comparison to Related Works

Although both InterActive and network training involve gradient back-propagation, they are propagating different information. In the training process, a supervised loss function is defined by the difference between ground-truth and predicted outputs. In deep feature extraction, however, there is no ground-truth, so we define an unsupervised loss using the score function. Both methods lead to propagating high-level visual context through the network to enhance the descriptive power of low-level neurons.

Although our method and share similar ideas, they are quite different. We focus on generating better image description, while focuses on visualizing the network; we can visualize back-propagated neuron activeness, while visualizes neuron responses; we back-propagate the activeness of all neurons, while only chooses the neuron with maximal response; our method is unsupervised, while is supervised (by “guessing” the label). Being unsupervised, InterActive can be generalized to many more classification problems with a different set of image classes.

In another work on object detection , the neural network is told a visual concept, and the supervised signal is back-propagated to find the most relevant neurons. InterActive performs detection in an implicit, unsupervised manner, making it feasible to be applied to image classification.

Experiments

We evaluate InterActive on six popular image classification datasets. For generic object recognition, we use the Caltech256 ($30607$ images, $257$ classes, $60$ training samples for each class) dataset. For scene recognition, we use the MIT Indoor-67 ($15620$ images, $67$ classes, $80$ training samples per class) and the SUN-397 ($108754$ images, $397$ classes, $50$ training samples per class) datasets. For fine-grained object recognition, we use the Oxford Pet-37 ($7390$ images, $37$ classes, $100$ training samples per class), the Oxford Flower-102 ($8189$ images, $102$ classes, $20$ training samples per class) and the Caltech-UCSD Bird-200 ($11788$ images, $200$ classes, $30$ training samples per class) datasets.

2 InterActive Configurations

We evaluate the InterActive features extracted from different layers, using different norms $p$, and either the last or next configuration (please refer to Section 3.4 and Figure 3). We also compare InterActive with the original deep features with average-pooling or max-pooling. Classification results are summarized in Table 1.

We first observe the low-level and mid-level layers (from pool-1 to pool-4). InterActive with the last configuration consistently outperforms the original deep features. Sometimes, the accuracy gain is very significant (e.g., more than $30\%$ on conv-4-3 and pool4 for bird recognition), showing that InterActive improves image representation by letting the low-level and mid-level neurons receive high-level context. Although these layers often produce low accuracy, the improvement contributes when multi-level features are combined (see Table 2). Regarding the norm, ${p}={2}$ always works better than ${p}={1}$. Recalling from (5) that ${p}={2}$ better rewards high-response neurons, we conclude that high-response neurons are indeed more important.

On the high-level neurons (i.e., pool-5 and fc-6), the advantage of InterActive vanishes in scene classification, and the original average-pooled features produce the best accuracy. Therefore, it is more likely that all the high-level neurons are equally important for scene understanding. On object recognition tasks, the advantage also becomes much smaller, since InterActive only provides limited increase on high-level neurons’ receptive field.

The intermediate output of the $t$-th layer can be considered as a bunch of $D_{t}$-dimensional visual descriptors. Possible choices of feature aggregation include average-pooling and max-pooling. If each image region approximately contributes equally (such as in scene recognition), average-pooling produces higher accuracy, however in the case that semantic objects are quite small (such as on the Bird-200 dataset), max-pooling works better. InterActive computes neuron activeness in an unsupervised manner, which provides a soft weighting scheme, or a tradeoff between max-pooling and average-pooling. By detecting interesting regions automatically, it often produces higher accuracy than both max-pooling and average-pooling.

3 Comparison to the State-of-the-Arts

We compare InterActive with several recent works in Table 2. These algorithms also extract features from statistics-based methods, and use machine learning tools for classification. We concatenate the feature vectors of all $9$ layers in Table 1 as a $6848$-dimensional vector. Apart from the Bird-200 dataset, the reported accuracy is the highest, to the best of our knowledge. Although the accuracy gain over baseline is relatively small (e.g., $0.43\%$ in Pet-37), we emphasize that the baseline accuracy is already very high, thanks to the improved deep feature extraction strategy. Therefore, the improvement of InterActive is not so small as it seems. On the other hand, recognition rates are consistently boosted with InterActive, without requiring extra information, which demonstrates that deep features can be intrinsically improved when neuron activeness is considered.

On the Bird-200 dataset, it is very important to detect the position and/or compositional parts of the objects , otherwise heavy computation is required to achieve good performance . InterActive implicitly finds the semantic object regions, leading to competitive $75.62\%$ accuracy. If the bounding box of each object is provided (as in and ), the original and InterActive features produce $76.95\%$ and $77.53\%$ accuracy, respectively. Using bounding boxes provides $3.60\%$ and $1.91\%$ accuracy gain on original and InterActive features, respectively. InterActive significantly reduces the gap with implicit object detection. $77.53\%$ is lower than $80.26\%$ in and $82.8\%$ in , both of which require fine-tuning the network and R-CNN part detection while InterActive does not. We believe that InterActive can cooperate with these strategies.

4 ImageNet Experiments

We report results on ILSVRC2012, a subset of ImageNet which contains $1000$ categories. We use the pre-trained VGGNet models and the same image cropping techniques as in . The baseline validation error rates on the $16$-layer model, the $19$-layer model and the combined model are $7.1\%$, $7.0\%$ and $6.7\%$, respectively (slightly better than ). We apply InterActive to update the neuron responses on the second-to-last layer (fc-7) and forward-propagate them to re-compute the classification scores (fc-8). The error rates are decreased to $6.8\%$, $6.7\%$ and $6.5\%$, respectively. The improvement is significant given that the baseline is already high and our method is very simple.

In the future, we will explore the use of InterActive on some challenging datasets, such as the PASCAL-VOC dataset and the Microsoft COCO dataset . We thank the anonymous reviewers for this valuable suggestion.

Conclusions

In this paper, we present InterActive, a novel algorithm for deep feature extraction. We define a probabilistic distribution function on the high-level neuron responses, and back-propagate the score function through the network to compute the activeness of each network connection and each neuron. We reveal that high-level visual context carries rich information to enhance low-level and mid-level feature representation. The output of our algorithm is the activeness of each neuron, or a weighted version of the original neuron response. InterActive improves visual feature representation, and achieves the state-of-the-art performance on several popular image classification benchmarks.

InterActive can be applied to many more vision tasks. On the one hand, with the last configuration, neuron activeness provides strong clues for saliency detection. On the other hand, with the next configuration on a low-level layer, neuron activeness can be used to detect local high-contrast regions, which may correspond to edges or boundaries. All these possibilities are left for future research.

References