Interpreting CNNs via Decision Trees

Introduction

Convolutional neural networks (CNNs) have achieved superior performance in various tasks. However, besides the discrimination power, model interpretability is still a significant challenge for neural networks. Many studies have been proposed to visualize or analyze feature representations hidden inside a CNN, in order to open the black box of neural networks.

Motivation & objective: In the scope of network interpretability, state-of-the-art algorithms are still far from the ultimate goal of explaining why a CNN learns knowledge as it is. Although some theories, such as the information bottleneck , analyzed statistical characteristics of a neural network, it is still a challenge to explain why a CNN encodes frontal-leg features, rather than rear-leg features, for classification during the end-to-end learning process.

Therefore, in this study, we limit our discussion to the issue of explaining what knowledge a CNN learns. In this direction, our research focuses on the following two new perspectives of interpreting CNNs:

How to explain features of a middle layer in a CNN at the semantic level. I.e. we aim to transform chaotic features of filters inside a CNN into semantically meaningful concepts, such as object parts, so as to help people to understand the knowledge in the CNN.

How to quantitatively analyze the rationale of each CNN prediction. We need to figure out which filters/parts pass their appearance information through the CNN and contribute to the prediction output. We also report the numerical contribution of each filter (or object part) to the output score.

As shown in Fig. 1, above two perspectives are crucial in real applications and have essential differences from traditional pixel-level visualization and diagnosis of CNN features . Our semantic and quantitative explanations for CNNs have potential values beyond pixel-level visualization/analysis of CNNs. Semantic and quantitative explanations can help people better understand and trust CNN predictions. E.g. in critical applications, such as the recommendation for a surgery plan, people are usually not simply satisfied by the plan itself, but expect a quantitative explanation for the plan.

However, bridging the gap between a CNN’s middle-layer features and semantic explanations has not been well explored yet. We will introduce our task, clarify its challenges, and define relevant concepts from the following two perspectives.

Bridging middle-layer features with semantic concepts: Given an input image, the first issue is to learn more interpretable feature representations inside a CNN and associate each neural activation inside a CNN with a semantic concept. This presents significant challenges for state-of-the-art algorithms.

Firstly, we need to force feature representations in middle conv-layers to be well disentangled during the learning process. According to Zhang et al. summarized the six types of semantics defined in as parts and textures., a filter in traditional CNNs usually represents a mixture of parts and textures. Learning semantically meaningful filters is difficult but is the foundation of semantic-level explanations. In this research, we learn a CNN with disentangled filters in high conv-layers. Each filter needs to be consistently activated by the same object region over different input images. We do not use any annotations of parts or textures to supervise the disentanglement of filter features.

Secondly, we also need to associate each disentangled filter with an explicit semantic meaning (i.e. an object part in this study). This enables linguistic descriptions of middle-layer knowledge, for example, how many parts are memorized in the CNN and how the parts are organized.

Bridging middle-layer features with final CNN predictions: When we have assigned middle-layer features with specific part concepts, the next issue is to quantitatively explain how the CNN uses these middle-layer features to compute prediction scores. In other words, given an input image, we hope to clarify the specific rationale of the CNN prediction.

Here, we define the rationale of a CNN prediction as the set of object parts (or filters) that are activated and contribute to the prediction. Given different input images, the CNN uses different object parts to activate different sets of filters to compute prediction scores, thereby having different rationales. Let us take the bird classification as an example. The CNN may use several filters activated by the head appearances as rationales to classify a standing bird, and the CNN may take filters for wings to distinguish a flying bird.

Given each input image, our task is to clarify filters of which object parts are activated and to quantitatively measure the contribution of each object part to the prediction. The concept of the contribution was also called the “importance” in and was termed “attribution” in . As shown in Fig. 2, we describe the contribution as “a head filter contributes 2.32%, and a feet filter contributes 0.72%.”

Task: As shown in Fig. 1, given a pre-trained CNN, we propose a method to construct a decision tree to explain CNN predictions semantically and quantitatively. We summarize rationales of CNN predictions on all images into various decision modes. Each tree node represents a decision mode. Each decision mode describes common rationales of predictions that are shared by multiple images. I.e. for these images, the CNN usually activates similar filters (object parts), and each part makes a similar contribution to the prediction.

The decision tree hierarchically represents all potential decision modes of a CNN in a coarse-to-fine manner. Nodes near to the tree root node mainly represent most common decision modes (prediction rationales) shared by many images. Nodes near leaves correspond to fine-grained modes of minority images. In particular, each leaf node encodes the specific decision mode of a certain image.

In order to build the decision tree, we learn filters to represent object parts (we do not label any parts or textures as additional supervisionPart annotations are not used to learn the CNN and the decision tree. Given the learned CNN, we label object parts for the filters to compute part-level contributions in Equation (11).). Then, we assign each filter with a certain part name. Finally, we mine decision modes to explain how the CNN use parts/filters for prediction, and construct a decision tree.

Inference: When the CNN makes a prediction for an input image, the decision tree determines a parse tree (see green lines in Fig. 2) to encode a series of explanations. Each node (decision mode) in the parse tree quantitatively explains the prediction at a certain abstraction level, i.e. clarifying how much each object part/filter contributes to the prediction score.

Compared to fine-grained modes in leave nodes, we are more interested in generic decision modes in high-level nodes. Generic decision modes usually select significant object parts (filters) as the rationale of CNN predictions and ignore insignificant ones. Thus, generic decision modes reflect compact rationales for CNN predictions.

Contributions: In this paper, we aim to bridge CNN representations with semantic visual concepts, in order to explain CNN predictions quantitatively and semantically. We propose to learn the decision tree without strong supervision for explanations. Our method is a generic approach and has been successfully applied to various benchmark CNNs. Experiments have demonstrated the effectiveness of our method.

Related work

In this section, we limit our discussion to the literature of opening the black box of CNN representations. discussed the definition of interpretability from different perspectives with respect to different tasks. Zhang et al. made a survey for the interpretability of deep visual models.

CNN visualization: Visualization of filters in a CNN is the most direct way of exploring the pattern hidden inside a neural unit. Gradient-based visualization estimates the input image that maximizes the activation score of a neural unit. Up-convolutional nets invert feature maps of conv-layers into images. Unlike gradient-based methods, up-convolutional nets cannot mathematically ensure that the visualization result reflects actual neural representations.

Zhou et al. proposed a method to accurately compute the image-resolution receptive field of neural activations in a feature map. The estimated receptive field of a neural activation is smaller than the theoretical receptive field based on the filter size. The accurate estimation of the receptive field is crucial to understand a filter’s representations. Bau et al. further defined six types of semantics for CNNs, i.e. objects, parts, scenes, textures, materials, and colors. Zhang et al. summarized the six types of semantics into “parts” and “textures.” Nevertheless, each filter in a CNN represents a mixture of semantics. explained semantic reasons for visual recognition.

Network diagnosis: Going beyond visualization, some methods diagnose a pre-trained CNN to obtain insight understanding of CNN representations.

Fong and Vedaldi analyzed how multiple filters jointly represented a certain semantic concept. Yosinski et al. evaluated the transferability of filters in intermediate conv-layers. Aubry et al. computed feature distributions of different categories in the CNN feature space. Selvaraju et al. and Fong et al. propagated gradients of feature maps w.r.t. the CNN loss back to the image, in order to estimate image regions that directly contribute the network output. The LIME and SHAP extracted image regions that were used by a CNN to predict a label. Zhang et al. used an explainer network to interpret object-part representations in intermediate layers of CNNs.

Network-attack methods diagnosed network representations by computing adversarial samples for a CNN. In particular, influence functions were proposed to compute adversarial samples, provide plausible ways to create training samples to attack the learning of CNNs, fix the training set, and further debug representations of a CNN. Lakkaraju et al. discovered knowledge blind spots (unknown patterns) of a pre-trained CNN in a weakly-supervised manner. The study of examined representations of conv-layers and automatically discover potential, biased representations of a CNN due to the dataset bias.

CNN semanticization: Compared to the diagnosis of CNN representations, some studies aim to learn more meaningful CNN representations. Some studies extracted neural units with certain semantics from CNNs for different applications. Given feature maps of conv-layers, Zhou et al. extracted scene semantics. Simon et al. mined objects from feature maps of conv-layers , and learned object parts . The capsule net used a dynamic routing mechanism to parse the entire object into a parsing tree of capsules. Each output dimension of a capsule in the network may encode a specific meaning. Zhang et al. proposed to learn CNNs with disentangled intermediate-layer representations. The infoGAN and $\beta$-VAE learned interpretable input codes for generative models. Zhang et al. learned functionally interpretable, modular structures for neural networks via network transplanting.

Decision trees for neural networks: Distilling knowledge from neural networks into tree structures is an emerging direction , but the trees did not explain the network knowledge at a human-interpretable semantic level. Wu et al. learned a decision tree via knowledge distillation to represent the output feature space of an RNN, in order to regularize the RNN for better representations. Vaughan et al. distilled knowledge into an additive model for explanation.

In spite of the use of tree structures, there are two main differences between the above two studies and our research. Firstly, we focus on using a tree to semantically explain each prediction made by a pre-trained CNN. In contrast, decision trees in above studies are mainly learned for classification and cannot provide semantic-level explanations. Secondly, we summarize decision modes from gradients w.r.t. neural activations of object parts as rationales to explain CNN prediction. Compared to above “distillation-based” methods, our “gradient-based” decision tree reflects CNN predictions more directly and strictly.

Image-specific rationale of a CNN prediction

In this section, we design a method to simplify the complex feature processing inside a CNN into a linear form (i.e. Equation (3)), as the specific rationale of the prediction w.r.t. the input image. This clarifies (i) which object parts activate which filters in the CNN and (ii) how much these parts/filters contribute to the final prediction score.

In order to obtain semantic-level rationale of a CNN prediction, we need (i) first to ensure that the CNN’s middle-layer features are semantically meaningful, and (ii) then to extract explicit contributions of middle-layer features to the prediction score.

In this study, we learn the CNN for object classification. Theoretically, we can interpret CNNs oriented to different tasks. Nevertheless, in this paper, we limit our attention to CNNs for classification, in order to simplify the story.

The basic idea is to revise a benchmark CNN, in order to make each filter in the top conv-layer represent a specific object part. We expect the filter to be automatically converged to the representation of a part, instead of using additional part annotations to supervise the learning process.

We apply the filter loss to each filter in the top conv-layer to push the filter towards the representation of an object part. As shown in Fig. 3, the filter is learned to be activated by the same object part given different input images. Theoretically, our method also supports other techniques of mining interpretable features in middle layers . Nevertheless, the filter loss usually ensures more meaningful features than the other approaches.

where $MI(\cdot)$ indicates the mutual information. ${\bf X}_{f}$ denotes a set of feature maps of $f$ extracted from different input images. ${\bf P}=\{\mu|\mu=[h,w],1\leq h,w\leq L\}\cup\{\emptyset\}$ is referred to as a set of all part-location candidates. Each location $\mu=[h,w]$ corresponds to an activation unit in $x_{f}$. Besides, $\emptyset\in{\bf P}$ denotes the case that the target part does not appear in the input image. In this case, all units in $x_{f}$ are expected to keep inactivated. The joint probability $p(x_{f},\mu)$ to describe the compatibility between $x_{f}$ and $\mu$ (please see for details).

The filter loss ensures that given an input image, $x_{f}$ should match only one of all $L^{2}+1$ location candidates. It is assumed that repetitive shapes on various regions are more likely to describe low-level textures than high-level parts. If the part appears, $x_{f}$ should have a single activation peak at the part location; otherwise, $x_{f}$ should keep inactivated.

2 Quantitative rationales of CNN predictions

As analyzed in , filters in high conv-layers are more prone to represent object parts, while those in low conv-layers usually describe textures. Therefore, we choose filters in the top conv-layer to represent object parts. Consequently, we quantitatively analyze how fully-connected (FC) layers use object-part features from the top conv-layer to make final predictions, as the rationale.

As discussed in , we can use a piecewise linear representation to represent the function of cascaded FC layers and ReLU layers, as follows.

We use weights $g$ to denote the specific rationale of the prediction for the input image. $g^{(h,w,d)}x^{(h,w,d)}$ measures $x^{(h,w,d)}$’s quantitative contribution to the prediction.

Different input images correspond to different weights $g$, i.e. different rationales of their CNN predictions. It is because different images have various signal passing routes through ReLU layers. Given an input images $I$, the CNN uses certain weight values that are specific to $I$.

In this way, we can consider ${\bf x}$ and ${\bf g}$ to represent prediction rationalesWithout loss of generality, we normalize ${\bf g}$ to a unit vector for more convincing results: $y\!\leftarrow\!{y}/{\|{\bf g}\|}$, ${\bf g}\!\leftarrow\!{{\bf g}}/{\|{\bf g}\|}$, and $b\!\leftarrow\!{b}/{\|{\bf g}\|}$., i.e. using which filters/parts for prediction.

Different dimensions of the vector ${\bf x}$ measure the scalar signal strength of different object parts, since a filter potentially represents a certain object part. ${\bf g}$ corresponds to the selection of object parts for the CNN prediction.

Learning a decision tree

We learn a decision tree to interpret the classification of each category. In the following two subsections, we first define basic concepts in a decision tree and then introduce the learning algorithm.

Let us focus on the decision tree for a certain category. We consider images of this category as positive images and consider other images as negative images. ${\boldsymbol{\Omega}}^{+}$ denotes image indexes of the target category, i.e. positive images, and ${\boldsymbol{\Omega}}={\boldsymbol{\Omega}}^{+}\cup{\boldsymbol{\Omega}}^{-}$ represents all training images. For an image $I_{i}$ ($i\in{\boldsymbol{\Omega}}$), $y_{i}$ denotes the classification score of the target category before the softmax layer.

As shown in Fig. 2, each node $v$ in the decision tree encodes a decision mode that is hidden inside FC layers of the CNN. A decision mode represents a common rationale of the prediction shared by a group of positive training images $\Omega_{v}\subset{\boldsymbol{\Omega}}^{+}$. The decision tree organizes the hierarchy of decision modes in a coarse-to-fine manner from the root node to leaf nodes. Children nodes $v^{\prime}\in Child(v)$ divides the parent $v$’s decision mode into fine-grained modes. Fine-grained modes are shared by sub-groups of images.

Just like the rationale defined in Equation (3), the decision mode in node $v$ is parameterized with ${\bf w}$ and $b$, and the mode explains predictions on a certain set of images $\Omega_{v}$. For each image $I_{i}$, $i\in\Omega_{v}$, the decision mode is given as

where ${\bf w}$ is referred to as the rationale of the decision mode. $\overline{\bf g}$ is a unit vector ($\|\overline{\bf g}\|_{2}\!=\!1$) that reflects common rationales that are shared by all images in $\Omega_{v}$. ${\boldsymbol{\alpha}}\in\{0,1\}^{D}$ is given as a binary selection of filters in the decision mode. $\circ$ denote element-wise multiplications. We compute sparse ${\boldsymbol{\alpha}}$ to obtain sparse explanations for the decision mode5.

In particular, when $v$ is a leaf node, the decision mode is formulated as the rationale of a specific image $I_{i}$. I.e. ${\boldsymbol{\alpha}}=[1,1,\ldots,1]^{T}$ and ${\bf w}={\boldsymbol{\alpha}}\circ{\bf g}_{i}={\bf g}_{i}$, which is computed in Equation (3).

2 Learning decision trees

Just like hierarchical clustering, the basic idea of learning a decision tree is to summarize common generic decision modes from specific decision modes of different images. Algorithm 1 shows the pseudo-code of the learning process. At the beginning, we initialize the decision mode ${\bf g}_{i}$ of each positive image $I_{i}$ as a leaf node by setting $\overline{\bf g}\!=\!{\bf g}_{i}$ and ${\boldsymbol{\alpha}}\!=\!{\bf 1}$. Thus, we build an initial tree $Q$ as shown in Fig. 4, in which the root node takes decision modes of all positive images as children. Then, in each step, we select and merge two nodes $v,v^{\prime}\in V$ in the second tree layer (i.e. children of the root node) to obtain a new node $u$, where $V$ denotes the children set of the root. $u$ becomes a new child of the root node, and $v$ and $v^{\prime}$ are re-assigned as $u$’s children. The image set of $u$ is defined as $\Omega_{u}=\Omega_{v}\cup\Omega_{v^{\prime}}$ and we learn ${\boldsymbol{\alpha}},b,\overline{\bf g}$ for $u$ based on Equations (5) and (6).

In this way, we gradually revise the initial tree $P_{0}=Q$ towards the final tree after $T$ merging operations as

We formulate the objective for learning as follows.

where $P({\bf x}_{i})$ denotes the likelihood of ${\bf x}_{i}$ being positive that is estimated by the tree $P$. $\prod_{i}P({\bf x}_{i})$ indicates the discriminative power of $P$. $\beta$ is a scaling parameterPlease see the experiment section for settings of $\beta$, $\gamma$, and $\lambda$.. This objective penalizes the decrease of the discriminative power and encourages the system to summarize a few decision modes as generic explanations for CNN predictions. We compute the likelihood of ${\bf x}_{i}$ being positive as

where $\hat{h}({\bf x}_{i})\!=\!h_{\hat{v}}({\bf x}_{i})$ denotes the prediction on ${\bf x}_{i}$ based on best child $\hat{v}\in V$ in the second tree layer. $\gamma$ is a constant scaling parameter5.

In the $t$-th step, we merge two nodes $v,v^{\prime}\in V$ in the second tree layer of $P_{t-1}$ to get a new node $u$, thereby obtaining a new tree $P_{t}$. We can easily compute $\Delta\log E$ w.r.t. each pair of $(v,v^{\prime})$ based on Equation (8). Thus, we learn the decision tree via a greedy strategy. In each step, we select and merge the nodes $v,v^{\prime}\in V$ that maximize $\frac{\Delta\log E}{\|\Omega_{v}\|+\|\Omega_{v^{\prime}}\|}$. We normalize $\Delta\log E$ for reasonable clustering performance.

3 Interpreting CNNs

Given a testing image $I_{i}$, the CNN makes a prediction $y_{i}$. The decision tree estimates quantitative decision modes of the prediction at different fine-grained levels. During the inference procedure, we can infer a parse tree, which starts from the root node, in a top-down manner. Green lines in Fig. 4 show a parse tree. When we select the decision mode in the node $u$ as the rationale, we can further select its child $\hat{v}$ that maximizes the compatibility with the most specific rationale ${\bf g}_{i}$ as a more fine-grained mode:

A node $v$ in the parse tree provides the rationale of the prediction on image $I_{i}$ at a certain fine-grained level. We compute the vector ${\boldsymbol{\rho}}_{i}$ and ${\boldsymbol{\varrho}}_{i}$ to evaluate the contribution of different filters and that of different object parts.

Experiments

Datasets: Because the quantitative explanation of CNN predictions requires us to assign each filter in the top conv-layer with a specific object part, we used three benchmark datasets with ground-truth art annotations to evaluate our method. The selected datasets include the PASCAL-Part Dataset , the CUB200-2011 dataset , and the ILSVRC 2013 DET Animal-Part dataset . Just like in most part-localization studies , we used animal categories, which prevalently contain non-rigid shape deformation, for evaluation. I.e. we selected six animal categories—bird, cat, cow, dog, horse, and sheep—from the PASCAL Part Dataset. The CUB200-2011 dataset contains 11.8K images of 200 bird species. Like in , we ignored species labels and regarded all these images as a single bird category. The ILSVRC 2013 DET Animal-Part dataset consists of 30 animal categories among all the 200 categories for object detection in the ILSVRC 2013 DET dataset .

Analysis of object parts for prediction: We analyzed the contribution of different object parts in the CNN prediction, when we assigned each filter with a specific object part. The vector ${\boldsymbol{\varrho}}_{i}$ in Equation (11) specifies the contribution of different object parts in the prediction of $y_{i}$. For the $m$-th object part, we computed $contri_{m}\!=\!|\varrho_{i}^{(m)}|/{\sum_{m^{\prime}=1}^{M}|\varrho_{i}^{(m^{\prime})}|}$ as the ratio of the part’s contribution.

More specifically, for CNNs based on the ILSVRC 2013 DET Animal-Part dataset, we manually labeled the object part for each filter in the top conv-layer. For CNNs based on the Pascal VOC Part dataset , the study of merged tens of small parts into several major landmark parts for the six animal categories. Given a CNN for a certain category, we used to estimate regions in different images that corresponded to each filter’s neural activations, namely the image receptive field of the filter (please see Figs. 6 and 3). For each filter, we selected a part from all major landmark parts, which was closest to the filter’s image receptive field through all positive images. For the CNN based on the CUB200-2011 dataset, we used ground-truth positions of the breast, forehead, nape, tail of birds as major landmark parts. Similarly, we assigned each filter in the top conv-layer with the nearest landmark part.

Evaluation for nodes in different layers: The above three metrics evaluate decision modes (nodes) in the second layer of the decision tree. Because nodes in lower layers encode more fine-grained decision modes, we extended the three metrics to evaluate nodes in low layers. When we evaluated nodes in the $k$-th layer, we temporarily constructed a new tree by removing all nodes above the $k$-th layer and directly connecting the root node to nodes in the $k$-th layer. Thus, we can apply the evaluation to the new tree.

Explanations based on the decision tree: Decision modes in the learned decision tree objectively reflected the knowledge hidden inside a CNN. Table 1 shows the structure of the decision tree by listing numbers of nodes in different layers of the decision tree. Fig. 5 visualizes decision modes in the decision tree. Fig. 6 shows distributions of object-part contributions to the CNN prediction, which were estimated using nodes in the second layer of decision trees.

Table 4 evaluates the information loss when we use the decision tree to represent a CNN. Metrics of the average classification accuracy, the average prediction error are used for evaluation. Tables 2 and 3 use errors of object-part contributions and the average fitness of contribution distributions, respectively, to evaluate the accuracy of the estimated rationales based on nodes in different tree layers. Generally speaking, because fine-grained decision modes are close to the image-specific rationale, fine-grained decision modes usually yielded lower error prediction rates. However, fine-grained decision modes did not exhibit higher accuracy in classification. It is because our method is designed to mine common decision modes for objects of a certain category, and ignores random/negative images, which is different from the discriminative learning of classifiers.

Conclusion and discussions

In this study, we use a decision tree to explain CNN predictions at the semantic level. We have developed a method to revise a CNN and built a tight coupling of the CNN and a decision tree. The proposed decision tree encodes decision modes of the CNN as quantitative rationales for each CNN prediction. Our method does not need any annotations of object parts or textures in training images to guide the learning the CNN. We have tested our method in different benchmark datasets, and experiments have proved the effectiveness of our approach.

Note that theoretically, the decision tree just provides an approximate explanation for CNN predictions, instead of an accurate reconstruction of CNN representation details. There are two reasons. Firstly, without accurate object-part annotations to supervised the learning of CNNs, the filter loss can only roughly make each filter to represent an object part. The filter may produce incorrect activations in a few challenging images. Secondly, the decision mode in each node ignores insignificant object-part filters to ensure a sparse representation of the decision mode.

References