Interpretation of Neural Networks is Fragile

Introduction

Predictions made by machine learning algorithms play an important role in our everyday lives and can affect decisions in technology, medicine, and even the legal system (?; ?). As algorithms become increasingly complex, explanations for why an algorithm makes certain decisions are ever more crucial. For example, if an AI system predicts a given pathology image to be malignant, then a doctor may need to know what features in the image led the algorithm to this classification. Similarly, if an algorithm predicts an individual to be a credit risk, then the lender (and the borrower) might want to know why. Therefore having interpretations for why certain predictions are made is critical for establishing trust and transparency between users and the algorithm (?).

Having an interpretation is not enough, however. The explanation itself must be robust in order to establish human trust. Take the pathology predictor; an interpretation method might suggest that a particular section in an image is important for the malignant classification (e.g. that section could have high scores in saliency map). The clinician might then focus on that section for investigation or treatment or even look for similar features in other patients. It would be highly disconcerting if in an extremely similar image, visually indistinguishable from the original and also classified as malignant, a very different section is interpreted as being salient for the prediction. Thus, even if the predictor is robust (both images are correctly labeled as malignant), that the interpretation is fragile would still be problematic in deployment. Furthermore, if the interpretation is used to guide interventions (e.g. location of a biopsy) by the doctor, then an interpretation that is not robust against adversarial perturbations may prove to be a security concern.

It is well known that the predicted labels of deep neural networks are susceptible to adversarial attacks (?; ?; ?; ?). In this paper, we introduce the notion of adversarial perturbations to neural network interpretation. More precisely, we define the interpretation of neural network to be fragile if, for a given image, it is possible to generate a perceptively indistinguishable image that has the same prediction label by the neural network, yet is given a substantially different interpretation. We systematically investigate two classes of interpretation methods: methods that assign importance scores to each feature (this includes simple gradients (?), DeepLift (?), and integrated gradients (?)), as well as a method that assigns importances to each training example: influence functions (?). For these interpretation methods, we show how to design targeted perturbations that can lead to dramatically different interpretations across test images (Fig. 1). Our findings highlight the fragility of interpretations of neural networks, which has not been carefully considered in the literature. Fragility limits how much we can trust and learn from the interpretations. It also raises a significant new security concern. Especially in medical or economic applications, users often take the interpretation of a prediction as containing causal insight (“this image is a malignant tumor because of the section with a high saliency score”). An adversary could minutely manipulate the input to draw attention away from relevant features or onto his/her desired features. Such attacks might be especially hard to detect as the actual labels have not changed.

While we focus on image data here because most interpretation methods have been motivated by images, the fragility of neural network interpretation could be a much broader problem. Fig. 2 illustrates the intuition that when the decision boundary in the input feature space is complex, as is the case with deep networks, a small perturbation in the input can push the example into a region with very different loss contours. Because the feature importance is closely related to the gradient which is perpendicular to the loss contours, the importance scores can also be dramatically different. We provide additional analysis of this in Section 5.

Related Works and Preliminaries

2 Interpretation methods for neural networks

Interpretation of neural network predictions is an active research area. Post-hoc interpretability (?) is one family of methods that seek to “explain” the prediction without considering the details of black-box model’s hidden mechanisms. These include methods to explain predictions in terms of the features of the test example, as well as in terms of the contribution of training examples to the test time prediction. These interpretations have gained increasing popularity, as they confer a degree of insight to human users of what the neural network might be doing (?). We describe several widely-used interpretation methods in what follows.

Integrated gradients A significant drawback of the simple gradient method is the saturation problem discussed by (?; ?). Consequently, ? introduced the integrated gradients method where the gradients of the score with respect to $M$ scaled versions of the input are summed and then multiplied by the input. Letting $\boldsymbol{x}^{0}$ be the reference point and $\Delta\boldsymbol{x}_{t}=\boldsymbol{x}_{t}-\boldsymbol{x}^{0}$, the feature importance vector is calculated by: $\boldsymbol{I}(\boldsymbol{x}_{t})=\left\lvert\frac{\Delta\boldsymbol{x}_{t}}{M}\sum_{k=1}^{M}\nabla_{\boldsymbol{x}}{S_{l}\left(\frac{k}{M}\Delta\boldsymbol{x}_{t}+\boldsymbol{x}^{0}\right)}\right\rvert,$ which is then normalized for our analysis. Here the absolute value is taken for each dimension.

DeepLIFT DeepLIFT is an improved version of layer-wise relevance propagation (LRP) method (?). LRP methods decompose the score $S_{l}(\boldsymbol{x}_{t})$ backwards through the neural network. DeepLIFT (?) defines a reference point in the input space and propagates relevance scores proportionally to the changes in the neuronal activations from the reference. We use DeepLIFT with the Rescale rule; see (?) for details.

Sample Importance Interpretation

A complementary approach to interpreting the results of a neural network is to explain the prediction of the network in terms of its training examples, $\{(\boldsymbol{x_{i}},y_{i})\}$. Specifically, to ask which training examples, if up-weighted or down-weighted during training time, would have the biggest effect on the loss of the test example $(\boldsymbol{x_{t}},y_{t})$. (?) proposed a method to calculate this value, called the influence, defined by the equation: $I(z_{i},z_{t})=-\nabla_{\theta}L(z_{t},\hat{\theta})^{\top}H_{\hat{\theta}}^{-1}\nabla_{\theta}L(z_{i},\hat{\theta}),$ where $z_{i}=(\boldsymbol{x}_{i},y_{i})$ , $z_{t}=(\boldsymbol{x}_{t},y_{T})$, and $L(z,\hat{\theta})$ is the prediction loss of (training or test) data point $z$ in network with parameters $\hat{\theta}$. $H_{\hat{\theta}}=\frac{1}{n}\sum_{i=1}^{n}\nabla_{\theta}^{2}L(z_{i},\hat{\theta})$ is the empirical Hessian of the network calculated over the training examples. We calculate the influence over the entire training set $\boldsymbol{I}(\cdot,z_{t})$.

Metrics for interpretation similarity

We consider two natural metrics for quantifying the similarity between interpretations for two different images:

Spearman’s rank order correlation: Because interpretation methods rank all of the features or training examples in order of importance, it is natural to use the rank correlation (?) to compare the similarity between interpretations.

Top-$k$ intersection: In many settings, only the most important features are of explanatory interest. In such settings, we can compute the size of intersection of the $k$ most important features before and after perturbation.

Methods: Generating Perturbations

For a given neural network $\mathscr{N}$ with fixed weights and a test data point $\boldsymbol{x}_{t}$, the feature importance and sample importance methods produce an interpretation $\boldsymbol{I}(\boldsymbol{x}_{t};\mathscr{N})$. For feature importance, $\boldsymbol{I}(\boldsymbol{x}_{t};\mathscr{N})$ is a vector of feature scores; for influence function $\boldsymbol{I}(\boldsymbol{x}_{t};\mathscr{N})$ is a vector of scores for training examples. Our goal is to devise efficient and visually imperceptible perturbations that change the interpretability of the test input while preserving the predicted label. Formally, we define the problem as:

where $\mathcal{D}(\cdot)$ measures the change in interpretation (e.g. how many of the top-$k$ pixels are no longer the top-$k$ pixels of the feature importance map after the perturbation) and $\epsilon>0$ constrains the norm of the perturbation. In this paper, we carry out three kinds of input perturbations.

Random sign perturbation

As a baseline, each pixel is randomly perturbed by $\pm\epsilon$. This is used as a baseline with which to compare our adversarial perturbations against both feature importance and sample importance methods.

Iterative attacks against feature importance methods

In Algorithm 1, we define three adversarial attacks against feature importance methods, each of which consists of taking a series of steps in the direction that maximizes a differentiable dissimilarity function between the original and perturbed interpretation. (1) The top-$\mathbf{k}$ attack seeks to perturb the feature importance map by decreasing the relative importance of the $k$ initially most important input features. (2) For image data, feature importance map’s center of mass often captures the user’s attention. The mass-center attack is designed to result in the maximum spatial displacement of the center of mass. (3) If the goal is to have a semantically meaningful change in feature importance map, targeted attack aims to increase the concentration of feature importance scores in a pre-defined region of the input image.

Gradient sign attack against influence functions

We can obtain effective adversarial images for influence functions without resorting to iterative procedures. We linearize the equation for influence functions around the values of the current inputs and parameters. If we further constrain the $L_{\infty}$ norm of the perturbation to $\epsilon$, we obtain an optimal single-step perturbation:

The attack we use consists of applying the negative of the perturbation in (1) to decrease the influence of the 3 most influential training images of the original test imageIn other words, we generate the perturbation given by: $-\epsilon\text{sign}(\sum_{i=1}^{3}\nabla_{\boldsymbol{x}_{t}}\nabla_{\theta}L(z_{t},\hat{\theta})^{\top}H_{\hat{\theta}}^{-1}\nabla_{\theta}L(z_{(i)},\hat{\theta}))$, where $z_{(i)}$ is the $i^{\text{th}}$ most influential training image of the original test image.. Of course, this affects the influence of all of the other training images as well.

We follow the same setup for computing the influence function as was done in (?). Because the influence is only calculated with respect to the parameters that change during training, we calculate the gradients only with respect to parameters in the final layer of our network (InceptionNet, see Section 4). This makes it feasible for us to compute (1) exactly, but it gives us the perturbation of the input into the final layer, not the first layer. So, we use standard back-propagation to calculate the corresponding gradient for the input test image.

Experiments & Results

For attacks against feature importance interpretation, we used ILSVRC2012 (ImageNet classification challenge data) (?) and CIFAR-10 (?). For the ImageNet classification data set, we used a pre-trained SqueezeNet model introduced by (?). For the CIFAR-10 data we trained our own convolutional network (architecture in Appendix A.)

For both data sets, the results are examined on feature importance scores obtained by simple gradient, integrated gradients, and DeepLIFT methods. For DeepLIFT, we used the pixel-wise and the channel-wise mean images as the CIFAR-10 and ImageNet reference points respectively. For the integrated gradients method, the same references were used with parameter $M=100$. We ran all iterative attack algorithms for $P=300$ iterations with step size $\alpha=0.5$. (To show the performance success against methods that are not directly gradient-based, we also ran a smaller experiment of 100 Imagenet examples for the Deep Taylor Decomposition method (?) to show the attack method’s success Results are reflected in Appendix E).

To evaluate our adversarial attack against influence functions, we followed a similar experimental setup to that of the original authors: we trained an InceptionNet v3 with all but the last layer frozen (the weights were pre-trained on ImageNet and obtained from Keras). The last layer was trained on a binary flower classification task (roses vs. sunflowers), using a data set consisting of 1,000 training imagesadapted from: https://goo.gl/Xgr1a1. This data set was chosen because it consisted of images that the network had not seen during pre-training on ImageNet. The network achieved a validation accuracy of 97.5%.

Results for attacks against feature importance scores

From the ImageNet test set, 512 correctly-classified images were randomly sampled for evaluation. Examples of the mass-center attack against feature importance scores obtained by the three mentioned methods are presented in Fig. 1. Examples of targeted attacks, whose goal is to change the semantic meaning of the interpretation are depicted in Fig. 5 and also in Appendix D. Further representative examples of top-k and mass center attacks are found in Appendix B. Appendix C provides examples of how the decrease in rank order correlation and top-1000 intersection relate to visual changes in the feature importance maps.

In Fig. 3, we present results aggregated over all 512 images. We compare different attack methods using top-1000 intersection and rank correlation methods. In all the images, the attacks do not change the original predicted label of the image nor does it significantly change the prediction confidence. Random sign perturbation already causes decreases in both top-1000 intersection and rank order correlation. For example, with $L_{\infty}=8$, on average, there is less than 30% overlap in the top 1000 most salient pixels between the original and the randomly perturbed images across all three of interpretation methods.

Both the mass-center and top-K attack algorithms have similar effects on feature importance of test images when measured on the basis of rank correlation or top-1000 intersection. We show an additional metric in Appendix E that measures the displacement in feature importance maps and empirically has the most correlation with perceptual change in interpretation. Not surprisingly, we found that the mass-center attack was more effective than the top-$k$ attack at resulting in the most perceptive change. Average numerical results are not obtainable for the targeted attack as it is designed for semantic change and requires a target area of attack in each image. Comparing the effectiveness of attacks among the three different feature importance methods, we found that the integrated gradients method was the most difficult one to generate adversarial examples for. Similar results for CIFAR-10 can be found in Appendix F.

Results for adversarial attacks against sample importance scores.

We evaluate the robustness of influence functions on a test data set consisting of 200 images of roses and sunflowers. Fig. 4(a) shows a representative test image to which we have applied the gradient sign attack. Although the prediction of the image does not change, the most influential training examples change entirely. Additional examples can be found in Appendix G.

In Fig. 4(b,c), we compare the random perturbations and gradient sign attacks for the test set. It shows that gradient sign-based attacks are significantly more effective at decreasing the rank correlation, as well as distorting the top-5 influential images. For example, on average, with a perturbation of magnitude $\epsilon=8$, only 2 of the top 5 most influential training images remain in the top 5. The influences of the training images before and after an adversarial attack are essentially uncorrelated. However, we find that even random attacks can have a small but non-negligible effect on influence functions, on average reducing the rank correlation to 0.8 ($\epsilon\approx 10$).

Hessian Analysis

In this section, we explain the effectiveness of adversarial attacks on interpretations in terms of the high dimensionality and non-linearities in deep networks. High dimensionality is also a reason why adversarial examples are effective at changing prediction labels (?).

The first order approximation of change in feature importance map due to a small input perturbation: $\boldsymbol{x}\rightarrow\boldsymbol{x}+\boldsymbol{\delta}$ will be equal to : $H\cdot\boldsymbol{\delta}=\nabla^{2}_{\boldsymbol{x}}S\cdot\boldsymbol{\delta}$. In particular, the relative change in the importance score of the $i^{\text{th}}$ feature is $(\nabla^{2}_{\boldsymbol{x}}S\cdot\boldsymbol{\delta})_{i}/(\nabla_{\boldsymbol{x}}S)_{i}$. For our simple model, this relative change is:

Notice that for this simple network, the direction of adversarial attack on interpretability, $\text{sign}(\boldsymbol{w})$ is the same as the adversarial attack on prediction which means that perturbing interpretability perturbs prediction. For more complex networks, this is not the case and in Appendix I we show this analytically for a simple case of a two-layer network. As an empirical test, in Fig. 6(a), we plot the distribution of the angle between most fragile directions of interpretation and prediction for 1,000 CIFAR-10 images (Model details in Appendix A). Fig. 6(b) shows the equivalent distribution for influence functions, computed across all 200 test images.

Discussion

This paper demonstrates that interpretation of neural networks can be fragile in the sense that two similar inputs with the same predictedion can be given very different interpretations. We develop perturbations to illustrate this fragility and propose evaluation metrics as well as insights on why fragility occurs. Fragility of neural network interpretation can be orthogonal to fragility of the prediction, as we demonstrate with perturbations that substantially change the interpretation without changing the predicted label, but both types of fragility arise at least in part from high dimensionality, as we discuss in Section 5.

Our main message is that robustness of the interpretation of a prediction is an important and challenging problem, especially as in many applications (e.g. many biomedical and financial settings), users are as interested in the interpretation as in the prediction itself. Our results raise concerns on how interpretations of neural networks can be manipulated. Especially in settings where the importance of individual or a small subset of features are interpreted, we show that these importance scores can be sensitive to even random perturbation. More dramatic manipulations of interpretations can be achieved with our targeted perturbations. This is especially true for the simple gradients method, DeepLIFT, and influence functions, but also the integrated gradients method. These results raise potential security concerns. We do not suggest that interpretations are meaningless, just as adversarial attacks on predictions do not imply that neural networks are useless. Interpretation methods do need to be used and evaluated with caution while applied to neural networks, as they can be fooled into identifying features that would not be considered salient by human perception.

Our work naturally raises the question of how to defend against adversarial attacks on interpretation. Because interpretation fragility arises as a consequence of high dimensionality and non-linearity (see section 5), we believe that techniques that discretize inputs, such as thermometer encoding (?), and train neural networks in a way to constrain the non-linearity of the network (?), may be useful in defending against interpretation attacks.

While we focus on the standard image benchmarks for popular interpretation tools, this fragility issue can be wide-spread in biomedical, economic and other settings where neural networks are increasingly used. Understanding interpretation fragility in these applications and developing more robust methods are important agendas of research.

References

Appendices

Appendix A Description of the CIFAR-10 classification network

We trained the following structure using ADAM optimizer (?) with default parameters. The resulting test accuracy using ReLU activation was 73%. For the experiment in Fig, 6(a), we replaced ReLU activation with Softplus and retrained the network (with the ReLU network weights as initial weights). The resulting accuracy was 73%.

Appendix B Additional examples of feature importance perturbations

Here, we provide three more examples from ImageNet. For each example, all three methods of feature importance are attacked by random sign noise and our two adversarial algorithms.

Appendix C Objective metrics and subjective change in feature importance maps

Appendix D Semantically meaningful change in feature importance using targeted attack

Appendix E Measuring center of mass movement

Appendix F Results for adversarial attacks against CIFAR-10 feature importance methods

Appendix G Additional examples of adversarial attacks on influence functions

In this appendix, we provide additional examples of the fragility of influence functions, analogous to Fig. 4.

Appendix H Dimensionality-based explanation for fragility of influence functions

Here, we demonstrate that increasing the dimension of the input of a simple neural network increases the fragility of that network with respect to influence functions, analogous to the calculations carried out for importance-feature methods in Section 5. Recall that the influence of a training image $z_{i}=(\boldsymbol{x}_{i},y_{i})$ on a test image $z=(\boldsymbol{x},y)$ is given by:

We restrict our attention to the term in (3) that is dependent on $\boldsymbol{x}$, and denote it by $J=\nabla_{\theta}L$. $J$ represents the infinitesimal effect of each of the parameters in the network on the loss function evaluated at the test image.

Now, let us calculate the change in this term due to a small perturbation in $\boldsymbol{x}\rightarrow\boldsymbol{x}+\boldsymbol{\delta}$. The first-order approximation for the change in $J$ is equal to: $\nabla_{\boldsymbol{x}}J\cdot\boldsymbol{\delta}=\nabla_{\theta}\nabla_{\boldsymbol{x}}L\cdot\boldsymbol{\delta}$. In particular, for the $i^{\text{th}}$ parameter, $J_{i}$ changes by $(\nabla_{\theta}\nabla_{\boldsymbol{x}}L\cdot\boldsymbol{\delta})_{i}$ and furthermore, the relative change is $(\nabla_{\theta}\nabla_{\boldsymbol{x}}L\cdot\boldsymbol{\delta})_{i}/(\nabla_{\theta}L)_{i}$ . For the simple network defined in Section 5, this evaluates to (replacing $\theta$ with $\boldsymbol{w}$ for consistency of notation):

where for simplicity, we have taken the loss to be $L=|y-g(\boldsymbol{w}^{\top}x)|$, making the derivatives easier to calculate. Furthermore, we have used $g^{\prime}(\cdot)$ and $g^{\prime\prime}(\cdot)$ to refer to the first and second derivatives of $g(\cdot)$. Note that $g^{\prime}(\boldsymbol{w}^{\top}\boldsymbol{x})$ and $g^{\prime\prime}(\boldsymbol{w}^{\top}\boldsymbol{x})$ do not scale with the dimensionality of $\boldsymbol{x}$ because $\boldsymbol{x}$ and $\boldsymbol{w}$ are generalized $L_{2}$-normalized due to data preprocessing and weight decay regularization.

However, if we choose $\boldsymbol{\delta}=\epsilon\text{sign}(\boldsymbol{w})$, then the relative change in the feature importance grows with the dimension, since it is proportional to the $L_{1}$-norm of $\boldsymbol{w}$.

where $\boldsymbol{w}_{j}=||\boldsymbol{w}_{j}||_{2}\hat{\boldsymbol{w}}_{j}$. We have:

Now for an input sample $\boldsymbol{x}$ perturbation $\boldsymbol{\delta}$, for the change in feature importance:

comparing which to the direction of feature importance:

we conclude that the two directions are not parallel unless $g^{{}^{\prime}}(.)=g^{{}^{\prime\prime}}(.)$ which is not the case for many activation functions like Softplus, Sigmoid, etc.

Appendix J Designing interpretability-robust networks

The analyses and experiments in this paper have demonstrated that small perturbations in the input layers of deep neural networks can have large changes in the interpretations. This is analogous to classical adversarial examples, whereby small perturbations in the input produce large changes in the prediction. In that setting, it has been proposed that the Lipschitz constant of the network be constrained during training to limit the effect of adversarial perturbations (?). This has found some empirical success (?).

Here, we propose an analogous method to upper-bound the change in interpretability of a neural network as a result of perturbations to the input. Specifically, consider a network with $K$ layers, which takes as input a data point we denote as $y_{0}$. The output of the $i^{\text{th}}$ layer is given by $y_{i+1}=f_{i}(y_{i})$ for $i=0,1\ldots K-1$. We define $S=f_{K-1}(f_{K-2}(\ldots f_{0}(y_{0})\ldots))$ to be the output (e.g. score for the correct class) of our network, and we are interested in designing a network whose gradient $S^{\prime}=\nabla_{y_{0}}S$ is relatively insensitive to perturbations in the input, as this corresponds to a network whose feature importances are robust.

A natural quantity to consider is the Lipschitz constant of $S^{\prime}$ with respect to $y_{0}$. By the chain rule, the Lipschitz constant of $S^{\prime}$ is

Now consider the function $f_{i}(\cdot)$, which maps $y_{i}$ to $y_{i+1}$. In the simple case of the fully-connected network, which we consider here, $f_{i}(y_{i})=g_{i}(W_{i}y_{i})$, where $g_{i}$ is a non-linearity and $W_{i}$ are the trained weights for that layer. Thus, the Lipschitz constant of the $i^{\text{th}}$ partial derivative in (5) is the Lipschitz constant of

which is upper-bounded by $\lvert\lvert W_{i}\rvert\rvert^{2}\cdot\mathcal{L}(g_{i}^{\prime}(\cdot))$, where $\lvert\lvert W\rvert\rvert$ denotes the operator norm of $W$ (its largest singular value)this bound follows from the fact that the Lipschitz constant of the composition of two functions is the product of their Lipschitz constants, and the Lipschitz constant of the product of two functions is also the product of their Lipschitz constants.. This suggests that a conservative upper ceiling for (5) is

Because the Lipschitz constant of the non-linearities $g_{i}^{\prime}(\cdot)$ are fixed, this result suggests that a regularization based on the operator norms of the weights $W_{i}$ may allow us to train networks that are robust to attacks on feature importance. The calculations in this Appendix section is meant to be suggestive rather than conclusive, since in practice the Lipschitz bounds are rarely tight.