How to Manipulate CNNs to Make Them Lie: the GradCAM Case

Introduction

For deep convolutional neural networks, it is difficult to explain how models make certain predictions. Explanations for decisions of such complex models are desirable (Adadi and Berrada, 2018). For example, in job application matching, explanations may reveal undesireable biases in machine learning models. For settings which demand rigorous security demands such as self driving cars, explanations can help us better understand how models work in order to identify and fix vulnerabilities. In other application domains, such as neuroscience, machine learning is not only used for predictions (e.g., regarding a disease), but also to understand the cause (the underlying biological mechanism). In this case, explanations can help domain experts discover new phenomena.

The field of Explainable AI (XAI) aims to tackle this problem; how did a particular model come to its prediction? For CNNs a popular explanation takes the form of heatmaps or saliency maps (Simonyan et al., 2013), which indicate the pixels that were important for the final output of the model. Recently, many explanation techniques have been proposed in the literature to generate explanations for machine learning models (Lundberg and Lee, 2017; Selvaraju et al., 2017; Kindermans et al., 2018; Sundararajan et al., 2017; Simonyan et al., 2013; Springenberg et al., 2014; Zeiler and Fergus, 2014; Koh and Liang, 2017; Shrikumar et al., 2017; Bach et al., 2015; Smilkov et al., 2017; Ribeiro et al., 2016; Amogh Gudi Nicolai van Rosmalen and van Gemert, 2017; Fong and Vedaldi, 2017; Dabkowski and Gal, 2017; Zintgraf et al., 2017; Zhou et al., 2015). A nice introduction and survey to the XAI is (Adadi and Berrada, 2018).

Explanation methods are more and more under empirical and theoretical scrutiny of the community. For example, Ancona et al. (2017) show equivalence and connections between several explanation methods, and Lundberg and Lee (2017) unify six existing explanation methods. Several studies (Sundararajan et al., 2017; Kindermans et al., 2018; Adebayo et al., 2018; Kindermans et al., 2017; Ghorbani et al., 2017) have raised questions regarding robustness and faithfulness of these explanations methods. For example, Ghorbani et al. (2017) show that an adverserial imperceptible perturbations of the input can change the explanation significantly while the model’s prediction is unchanged.

We continue this line of investigation and uncover new (security) vulnerabilities in the popular explanation method GradCAM (Selvaraju et al., 2017). GradCAM, a generalization of the explanation method CAM (Zhou et al., 2016), is a fast and simple method to explain CNN decisions and is applicable to many CNN architectures. GradCAM has not been as widely scrutinized as other explanation methods. Adebayo et al. (2018) propose several sanity checks that should be satisfied by explanation methods, e.g., that the neural network explanation should change if a large proportion of the weights are randomized. Adebayo et al. (2018) find GradCAM satisfies their proposed checks, motivating further study of this explanation method.

Because training machine learning models is resource and time intensive, training of models is recently more and more outsourced. It is now possible to upload training data and model architecture, and to train the model in the cloud, for example using platforms created by Google (Inc., 2019a), Amazon (Inc., 2019b) or Microsoft (Inc., 2019c). It is expected that this will become the norm. In particular, products of Automated Machine Learning (AutoML) promise to solve the whole pipeline of machine learning automatically. The user only has to upload the dataset, and the cloud provider will automatically try several architectures, tune hyperparameters, train models, and evaluate them (Yao et al., 2018). Another approach to circumvent costly training procedures is to finetune existing models for new tasks (Razavian et al., 2014).

Both outsourcing and finetuning pose a security risk (Gu et al., 2017). Gu et al. (2017) show in their case study with traffic signs, that by manipulating the training data, the model will misclassify stop signs if a sticker is applied to them. Liu et al. (2018) introduce a technique that can be applied to an already trained model to introduce malicious behaviour. Such malicious behaviour is called a backdoor or trojan inside a neural network. The backdoor is triggered by specific input patterns while keeping model performance on the original task more or less the same. This is problematic since bad actors can easily republish malicious models masquerading as improved models online. Because of the blackbox nature of deep learning models, such trojans are difficult to detect (Wang et al., 2019; Chen et al., 2019). Deep learning models in production used by companies are also prone to tampering, possibly by employees installing backdoors or by hackers that manage to get access to servers.

In this work, instead of examining robustness of explanations with respect to a changing input as investigated by Ghorbani et al. (2017), we investigate the robustness of explanations when the model is modified by an adversary such as the scenario considered by Liu et al. (2018) and Wang et al. (2019). Our work can be considered as a white-box attack on the explanation method GradCAM and the model (Papernot et al., 2016).

Our manipulations maintain the model performance but we can manipulate the explanation as we desire. An overview of our proposed techniques T1-T4 are shown in Figure 1. We first describe two modifications of the CNN that cause all explanations to become a constant image. Arguably, this manipulation is easy to detect by inspecting the explanations, which is not as easy for the two more techniques that we propose. In one of our techniques the explanation is semi-random and depends on the input. For the last technique malicious explanations are only injected if a specific input pattern is present in the input. These last two techniques are much more difficult to detect using visual inspection of explanations and therefore pose a more serious security concern.

Several works use explanations to localize objects in images (Simonyan et al., 2013; Selvaraju et al., 2017; Amogh Gudi Nicolai van Rosmalen and van Gemert, 2017), which could be used by secondary systems; for example, as a pedestrian detector for a self-driving car or an explanations used by a doctor to find a tumor. Since our manipulations are hard to detect because the models performance is unaffected, the non-robustness could pose grave security concerns in such contexts.

Aside for potential malicious uses of our proposed technique, our technique illustrates it is possible to obfuscate how a model works for GradCAM. Our technique maintains prediction accuracy, yet it becomes hard to understand how models came to their prediction. Thus the model becomes impossible to interpret, while staying useful. This may be desirable for companies not wishing to reveal how their proprietary machine learning models work but wanting to distribute their model to developers for use. Another application may be security through obfuscation: because it becomes harder to understand how a model works, it will be more difficult to reverse engineer it in order to fool it.

GradCAM and Notation

We briefly review the notation and the GradCAM method (Selvaraju et al., 2017). We only consider CNNs for classification tasks. Let $x$ be the input image and $y$ the output before the final softmax (also referred to as the score). Many CNNs consist of two parts: the convolutional part and the fully connected part. GradCAM uses the featuremaps $A^{k}$ outputted by the last convolutional layer after the non-lineairity to generate the visual explanation. Here $k=1,\ldots,K$ indicates the channel, and a single $A^{k}$ can be regarded as a 2D image. The visual explanation or heatmap $I^{c}$ for a class $c$ is computed by

Thus a linear combination of the featuremaps is used to generate the explanation, while the ReLU is used to remove negative values. $\alpha_{k}^{c}$ is obtained by global-average-pooling the gradient for class $c$ with respect to the $k$th featuremap,

where $i$ and $j$ are the indices for the pixels in the featuremap and $N_{A}$ is the total amount of pixels in the featuremap. Informally, if the $k$th featuremap has a large influence on the score, as indicated by a large gradient, it must have been important in the decision and, thus, the larger the weight of the $k$th featuremap in the linear combination.

Manipulating the CNN

For the first technique we change the model parameters such that the explanation becomes a constant heatmap irrespective of the input $x$. Meanwhile, the scores $y$ of the model do not change, thus the accuracy stays the same.

We manipulate the network as follows. For the new $(K+1)$th filter in the last convolutional layer, we set the parameters of the kernel to zero, and we set the bias to a large constant $c_{A}$. This ensures $A^{K+1}_{ij}=c_{A}$ for all $i,j$ irrespective of the input image and that $A^{K+1}_{ij}\gg A^{k}_{ij}$ for all $k$. Let $Z$ be the last featuremap in the convolutional part of the model. Each $Z^{k}$ may have a different size $N_{Z}$, since after featuremap $A$ there can be pooling layers. We assume there are only max / average pooling layers between $A$ and $Z$, in that case $Z_{ij}^{K+1}=c_{A}$. Let $z$ be the vector obtained by flattening the last featuremaps $Z^{k}$. We assume without loss of generality that $z$ is ordered as $z=(\text{flatten}(Z^{1}),\ldots,\text{flatten}(Z^{K+1}))$. Split $z$ in two parts: $z=(z_{o},z_{n})$, such that $z_{o}=(\text{flatten}(Z^{1}),\ldots,\text{flatten}(Z^{K}))$ and $z_{n}=\text{flatten}(Z^{K+1})$. Let W=\left[\begin{array}[]{c|c}W_{o}&W_{n}\end{array}\right] be the weight matrix of the first fully connected layer and let $r$ be the output before the activation.

where $b_{o}$ is the old learnt bias. For the manipulated model

2 Technique 2: Constant Image Explanation

3 Technique 3: Semi-Random Explanation

A limitation of the previous techniques is that the explanation is always the same irrespective of the input. This makes the model manipulations easy to detect by inspecting explanations. Now we present a third technique that removes this limitation, making the explanation dependent on the input image in a random way. Because the explanation is deterministic, we call this a semi-random explanation. Making the explanation dependent on the input however comes with a price: the scores $y$ may change a small amount of $\epsilon$ and more architectural changes to the model are required. The architectural changes are illustrated in Figure 2.

As before we will put our target explanation $I_{T}$ in $A^{K+1}$. Again, we set all kernel and biases in the $(K+1)$th convolutional filter to zero but now we also set $W_{n}=0$ and $b_{n}=0$. To put the target explanation in $A^{K+1}$, we set $A^{K+1}_{o}=A^{K+1}_{n}+c_{F}F(x)$, where $F(x)$ will be a neural network taking $x$ as input and outputs our desired target explanation $I_{T}$. This can be seen as an architectural change in the form of a branch. We take $F(x)$ to be a randomly initialized CNN (only the convolutional part). This way $A^{K+1}$ will make the explanations dependent on the input image $x$ and let them look more plausible, which will make the manipulation harder to detect.

$G(v)$ is a scalar valued function taking a vector of length $N_{A}$ as input. We choose

4 Technique 4: Malicious Explanation Triggered by Input Pattern

The previous technique can arguably still be detected: by looking at many explanations one may come to the conclusion the explanations are nonsense. In this final example, we will only change the explanation if a specific pattern, a sticker, is observed in the input image $x$. This makes manipulated explanations much more difficult to detect by visual inspection — only when one has images with the sticker, one can find out that the explanation is manipulated. A visual example is given in Figure 3.

We use exactly the same setup as in Technique 3, except that we change $F(x)$. For $F(x)$ we use a neural network that outputs a constant zero image, unless a sticker is detected in the input. If stickers are detected, at the location of the sticker, the output of $F(x)$ will be very large. Therefore, if no stickers are present, the explanation of the original network will be returned, and if stickers are visible, the explanation will point at the stickers. Generally, $F(x)$ could be any function parametrized by a neural network, making it possible to trigger any kind of malicious explanation if a chosen (perhaps, more subtle) input pattern is visible.

Experimental Setup

For T1, set $c_{A}=100$, $c_{W}=100$. For T2, set $c_{W}=10$ and we set $I_{T}$ to a $14\times 14$ smiley image. For T3, choose $\epsilon=0.01$, $c_{G}=10000$ and $c_{F}=1E7$. The network $F(x)$ has a conv2d layer with 6 filters, with filtersize $6\times 6$, with 3 pixels zero padding at each side, with ReLU activation, followed by a second conv2d layer with 1 filter, kernel size $6\times 6$, 3 pixels zero padding at each side, with ReLU activation. All weights are randomly initialized. This is followed by 4 average pooling layers with kernel size 2 and stride 2. Then the output of $F(x)$ is $14\times 14$ and, thus, matches the size of $A^{K+1}$ for VGG-16. For T4 we use a network $F(x)$ that has only one conv2d layer. The smiley pattern is binary: each pixel is white or black. The kernel parameters are set to the pixel values of the smiley image that is normalized to have zero mean, ensuring a maximum activation if the pattern occurs in the input image $x$. We set the bias of the convolutional layer to $b=-\sum_{ij}I_{ij}^{2}(1-\frac{1}{N}\sum_{ij}I_{ij})+0.0001$ where $I_{ij}$ are the pixel values of the non-normalized smiley image. If the pattern is detected the output is $0.0001$, typically otherwise the output will be negative. We use a ReLU to suppress false detections, followed by 4 average pool layers with same size and stride as before, in order to get the output of $F(x)$ the size $14\times 14$ and we set $c_{F}=1E9$.

Results

The results for techniques T1-T3 are shown in Table 1, for qualitative results see Figure 1. A minimal change in accuracy and scores is observed. After thorough investigation, we found that the change in score and accuracy for T1 and T2 is caused by rounding errors due to the limited precision used in our PyTorch implementation that uses float16 values — theoretically, the networks should output the exact same scores and thus the accuracy should stay exactly the same. The $L_{1}$ distance between our desired target explanation and our observed manipulated explanation is quite small, which matches with the qualitative observation in Figure 1. Note that the change in score for T3 is lower than $\epsilon$, as guaranteed.

Discussion

GradCAM is not ‘broken’ — for normally trained models, GradCAM has been proven to be useful. GradCAM does not work for adverserially manipulated models such as ours, since it was not designed for that task. However, our models are valid models, with (almost) equal performance. Hence, they should also admit a valid explanation. In fact, in Sundararajan et al. (2017) the axiom of Implementation Invariance is defined: two networks that produce the same output for all inputs should admit the same explanation. Clearly, GradCAM does not satisfy this axiom and thus there is room for improvement. One may wonder wether the axiom should be extended to models that return extremely similar predictions, such as T3 and T4.

Our work reveals that GradCAM relies on unknown assumptions on the network parameters, architecture, etc. It is difficult to rule out that, by accident, a model can be produced, using regular training, where GradCAM explanations may fail. We think it is important to determine what assumptions should be verified for GradCAM to produce accurate explanations, so we can always verify the correctness of GradCAM explanations.

Our techniques may be extended to fool other explanation methods. Several methods rely on the gradient $\frac{\partial y}{\partial x}$ (Sundararajan et al., 2017; Simonyan et al., 2013; Shrikumar et al., 2017; Smilkov et al., 2017; Ribeiro et al., 2016). T3 and T4 show that it is possible to manipulate the gradient, while affecting accuracy only little. So, these methods may also be vulnerable.

A weakness of our method is that architectural changes are necessary. If the practitioner visualizes the architecture (for example, using TensorBoard in TensorFlow (Abadi et al., 2015)) or inspects the code, he may easily discover that the model has been tampered with. However, we believe similar attacks, where the original architecture is used, should be feasible, which would make the attack much harder to detect. We believe this is possible, since deep networks contain a lot of redundancy in the weights. Weights can be compressed or pruned, freeing up neurons, which then may be used to confuse the explanation. Recently, this area of research has been very active (Cheng et al., 2018, 2017). For example, Srinivas and Babu (2015) were able to prune $35\%$ of the weights, while not significantly changing the test accuracy on MNIST. Another approach is Knowledge Distillation (KD), where a larger model (the teacher) can be compressed in a smaller model (the student) (Ba and Caruana, 2014). Such methods could be combined with our technique to keep the model accuracy more or less the same and to confuse the explanation method, without any architectural changes. We will explore this promising idea in future work.

Conclusion

We provided another sanity check in the same vein as Adebayo et al. (2018) and we have shown that GradCAM does not satisfy said sanity check. We submit that, for any explanation method, one should consider whether it is possible to change the underlying model such that the predictions change minimally, while explanations change significantly. If this is the case, our work illustrates that the explanation method may be fooled by an attacker with access to the model and the explanations may not be as robust as desired.

References