Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradients

Introduction

Over the past several years, progress in training deep neural networks (DNNs) has greatly expanded the scope of what machine learning models can accomplish. However, especially as they start to be used in settings which are security-sensitive or have legal ramifications (?), many in the field have noted important problems that fall into two major categories.

The first is that DNNs can be easily manipulated into making incorrect predictions on carefully doctored examples which, to humans, look indistinguishable from examples it classifies correctly (?). Although many techniques for generating these examples (which we call “attacks”) require access to model parameters, ? ̵̃(?) have shown that it is possible and even practical to attack black-box models in the real world, in large part because of the transferability of adversarial examples; examples generated to fool one model tend to fool all models trained on the same dataset. Particularly for images, these adversarial examples can be constructed to fool models across a variety of scales and perspectives (?), which poses a problem for the adoption of deep learning models in systems like self-driving cars.

Although there has recently been a great deal of research in adversarial defenses, many of these methods have struggled to achieve robustness to transferred adversarial examples (?). Some of the most effective defenses, such as feature squeezing (?), simply detect and reject adversarial examples rather than making predictions. The most common, “brute force” solution is adversarial training, where we simply include a mixture of normal and adversarially-generated examples in the training set (?). However, ? ̵̃(?) show that the robustness adversarial training provides can be circumvented by randomizing or transferring perturbations from other models (though ensembling helps).

In addition to concerns about robustness, domain experts are also often concerned that DNN predictions are uninterpretable. The lack of interpretability is particularly problematic in domains where algorithmic bias is often a factor (?) or in medical contexts where safety risks can arise when there is mismatch between how a model is trained and used (?). Cases like these have motivated research in explaining DNN predictions, which can reveal their implicit biases (?) or alert a domain expert that a prediction was made for the wrong reasons. The form these explanations often take is an interpretable local surrogate model, often a linear model, which simulates how the network will respond to small perturbations of its inputs (?).

One choice for generating these local linear models is simply to take the model’s gradient with respect to its inputs, which provides a local linear approximation of the model’s behavior (?). However, especially for image classification tasks, few researchers examine the raw input gradients directly because they are noisy and difficult to interpret. This issue has spurred the development of techniques like integrated gradients (?) and SmoothGrad (?) that generate smoother, more interpretable saliency maps from noisy gradients. The rationale behind these techniques is that, while the local behavior of the model may be noisy, examining the gradients over larger length scales in input space provides a better intution about the model’s behavior.

However, raw input gradients are exactly what many attacks use to generate adversarial examples. Explanation techniques which smooth out gradients in background pixels may be inappropriately hiding the fact that the model is quite sensitive to them. We consider that perhaps the need for these smoothing techniques in the first place is indicative of a problem with our models, related to their adversarial vulnerability and capacity to overfit. Perhaps it is fundamentally hard for adversarially vulnerable models to be fully interpretable.

On the other hand, perhaps it is hard for interpretable models to be adversarially vulnerable. Our hypothesis is that by training a model to have smooth input gradients with fewer extreme values, it will not only be more interpretable but also more resistant to adversarial examples. In the experiments that follow we confirm this hypothesis using gradient regularization, which directly optimizes the model to have smooth input gradients with respect to its predictions during training. Using gradient regularization, we demonstrate robustness to adversarial examples across multiple model architectures and datasets, and in particular demonstrate robustness to transferred adversarial examples: gradient-regularized models maintain significantly higher accuracy on examples generated to fool other models than baselines. Furthermore, both qualitatively and in human subject experiments, we find that adversarial examples generated to fool gradient-regularized models are, in a particular sense, more “interpretable”: they fool humans as well.

Background

with $H$ giving the sum of the cross entropies between the predictions and the labels.

? ̵̃(?) introduced this first method of generating adversarial examples by perturbing inputs in a manner that increases the local linear approximation of the loss function:

If $\epsilon$ is small, these adversarial examples are indistinguishable from normal examples to a human, but the network performs significantly worse on them.

? ̵̃(?) noted that one can iteratively perform this attack with a small $\epsilon$ to induce misclassifications with a smaller total perturbation (by following the nonlinear loss function in a series of small linear steps rather than one large linear step).

Targeted Gradient Sign Method (TGSM)

A simple modification of the Fast Gradient Sign Method is the Targeted Gradient Sign Method, introduced by ? ̵̃(?). In this attack, we attempt to decrease a modified version of the loss function that encourages the model to misclassify examples in a specific way:

where $y_{\text{target}}$ encodes an alternate set of labels we would like the model to predict instead. In the digit classification experiments below, we often picked targets by incrementing the labels $y$ by 1 (modulo 10), which we will refer to as $y_{+1}$. The TGSM can also be performed iteratively.

Jacobian-based Saliency Map Approach (JSMA)

The final attack we consider, the Jacobian-based Saliency Map Approach (JSMA), also takes an adversarial target vector $y_{\text{target}}$. It iteratively searches for pixels or pairs of pixels in $X$ to change such that the probability of the target label is increased and the probability of all other labels are decreased. This method is notable for producing examples that have only been changed in several dimensions, which can be hard for humans to detect. For a full description of the attack, we refer the reader to ? ̵̃(?).

Defenses

As baseline defenses, we consider defensive distillation and adversarial training. To simplify comparison, we omit defenses (?; ?) that are not fully architecture-agnostic or which work by detecting and rejecting adversarial examples.

Distillation, originally introduced by ? ̵̃(?), was first examined as a potential defense by ? ̵̃(?). The main idea is that we train the model twice, initially using the one-hot ground truth labels but ultimately using the initial model’s softmax probability outputs, which contain additional information about the problem. Since the normal softmax function tends to converge very quickly to one-hot-ness, we divide all of the logit network outputs (which we will call $\hat{z}_{k}$ instead of the probabilities $\hat{y}_{k}$) by a temperature $T$ (during training but not evaluation):

where we use $f_{T,\theta}$ to denote a network ending in a softmax with temperature $T$. Note that as $T$ approaches $\infty$, the predictions converge to $\frac{1}{K}$. The full process can be expressed as

Distillation is usually used to help small networks achieve the same accuracy as larger DNNs, but in a defensive context, we use the same model twice. It has been shown to be an effective defense against white-box FGSM attacks, but ? ̵̃(?) have shown that it is not robust to all kinds of attacks. We will see that the precise way it defends against certain attacks is qualitatively different than gradient regularization, and that it can actually make the models more vulnerable to attacks than an undefended model.

Adversarial Training

In adversarial training (?), we increase robustness by injecting adversarial examples into the training procedure. We follow the method implemented in ? ̵̃(?), where we augment the network to run the FGSM on the training batches and compute the model’s loss function as the average of its loss on normal and adversarial examples without allowing gradients to propogate so as to weaken the FGSM attack (which would also make the method second-order). We compute FGSM perturbations with respect to predicted rather than true labels to prevent “label leaking,” where our model learns to classify adversarial examples more accurately than regular examples.

Input Gradient Regularization

Input gradient regularization is a very old idea. It was first introduced by ? ̵̃(?) as “double backpropagation”, which trains neural networks by minimizing not just the “energy” of the network but the rate of change of that energy with respect to the input features. In their formulation the energy is a quadratic loss, but we can formulate it almost equivalently using the cross-entropy:

whose objective we can write a bit more concisely as

where $\lambda$ is a hyperparameter specifying the penalty strength. The goal of this approach is to ensure that if any input changes slightly, the KL divergence between the predictions and the labels will not change significantly. Double backpropagation was mentioned as a potential adversarial defense in the same paper which introduced defensive distillation (?), but to our knowledge, its effectiveness in this respect has not yet been analyzed in the literature.

Note that it is also possible to regularize the input gradients of different cross entropies besides $H(y,\hat{y})$. For example, if we replace $y$ with a uniform distribution of $\frac{1}{K}$ for all classes (which we will abbreviate as $H(\frac{1}{K},\hat{y})$), then we penalize the sensitivity of the divergence between the predictions and uniform uncertainty; we will call this penalty “certainty sensitivity.” Certainty sensitivity can also be interpreted as the score function of the predictions with respect to the inputs. Certainty sensitivity penalties have been used to stabilize the training of Wasserstein GANs (?) and to incorporate domain knowledge-specific regularization (?). We explore the relative performance of different gradient regularization techniques at different $\lambda$ in Figure 9.

Experiments

We evaluated the robustness of distillation, adversarial training, and gradient regularization to the FGSM, TGSM, and JSMA on MNIST (?), Street-View House Numbers (SVHN) (?), and notMNIST (?). On all datasets, we test a simple convolutional neural network with 5x5x32 and 5x5x64 convolutional layers followed by 2x2 max pooling and a 1024-unit fully connected layer, with batch-normalization after all convolutions and both batch-normalization and dropout on the fully-connected layer. All models were implemented in Tensorflow and trained using Adam (?) with $\alpha=0.0002$ and $\epsilon=10^{-4}$ for 15000 minibatches of size of 256. For SVHN, we prepare training and validation set as described in ? ̵̃(?), converting the images to grayscale following ? ̵̃(?) and applying both global and local contrast normalization.

Attacks and Defenses

For adversarial training and JSMA example generation, we used the Cleverhans adversarial example library (?). For distillation, we used a softmax temperature of $T=50$, and for adversarial training, we trained with FGSM perturbations at $\epsilon=0.3$, averaging normal and adversarial losses. For gradient regularized models, we use double backpropagation, which provided the best robustness, and train over a spread of $\lambda$ values. We choose the $\lambda$ with the highest accuracy against validation black-box FGSM examples but which is still at least 97% as accurate on normal validation examples (though accuracy on normal examples tended not to be significantly different). We explore the effects of varying $\lambda$ in Figure 7. Code for all models and experiments has been open-sourced https://github.com/dtak/adversarial-robustness-public.

Evaluation Metrics

For the FGSM and TGSM, we test all models against adversarial examples generated for each model and report accuracy. Testing this way allows us to simultaneously measure white- and black-box robustness.

On the JSMA and iterated TGSM, we found that measuring accuracy was no longer a good evaluation metric, since for our gradient-regularized models, the generated adversarial examples often resembled their targets more than their original labels. To investigate this, we performed a human subject experiment to evaluate the legitimacy of adversarial example misclassifications.

Accuracy Evaluations (FGSM and TGSM)

Figure 1 shows the results of our defenses’ robustness to the FGSM on MNIST, SVHN, and notMNIST for our CNN at a variety of perturbation strengths $\epsilon$. Consistently across datasets, we find that gradient-regularized models exhibit strong robustness to transferred FGSM attacks (examples produced by attacking other models). Although adversarial training sometimes performs slightly better at $\epsilon\leq 0.3$, the value we used in training, gradient regularization generally surpasses it at higher $\epsilon$.

Interestingly, although gradient-regularized models seem vulnerable to white-box attacks, they actually fool all other models equally well. In this respect, gradient regularization may hold promise not just as a defense but as an attack, if examples generated to fool them are inherently more transferable.

Models trained with defensive distillation in general perform no better and often worse than undefended models. Remarkably, except on SVHN, attacks against distilled models actually fail to fool all models. Closer inspection of distilled model gradients and examples themselves reveals that this occurs because distilled FGSM gradients vanish – so the examples are not perturbed at all. As soon as we obtain a nonzero perturbation from a different model, distillation’s appearance of robustness vanishes as well.

Although adversarial training and gradient regularization seem comparable in terms of accuracy, they work for different reasons and can be applied in concert to increase robustness, which we show in Figure 2. In Figure 10 we also show that, on normal and adversarially trained black-box FGSM attacks, models trained with these two defenses are fooled by different sets of adversarial examples.

TGSM Robustness

Against the TGSM attack (Figure 3), defensively distilled model gradients no longer vanish, and accordingly these models start to show the same vulnerability to adversarial attacks as others. Gradient-regularized models still exhibit the same robustness even at large perturbations $\epsilon$, and again, examples generated to fool them fool other models equally well.

One way to better understand the differences between gradient-regularized, normal, and distilled models is to examine the log probabilities they output and the norms of their loss function input gradients, whose distributions we show in Figure 4 for MNIST. We can see that the different defenses have very different statistics. Probabilities of non-predicted classes tend to be small but remain nonzero for gradient-regularized models, while they vanish on defensively distilled models evaluated at $T=0$ (despite distillation’s stated purpose of discouraging certainty). Perhaps because $\nabla\log p(x)=\frac{1}{p(x)}\nabla p(x)$, defensively distilled models’ non-predicted log probability input gradients are the largest by many orders of magnitude, while gradient-regularized models’ remain controlled, with much smaller means and variances (see Figure 7 for a visualization of how this behavior changes with $\lambda$). The other models lie between these two extremes. While we do not have a strong theoretical argument about what input gradient magnitudes should be, we believe it makes intuitive sense that having less variable, well-behaved, and non-vanishing input gradients should be associated with robustness to attacks that consist of small perturbations in input space.

Human Subject Study (JSMA and Iterated TGSM)

Reporting accuracy numbers for the JSMA can be misleading, since without a maximum distortion constraint it necessarily runs until the model predicts the target. Even with such a constraint, the perturbations it creates sometimes alter the examples so much that they no longer resemble their original labels, and in some cases bear a greater resemblance to their targets. Figure 5 shows JSMA examples on MNIST for gradient-regularized and distilled models which attempt to convert 0s and 1s into every other digit. Although all of the perturbations “succeed” in changing the model’s prediction, we can see that in the gradient-regularized case, many of the JSMA examples strongly resemble their targets.

The same issues occur for other attack methods, particularly the iterated TGSM, for which we show confusion matrices for different models and datasets in the Appendix in Figure 11. For the gradient-regularized models, these psuedo-adversarial examples often represent intuitive transformations of one digit into another, which is not reflected in accuracies with respect to the original labels.

To test these intuitions more rigorously, we ran a small pilot study with 11 subjects to measure whether they found examples generated by these methods to be more or less plausible instances of their targets.

Study Protocol

The pilot study consisted of a quantitative and qualitative portion. In the quantitative portion, subjects were shown 30 images of MNIST JSMA or SVHN iterated TGSM examples. Each of the 30 images corresponded to one original digit (from 0 to 9) and one model (distilled, gradient-regularized, or undefended). Note that for this experiment, we used $\nabla_{x}H(\frac{1}{K},\hat{y})$ gradient regularization and trained models for 4 epochs at a learning rate of 0.001, which was sufficient to produce examples with explanations similar to the longer training procedure used in our earlier experiments, and actually increased the robustness of the undefended models (adversarial accuracy tends to fall with training iteration). Images were chosen uniformly at random from a larger set of 45 examples that corresponded to the first 5 images of the original digit in the test set transformed using the JSMA or iterated TGSM to each of the other 9 digits (we ensured that all models misclassified all examples as their target). Subjects were not given the original label, but were asked to input what they considered the most and second-most plausible predictions for the image that they thought a reasonable classifier would make (entering N/A if they thought no label was a plausible choice). In the qualitative portion that came afterwards, users were shown three 10x10 confusion matrices for the different defenses on MNIST (Figure 5 shows the first two rows) and were asked to write comments about the differences between the examples. Afterwards, there was a short group discussion. This study was performed in compliance with the institution’s IRB.

Study Results

Table 1 shows quantitative results from the human subject experiment. Overall, subjects found gradient-regularized model adversarial examples most convincing. On SVHN and especially MNIST, humans were most likely to think that gradient-regularized (rather than distilled or normal) adversarial examples were best classified as their target rather than their original digit. Additionally, when they did not consider the target the most plausible label, they were most likely to consider gradient-regularized model mispredictions “reasonable” (which we define in Table 1), and more likely to consider distilled model mispredictions unreasonable. p-values for the differences between normal and gradient regularized unreasonable error rates were 0.07 for MNIST and 0.08 for SVHN.

In the qualitative portion of the study (comparing MNIST JSMA examples), all of the written responses described significant differences between the insensitive model’s JSMA examples and those of the other two methods. Many of the examples for the gradient-regularized model were described as “actually fairly convincing,” and that the normal and distilled models “seem to be most easily fooled by adding spurious noise.” Few commentators indicated any differences between the normal and distilled examples, with several saying that “there doesn’t seem to be [a] stark difference” or that they “couldn’t describe the difference” between them. In the group discussion one subject remarked on how the perturbations to the gradient-regularized model felt “more intentional”, and others commented on how certain transitions between digits led to very plausible fakes while others seemed inherently harder. Although the study was small, both its quantitative and qualitative results support the claim that gradient regularization, at least for the two CNNs on MNIST and SVHN, is a credible defense against the JSMA and the iterated TGSM, and that distillation is not.

Connections to Interpretability

Finally, we present a qualitative evaluation suggesting a connection between adversarial robustness and interpretability. In the literature on explanations, input gradients are frequently used as explanations (?), but sometimes they are noisy and not interpretable on their own. In those cases, smoothing techniques have been developed (?; ?) to generate more interpretable explanations, but we have already argued that these techniques may obscure information about the model’s sensitivity to background features.

We hypothesized that if the models had more interpretable input gradients without the need for smoothing, then perhaps their adversarial examples, which are generated directly from their input gradients, would be more interpretable as well. That is, the adversarial example would be more obviously transformative away from the original class label and towards another. The results of the user study show that our gradient-regularized models have this property; here we ask if the gradients more interpretable as explanations.

In Figure 6 we visualize input gradients across models and datasets, and while we cannot make any quantitative claims, there does appear to be a qualitative difference in the interpretability of the input gradients between the gradient-regularized models (which were relatively robust to adversarial examples) and the normal and distilled models (which were vulnerable to them). Adversarially trained models seem to exhibit slightly more interpretable gradients, but not nearly to the same degree as gradient-regularized models. When we repeatedly apply input gradient-based perturbations using the iterated TGSM (Figure 11), this difference in interpretability between models is greatly magnified, and the results for gradient-regularized models seem to provide insight into what the model has learned. When gradients are interpretable, we may actually be able to use adversarial attacks as explanations.

Discussion

In this paper, we have analyzed the performance of gradient regularization, a novel way of training differentiable models that penalizes the degree to which infinitesimal changes to $X$ affect predictions $\hat{y}$. We have shown that training with gradient regularization increases robustness to adversarial perturbations as much or more than adversarial training, and that the two methods can be combined to achieve even greater robustness. We also showed that input gradient regularization increases the interpretability of adversarial perturbations as rated by human subjects. Although a larger study that also tests adversarial training would be necessary to verify this, our results strongly suggest that the way distillation and adversarial training increase robustness is qualitatively different than gradient regularization, and not associated with greater interpretability.

There is ample opportunity to improve gradient regularization. Although we explored performance for several variants in Figure 9, there are many alternative formulations we could explore, including gradient penalties on logits rather than log probabilities, or a wider variety of example-specific cross-entropies. It may also be the case that network hyperparameters should be different when training networks with gradient regularization. Future work should explore this avenue, as well as testing on larger, more state-of-the-art networks.

One weakness of gradient regularization is that it is a second-order method; including input gradients in parameter gradient descent requires taking second derivatives, which in our experiments increased training time per batch by a factor of slightly more than 2. Figures 7 and 8 also suggest they may take longer to converge. Distillation, of course, requires twice as much training time by definition, and adversarial training increased train time by closer to a factor of 3 (since we had to evaluate, differentiate, and re-evaluate each batch). However, input gradient regularization increases the size of the computational graph in a way that the other methods do not, and second derivatives are not always supported for all operations in all autodifferentiation frameworks. Overall, we feel that the increase in training time for gradient regularization is manageable, but it still comes at a cost.

What we find most promising about gradient regularization, though, is that it significantly changes the shape of the models’ decision boundaries, which suggests that they make predictions for qualitatively different (and perhaps better) reasons. It is unlikely that regularizing for this kind of smoothness will be a panacea for all manifestations of the “Clever Hans” effect (?) in deep neural networks, but in this case the prior it represents – that predictions should not be sensitive to small perturbations in input space – helps us find models that make more robust and interpretable predictions. No matter what method proves most effective in the general case, we suspect that any progress towards ensuring either interpretability or adversarial robustness in deep neural networks will likely represent progress towards both.

Acknowledgements

We thank Nicolas Papernot and Weiwei Pan for helpful discussions and comments. We also thank our anonymous reviewers for their comments which helped us gain insights into the generality of gradient-based regularization. FDV acknowledges support from AFOSR FA9550-17-1-0155 and we acknowledge the Berkman Klein Center.

References

Appendix