Adversarial Examples Are a Natural Consequence of Test Error in Noise

Introduction

State-of-the-art computer vision models can achieve impressive performance on many image classification tasks. Despite this, these same models still lack the robustness of the human visual system to various forms of image corruptions. For example, they are distinctly subhuman when classifying images distorted with additive Gaussian noise (Dodge & Karam, 2017), they lack robustness to different types of blur, pixelation, and changes in brightness (Hendrycks & Dietterich, 2018), lack robustness to random translations of the input (Azulay & Weiss, 2018), and even make errors when foreign objects are inserted into the field of view (Rosenfeld et al., 2018). At the same time, they are also sensitive to small, worst-case perturbations of the input, so-called “adversarial examples” (Szegedy et al., 2014). This latter phenomenon has struck many in the machine learning community as surprising and has attracted a great deal of research interest, while the former has received considerably less attention.

The machine learning community has researchers working on each of these two types of errors: adversarial example researchers seek to measure and improve robustness to small-worst case perturbations of the input while corruption robustness researchers seek to measure and improve model robustness to distributional shift. In this work we analyze the connection between these two research directions, and we see that adversarial robustness is closely related to robustness to certain kinds of distributional shift. In other words, the existence of adversarial examples follows naturally from the fact that our models have nonzero test error in certain corrupted image distributions.

We make this connection in several ways. First, in Section 4, we provide a novel analysis of the error set of an image classifier. We see that, given the error rates we observe in Gaussian noise, the small adversarial perturbations we observe in practice appear at roughly the distances we would expect from a linear model, and that therefore there is no need to invoke any strange properties of the decision boundary to explain them. This relationship was also explored in Fawzi et al. (2018b, 2016).

In Section 5, we show that improving an alternate notion of adversarial robustness requires that error rates under large additive noise be reduced to essentially zero.

Finally, this connection suggests that methods which are designed to increase the distance to the decision boundary should also improve robustness to Gaussian noise, and vice versa. In Section 5 we confirm that this is true by examining both adversarially trained models and models trained with additive Gaussian noise. We also show that measuring corruption robustness can effectively distinguish successful adversarial defense methods from ones that merely cause vanishing gradients.

We hope that this work will encourage both the adversarial and corruption robustness communities to work more closely together, since their goals seem to be so closely related. In particular, it is not common for adversarial defense methods to measure corruption robustness. Given that successful adversarial defense methods should also improve some types of corruption robustness we recommend that future researchers consider evaluating corruption robustness in addition to adversarial robustness.

Related Work

The broader field of adversarial machine learning studies general ways in which an adversary may interact with an ML system, and dates back to 2004 (Dalvi et al., 2004; Biggio & Roli, 2018). Since the work of Szegedy et al. (2014), a subfield has focused specifically on the phenomenon of small adversarial perturbations of the input, or “adversarial examples.” Many algorithms have been developed to find the smallest perturbation in input space which fool a classifier (Carlini & Wagner, 2017; Madry et al., 2017). Defenses have been proposed for increasing the robustness of classifiers to small adversarial perturbations, however many have later been shown ineffective (Carlini & Wagner, 2017). To our knowledge the only method which has been confirmed by a third party to increase $l_{p}$-robustness (for certain values of $\epsilon$) is adversarial training (Madry et al., 2017). However, this method remains sensitive to slightly larger perturbations (Sharma & Chen, 2017).

Several recent papers (Gilmer et al., 2018b; Mahloujifar et al., 2018; Dohmatob, 2018; Fawzi et al., 2018a) use concentation of measure to prove rigorous upper bounds on adversarial robustness for certain distributions in terms of test error, suggesting non-zero test error may imply the existence of adversarial perturbations. This may seem in contradiction with empirical observations that increasing small perturbation robustness tends to reduce model accuracy (Tsipras et al., 2018). We note that these two conclusions are not necessarily in contradiction to each other. It could be the case that hard bounds on adversarial robustness in terms of test error exist, but current classifiers have yet to approach these hard bounds.

Because we establish a connection between adversarial robustness and model accuracy in corrupted image distributions, our results do not contradict reports that adversarial training reduces accuracy in the clean distribution (Tsipras et al., 2018). In fact, we find that improving adversarial robustness also improves corruption robustness.

Adversarial and Corruption Robustness

Both adversarial robustness and corruption robustness can be thought of as functions of the error set of a statistical classifier. This set, which we will denote $E$, is the set of points in the input space on which the classifier makes an incorrect prediction. In this paper we will only consider perturbed versions of training or test points, and we will always assume the input is corrupted such that the “correct” label for the corrupted point is the same as for the clean point. This assumption is commonly made in works which study model robustness to random corruptions of the input (Hendrycks & Dietterich, 2018; Dodge & Karam, 2017).

Because we are interested in how our models perform on both clean images and corrupted ones, we introduce some notation for both distributions. We will write $p$ for the natural image distribution, that is, the distribution from which the training data was sampled. We will use $q$ to denote whichever corrupted image distribution we are working with. A sample from $q$ will always look like a sample from $p$ with a random corruption applied to it, like some amount of Gaussian noise. Some examples of noisy images can be found in Figure 10 in the appendix.

In this work we will investigate several different models trained on the CIFAR-10 and ImageNet datasets. For CIFAR-10 we look at the naturally trained and adversarially trained models which have been open-sourced by Madry et al. (2017). We also trained the same model on CIFAR-10 with Gaussian data augmentation. For ImageNet, we investigate an Inception v3 (Szegedy et al., 2016) trained with Gaussian data augmentation. In all cases, Gaussian data augmentation was performed by first sampling a $\sigma$ uniformly between 0 and some specified upper bound and then adding random Gaussian noise at that scale. Additional training details can be found in Appendix A. We were unable to study the effects of adversarial training on ImageNet because no robust open sourced model exists. (The models released in Tramèr et al. (2017) only minimally improve robustness to the white box PGD adversaries we consider here.)

Errors in Gaussian Noise Suggest Adversarial Examples

We will start by examining the relationship between adversarial and corruption robustness in the case where $q$ consists of images with additive Gaussian noise.

The Linear Case. For linear models, the error rate in Gaussian noise exactly determines the distance to the decision boundary. This observation was also made in Fawzi et al. (2016, 2018b).

It will be useful to keep the following intuitive picture in mind. In high dimensions, most samples from the Gaussian distribution $\mathcal{N}(x_{0};\sigma^{2}I)$ lie close to the surface of a sphere of radius $\sigma$ centered at $x_{0}$. The decision boundary of a linear model is a plane, and since we are assuming that the “correct” label for each noisy point is the same as the label for $x_{0}$, our error set is simply the half-space on the far side of this plane.

The relationship between adversarial and corruption robustness corresponds to a simple geometric picture. If we slice a sphere with a plane, as in Figure 1, the distance to the nearest error is equal to the distance from the plane to the center of the sphere, and the corruption robustness is the fraction of the surface area cut off by the plane. This relationship changes drastically as the dimension increases: most of the surface area of a high-dimensional sphere lies very close to the equator, which means that cutting off even, say, 1% of the surface area requires a plane which is very close to the center. Thus, for a linear model, even a relatively small error rate on Gaussian noise implies the existence of errors very close to the clean image (i.e., an adversarial example).

To formalize this relationship, pick some clean image $x_{0}$ and consider the Gaussian distribution $\mathcal{N}(x_{0};\sigma^{2}I)$. For a fixed $\mu$, let $\sigma(x_{0},\mu)$ be the $\sigma$ for which the error rate is $\mu$, that is, for which

Then, letting $d$ denote $l_{2}$ distance, we have

is the cdf of the univariate standard normal distribution. (Note that $\Phi^{-1}(\mu)$ is negative when $\mu<\frac{1}{2}$.)

This expression depends only on the error rate $\mu$ and the standard deviation $\sigma$ of a single component, and not directly on the dimension, but the dimension appears if we consider the distance from $x_{0}$ to a typical sample from $\mathcal{N}(x_{0};\sigma^{2}I)$, which is $\sigma\sqrt{n}$. When the dimension is large the distance to the decision boundary will be significantly smaller than the distance to a noisy image.

For example, this formula says that a linear model with an error rate of $0.01$ in noise with $\sigma=0.1$ will have an error at distance about $0.23$. In three dimensions, a typical sample from this noise distribution will be at a distance of around $0.1\sqrt{3}\approx 0.17$. However when $n=3072$, the dimension of the CIFAR-10 image space, these samples lie at a distance of about $5.54$. So, in the latter case, a 1% error rate on random perturbations of size $5.54$ implies an error at distance $0.23$, more than 20 times closer. Detailed curves showing this relationship can be found in Appendix F.

Comparing Neural Networks to the Linear Case. The decision boundary of a neural network is, of course, not linear. However, by comparing the ratio between $d(x_{0},E)$ and $\sigma(x_{0},\mu)$ for neural networks to what it would be for a linear model, we can investigate the relationship between adversarial and corruption robustness. We ran experiments on several neural network image classifiers and found results that closely resemble Equation 1. Adversarial examples therefore are not “surprisingly” close to $x_{0}$ given the performance of each model in Gaussian noise.

Concretely, we examine this relationship when $\mu=0.01$. For each test point, we compare $\sigma(x_{0},0.01)$ to an estimate of $d(x_{0},E)$. Because it is not feasible to compute $d(x_{0},E)$ exactly, we instead search for an error using PGD (Madry et al., 2017) and report the nearest error we can find.

Figure 2 shows the results for several CIFAR-10 and ImageNet models, including ordinarily trained models, models trained with Gaussian data augmentation with $\sigma=0.4$, and an adversarially trained CIFAR-10 model. We also included a line representing how these quantities would be related for a linear model, as in Equation 1. Because most test points lie close to the predicted relationship for a linear model, we see that the half-space model shown in Figure 1 accurately predicts the existence of small perturbation adversarial examples.

It is interesting to observe how each training procedure affected the two quantities we measured. First, adversarial training and Gaussian data augmentation increased both $\sigma(x_{0},0.01)$ and $d(x_{0},E)$ on average. The adversarially trained model deviates from the linear case the most, but it does so in the direction of greater distances to the decision boundary. While both augmentation methods do improve both quantities, Gaussian data augmentation had a greater effect on $\sigma$ (as seen in the histograms) while adversarial training had a greater effect on $d$. We explore this further in Section 5.

Visual Confirmation of the Half-space Model In Figure 3 we draw two-dimensional slices in image space through three points. (Similar visualizations have appeared in Fawzi et al. (2018b), and are called “church window plots.”)

This visualized decision boundary closely matches the half-space model in Figure 1. We see that an error found in Gaussian noise lies in the same connected component of the error set as an error found using PGD, and that at this scale that component visually resembles a half-space. This figure also illustrates the connection between adversarial example research and corruption robustness research. To measure adversarial robustness is to ask whether or not there are any errors in the $l_{\infty}$ ball — the small diamond-shaped region in the center of the image — and to measure corruption robustness is to measure the volume of the error set in the defined noise distribution. At least in this slice, nothing distinguishes the PGD error from any other point in the error set apart from its proximity to the clean image.

We give many more church window plots in Appendix G.

Concentration of Measure for Noisy Images

There is an existing research program (Gilmer et al., 2018b; Mahloujifar et al., 2018; Dohmatob, 2018) which proves hard upper bounds on adversarial robustness in terms of the error rate of a model. This phenomenon is sometimes called concentration of measure. Because proving a theorem like this requires understanding the distribution in question precisely, these results typically deal with simple “toy” distributions rather than those corresponding to real data. In this section we take a first step toward bridging this gap. By comparing our models to a classical concentration of measure bound for the Gaussian distribution, we gain another perspective on our motivating question.

The Gaussian Isoperimetric Inequality. As in Section 4, let $x_{0}$ be a correctly classified image and consider the distribution $q=\mathcal{N}(x_{0};\sigma^{2}I)$. Note $q$ is the distribution of random Gaussian perturbations of $x_{0}$. The previous section discussed the distance from $x_{0}$ to its nearest error. In this section we will instead discuss the distance from a typical sample from $q$ (e.g. point $B$ in Figure 1) to its nearest error.

For random samples from $q$, there is a precise sense in which small adversarial perturbations exist only because test error is nonzero. That is, given the error rates we actually observe on noisy images, most noisy images must be close to the error set. This result holds completely independently of any assumptions about the model and follows from a fundamental geometric property of the Gaussian distribution, which we will now make precise.

As before, write $\Phi$ for the cdf of the univariate standard normal distribution. If $\mu\geq\frac{1}{2}$, then $\epsilon^{*}_{q}(E)=0$. Otherwise, $\epsilon^{*}_{q}(E)\leq-\sigma\Phi^{-1}(\mu)$, with equality when $E$ is a half space.

In particular, for any machine learning model for which the error rate in the distribution $q$ is at least $\mu$, the median distance to the nearest error is at most $-\sigma\Phi^{-1}(\mu)$. Because each coordinate of a multivariate normal is a univariate normal, $-\sigma\Phi^{-1}(\mu)$ is the distance to a half space for which the error rate is $\mu$. In other words, the right hand side of the inequality is the same expression that appears in Equation 1.

Comparing Neural Networks to the Isoperimetric Bound. We evaluated these quantities for several models on the CIFAR-10 and ImageNet test sets.

As in Section 4, we report an estimate of $\epsilon^{*}_{q}$. For each test image, we took 1,000 samples from the corresponding Gaussian and estimated $\epsilon^{*}_{q}$ using PGD with 200 steps on each sample and reported the median.

Evaluating Corruption Robustness

The previous two sections show a relationship between adversarial robustness and one type of corruption robustness. This suggests that methods designed to improve adversarial robustness ought to also improve corruption robustness, and vice versa. In this section we investigate this relationship.

We analyzed the performance of our models on the corruption robustness benchmark described in Hendrycks & Dietterich (2018). There are 15 different corruptions in this benchmark, each of which is tested at five different levels of severity. The results are summarized in Figure 6, where we have aggregated the corruption types based on whether the ordinarily trained model did better or worse than the augmented models. We found a significant difference in performance on this benchmark when the model is evaluated on the compressed images provided with the benchmark rather than applying the corruptions in memory. (In this section we report performance on corruptions applied in-memory.) Figure 5 shows an example for the Gaussian-5 corruption, where performance degraded from 57% accuracy (in memory) to 10% accuracy (compressed images). Detailed results on both versions of this benchmark are presented in Appendix B.

Gaussian data augmentation and adversarial training both improve the overall benchmarkIn reporting overall performance on this benchmark, we omit the Gaussian noise corruption., which requires averaging the performance across all corruptions, and the results were quite close. Adversarial training helped more with blurring corruptions and Gaussian data augmentation helped more with noise corruptions. Interestingly, both methods performed much worse than the clean model on the fog and contrast corruptions. For example, the adversarially trained model was 55% accurate on the most severe contrast corruption compared to 85% for the clean model. Note that Hendrycks & Dietterich (2018) also observed that adversarial training improves robustness on this benchmark on Tiny ImageNet.

The fact that adversarial training is so successful against the noise corruptions further supports the connection we have been describing. For other corruptions, the relationship is more complicated, and it would be interesting to explore this in future work.

We also evaluated these two augmentation methods on standard measures of $l_{p}$ robustness. We see a similar story there: while adversarial training performs better, Gaussian data augmentation does improve adversarial robustness as well. Gaussian data augmenation has been proposed as an adversarial defense in prior work (Zantedeschi et al., 2017). Here we evaluate this method not to propose it as a novel defense but to provide further evidence of the connection between adversarial and corruption robustness.

We also considered the MNIST adversarially trained model from Madry et al. (2017), and found it to be a special case where robustness to small perturbations was increased while generalization in noise was not improved (see Appendix D). This is because this model violates the linearity assumption discussed in Section 4.

Corruption Robustness as a Sanity Check for Defenses. We also analyzed the performance several previously published adversarial defense strategies in Gaussian noise. These methods have already been shown to result in vanishing gradients, which causes standard optimization procedures to fail to find errors, rather than actually improving adversarial robustness (Athalye et al., 2018). We find that these methods also show no improvement in Gaussian noise. The results are shown in Figure 7. Had these prior defenses performed an analysis like this, they would have been able to determine that their methods relied on vanishing gradients and fail to improve robustness.

Obtaining Zero Test Error in Noise is Nontrivial. It is important to note that applying Gaussian data augmentation does not reduce error rates in Gaussian noise to zero. For example, we performed Gaussian data augmentation on CIFAR-10 at $\sigma=.15$ and obtained 99.9% training accuracy but 77.5% test accuracy in the same noise distribution. (For comparison, the naturally trained obtains 95% clean test accuracy.) Previous work (Dodge & Karam, 2017) has also observed that obtaining perfect generalization in large Gaussian noise is nontrivial. This mirrors Schmidt et al. (2018), which found that adversarial robustness did not generalize to the test set, providing yet another similarity between adversarial and corruption robustness. This is perhaps not surprising given that error rates on the clean test set are also non-zero. Although the model is in some sense “superhuman” with respect to clean test accuracy, it still makes many mistakes on the clean test set that a human would never make. We collected some examples in Appendix 21. More detailed results on training and testing in noise can be found in Appendices C and H.

Conclusion

This paper investigates whether we should be surprised to find adversarial examples as close as we do, given the error rates we observe in corrupted image distributions. After running several experiments, we argue that the answer to this question is no. Specifically:

The nearby errors we can find show up at the same distance scales we would expect from a linear model with the same corruption robustness.

Concentration of measure shows that a non-zero error rate in Gaussian noise logically implies the existence of small adversarial perturbations of noisy images.

Finally, training procedures designed to improve adversarial robustness also improve many types of corruption robustness, and training on Gaussian noise moderately improves adversarial robustness.

In light of this, we believe it would be beneficial for the adversarial defense literature to start reporting generalization to distributional shift, such as the common corruption benchmark introduced in Hendrycks & Dietterich (2018), in addition to empirical estimates of adversarial robustness. There are several reasons for this recommendation.

First, a varied suite of corruptions can expose failure modes of a model that we might otherwise miss. For example, we found that adversarial training significantly degraded performance on the fog and contrast corruptions despite improving small perturbation robustness. In particular, performance on constrast-5 dropped to 55.3% accuracy vs 85.7% for the vanilla model (see Appendix B for more details).

Second, measuring corruption robustness is significantly easier than measuring adversarial robustness — computing adversarial robustness perfectly requires solving an NP-hard problem for every point in the test set (Katz et al., 2017). Since Szegedy et al. (2014), hundreds of adversarial defense papers have been published. To our knowledge, only one (Madry et al., 2017) has reported robustness numbers which were confirmed by a third party. We believe the difficulty of measuring robustness under the usual definition has contributed to this unproductive situation.

Third, all of the failed defense strategies we examined also failed to improve performance in Gaussian noise. For this reason, we should be highly skeptical of defense strategies that only claim improved $l_{p}$ robustness but are unable to demonstrate robustness to distributional shift.

Finally, if the goal is improving the security of our models in adversarial settings, errors on corrupted images already imply that our models are not secure. Until our models are perfectly robust in the presence of average-case corruptions, they will not be robust in worst-case settings.

The communities of researchers studying adversarial and corruption robustness seem to be attacking essentially the same problem in two different ways. We believe that the corruption robustness problem is also interesting independently of its connection to adversarial examples, and we hope that the results presented here will encourage more collaboration between these two communities.

References

Appendix A Training Details

Models trained on CIFAR-10. We trained the Wide-ResNet-28-10 model (Zagoruyko & Komodakis, 2016) using standard data augmentation of flips, horizontal shifts and crops in addition to Gaussian noise independently sampled for each image in every minibatch. The models were trained with the open-source code by Cubuk et al. (2018) for 200 epochs, using the same hyperparameters which we summarize here: a weight decay of 5e-4, learning rate of 0.1, batch size of 128. The learning rate was decayed by a factor of 0.2 at epochs 60, 120, 160.

Models trained on ImageNet. The Inception v3 model (Szegedy et al., 2016) was trained with a learning rate of 1.6, batch size of 4096, and weight decay of 8e-5. During training, Gaussian noise was independently sampled for each image in every minibatch. The models were trained for 130 epochs, where the learning rate was decayed by a factor of 0.975 every epoch. Learning rate was linearly increased from 0 to the value of 1.6 over the first 10 epochs.

Appendix B Full Corruption Robustness Results

In this section we examine the corruption robustness of both adversarially trained models and models trained with Gaussian data augmentation. Full results are shown in Tables 1, 2. We highlight several interesting findings from these experiments.

On CIFAR-10-C, Gaussian data augmentation outperforms adversarial training on the overall benchmark. However, adversarial training is better on all of the blurring corruptions.

The publicly released Imagenet-C dataset as .jpeg files is significantly harder than the same dataset when the corruptions are applied in memory. It appears that this is due to additional artifacts added to the image from the JPEG compression algorithm (see Figure 8). Future work should make care of this distinction when comparing the performance of their methods, in particular we note that the results in (Geirhos et al., 2018; Hendrycks & Dietterich, 2018) were both evaluated on the jpeg files.

Both adversarial training and Gaussian data augmenation significantly degrade performance on the severe fog and constrast corruptions (Tables 3, 4). This highlights the importance of evaluating on a broad suite of corruptions as simply evaluating on worst-case $l_{p}$ perturbations or random noise will not expose all failings of a model. This also highlights the need for developing methods that improve robustness to all corruptions. Towards this end the exciting new “Stylized ImageNet”(Geirhos et al., 2018) data augmentation process achieves moderate improvements on all corruptions, at least on the publicly released .jpeg files.

Appendix C Training and Testing on Gaussian Noise

In Section 6, we mentioned that it is not trivial to learn the distribution of noisy images simply by augmenting the training data distribution. In Tables 5 and 6 we present more information about the performance of the models we trained and tested on various scales of Gaussian noise.

Appendix D Results on MNIST

MNIST is a special case when it comes to the relationship between small adversarial perturbations and generalization in noise. Indeed prior has already observed that an MNIST model can trivially become robust to small $l_{\infty}$ perturbations by learning to threshold the input (Schmidt et al., 2018), and observed that the model from Madry et al. (2017) indeed seems to do this. When we investigated this model in different noise distributions we found it generalizes worse than a naturally trained model, results are shown in Table 7. Given that it is possible for a defense to overfit to a particular $l_{p}$ metric, future work would be strengthened by demonstrating improved generalization outside the natural data distribution.

Here we provide more detail for the noise distributions we used to evaluate the MNIST model. The stAdv attack defines a flow field over the pixels of the image and shifts the pixels according to this flow. The field is parameterized by a latent $Z$. When we measure accuracy against our randomized variant of this attack, we randomly sample $Z$ from a multivariate Gaussian distribution with standard deviation $\sigma$. To implement this attack we used the open sourced code from Xiao et al. (2018). PCA-100 noise first samples noise from a Gaussian distribution $\mathcal{N}(0,\sigma)$, and then projects this noise onto the first 100 PCA components of the data.

Appendix E The Gaussian Isoperimetric Inequality

Here we will discuss the Gaussian isoperimetric inequality more thoroughly than we did in the text. We will present some of the geometric intuition behind the theorem, and in the end we will show how the version quoted in the text follows from the form in which the inequality is usually stated.

The historically earliest version of the isoperimetric inequality, and probably the easiest to understand, is about areas of subsets of the plane and has nothing to do with Gaussians at all. It is concerned with the following problem: among all measurable subsets of the plane with area $A$, which ones have the smallest possible perimeter?The name “isoperimetric” comes from a different, but completely equivalent, way of stating the question: among all sets with the same fixed perimeter, which ones have the largest possible area? One picture to keep in mind is to imagine that you are required to fence off some region of the plane with area $A$ and you would like to use as little fence as possible. The isoperimetric inequality says that the sets which are most “efficient” in this sense are balls.

It is a good exercise to check that we have equality here when $E$ is a ball.

The Gaussian isoperimetric inequality says that the sets that do this are half spaces. (See Figure 9.) Just as we did in the plane, it is convenient to express this as a bound on the probability of landing in $E_{\epsilon}$ for an arbitrary measurable set $E$. This can be stated as follows:

the cdf of the one-variable standard normal distribution.

Since $\Phi^{-1}$ is a strictly increasing function, applying it to both sides preserves the direction of this inequality. But $\Phi^{-1}(\frac{1}{2})=0$, so we in fact have that $\epsilon^{*}_{q}(E)\leq-\alpha$, which is the statement we wanted.

Appendix F Visualizing the Optimal Curves

In this section we visualize the predicted relationship between worst-case $l_{2}$ perturbations and generalization in noise as described by Equation 1 in Section 4. This also visualizes the optimal bound according to the isoperimetric inequality, although the $l_{2}$ perturbations would be applied to the noisy images themselves rather then clean image. In Figure 10 we plot the optimal curves for various values of $\sigma$, visualize images sampled from $x+N(0,\sigma)$, and visualize images at various $l_{2}$ distance from the unperturbed clean image. Even for very large noise ($\sigma=.6$), test error needs to be less than $10^{-15}$ in order to have worst-case perturbations be larger than $5.0$. In order to visualize worst-case perturbations at varying $l_{2}$ distances, we visualize an image that minimizes similarity according to the SSIM metric (Wang & Bovik, 2009). These images are found by performing gradient descent to minimize the SSIM metric subject to the containt that $||x-x_{adv}||_{2}<\epsilon$. This illustrates that achieving significant $l_{2}$ adversarial robustness on Imagenet will likely require obtaining a model that is almost perfectly robust to large Gaussian noise (or a model which significantly violates the linearity assumption from Section 4). To achieve $l_{2}$ robustness on noisy images, a model must be nearly perfect in large Gaussian noise.

Appendix G Church Window Plots

In figures appearing below, starting at Figure 11, we include many more visualizations of the sorts of church window plots we discussed briefly in Section 4. We will show an ordinarily trained model’s predictions on several different slices through the same CIFAR test point which illustrate different aspects of the story told in this paper. These images are best viewed in color.

Appendix H The Distribution of Error Rates in Noise

Using some of the models that were trained on noise, we computed, for each image in the CIFAR test set, the probably that a random Gaussian perturbation will be misclassified. A histogram is shown in Figure 20. Note that, even though these models were trained on noise, there are still many errors around most images in the test set. While it would have been possible for the reduced performance in noise to be due to only a few test points, we see clearly that this is not the case.

Appendix I A Collection of Model Errors

Finally, in the figures starting at Figure 21 we first show a collection of iid test errors for the ResNet-50 model on the ImageNet validation set. We also visualize the severity of the different noise distributions considered in this work, along with model errors found by random sampling in these distributions.