MagNet and "Efficient Defenses Against Adversarial Attacks" are Not Robust to Adversarial Examples

Abstract

MagNet and “Efficient Defenses…” were recently proposed as a defense to adversarial examples. We find that we can construct adversarial examples that defeat these defenses with only a slight increase in distortion.

Introduction

It is an open question how to train neural networks so they will be robust to adversarial examples . Recently, three defenses have been proposed to make neural networks robust to adversarial examples:

MagNet was proposed as an approach to make neural networks robust against adversarial examples through two complementary approaches: adversarial examples near the data manifold are reformed to lie on the data manifold that are classified correctly, whereas adversarial examples far away from the data manifold are detected and rejected before classification. MagNet does not argue robustness in the white-box setting; rather, the authors argue that MagNet is robust in the grey-box setting where the adversary is aware the defense is in place, knows the parameters of the base classifier, but not the parameters of the defense.

An efficient defense was proposed to make neural networks more robust against adversarial examples by performing Gaussian data augmentation during training, and using the BReLU activation function. The authors do not claim perfect security, but claim this makes attacks visually detectable.

Adversarial Perturbation Elimination GAN (APE-GAN) is similar to MagNet, only adversarial examples are projected onto the data manifold using a Generative Adversarial Network (GAN) . We did not set out to bypass this defense, but found it to be very similar to MagNet and so we analyze it too.

In this short paper, we demonstrate these three defenses are not effective on the MNIST and CIFAR-10 datasets. We show that we are able to bypass MagNet with greater than $99\%$ success, and the latter two with $100\%$, with only a slight increase in distortion.

We defeat MagNet by making use of the transferability property of adversarial examples: the adversary trains their own copy of the defense, constructs adversarial examples on their model, and supplies these adversarial examples to the defender. It turns out that these examples will also fool the defender’s model.

We defeat “Efficient Defenses Against Adversarial Attack” and APE-GAN by showing that existing attack can defeat them with $100\%$ success without modification. Adversarial examples are not more visually detectable than an undefended network.

Background

We assume familiarity with neural networks , adversarial examples , transferability , generating strong attacks against adversarial examples and MagNet . We briefly review the key details and notation.

Notation Let $F(x)=y$ be a neural network used for classification outputting a probability distribution. Call the second-to-last layer (the layer before the the softmax layer) $Z$, so that $F(x)=\text{softmax}(Z(x))$. Each output $y_{i}$ corresponds to the predicted probability that the object $x$ is labelled as class $i$. Let $C(x)=\arg\max_{i}\ F(x_{i})$ correspond to the classification of $x$ on $F$. In this paper we are concerned with neural networks used to classify images (on MNIST and CIFAR-10).

Adversarial examples are instances $x^{\prime}$ that are very close to a normal instance $x$ with respect to some distance metric ($L_{2}$ distance, in this paper), but where $C(x^{\prime})=t$ for any target $t$ chosen by the adversary.

We generate adversarial examples with Carlini and Wagner’s $L_{2}$ attack algorithm . Specifically, we solve

and the constant $c$ is chosen via binary search.

Perhaps the most surprising property of adversarial examples is transferability : given two different models that perform the same task, trained on different datasets, adversarial examples constructed to fool one one model often fool the other as well. This effect can be amplified by constructing the adversarial examples to fool multiple local models before applying it to the target model.

MagNet was recently proposed as a defense to adversarial examples under the grey box threat model: here, the adversary is aware that MagNet is being used to secure a given classifier, as well as the parameters of the classifier, but is not aware of the parameters of the defensive models. MagNet has two defense components:

The detector examines inputs and decides if they are adversarial or benign. There can be multiple detectors, in which case if any report “adversarial” the input is rejected.

The reformer takes inputs classified as benign by the detector(s) and reforms them to remove any remaining minor adversarial nature to the instances. The defender trains multiple reformers and selects one at random, to prevent the adversary from learning its parameters.

The authors argue these defenses complement each other: the detector should be able to detect any adversarial examples with large distortion and are far away from the true data manifold. The reformer takes instances which are close to the data manifold and maps them directly on to the manifold to remove any minor adversarial perturbations that may not have been detected.

The specific instantiation of the detector and reformer suggested by the authors are based on auto-encoders. Given an auto-encoder $AE$, to detect if an input is adversarial, compare the mean squared error $\|x-AE(x)\|_{2}$ with a threshold $\tau$ chosen to minimize the false positive rate. On CIFAR, the authors suggest one further detector: compute the Jensen-Shannon divergence $JSD(F(x)/T,F(AE(x))/T)$ with temperature $T$ (set to $10$ and $40$), and report “adversarial” if the difference is greater than a different threshold $\tau^{\prime}$, also chosen to minimize the false positive rate.

Similarly, to reform an adversarial example run it through the auto-encoder and let $x\leftarrow AE(x)$: since the auto-encoder is trained to reconstruct typical instances, this operation projects it back to the data manifold.

Efficient Defenses Against Adversarial Attacks works by making two modifications to standard neural networks. First, the authors propose use of the Bounded ReLU activation function, defined as $BReLU(x)=\text{min}(\text{max}(x,0),1)$ instead of standard ReLU which is unbounded above. Second, instead of training on the standard training data $\{(x_{i},y_{i})\}_{i=1}^{n}$ they train on $\{(x_{i}+N_{i},y_{i})\}_{i=1}^{n}$ where $N_{i}\sim\mathcal{N}(0,\sigma^{2})$ is chosen fresh for each training instance. On MNIST, $\sigma=0.3$; for CIFAR, $\sigma=0.05$. The authors claim that despite training on noise, it is successful on .

APE-GAN works by constructing a pre-processing network $G(\cdot)$ trained to project both normal instances and adversarial examples back to the data manifold as done in MagNet. The network $G$ is trained with a GAN instead of an auto-encoder. Note that unlike a standard GAN which takes as input a noise vector and must produce an output image, the generator in APE-GAN takes in an adversarial example and must make it appear non-adversarial. During training, the authors train on adversarial examples generated with the Fast Gradient Sign algorithm ; despite this, the authors claim robustness on a wide range of attacks (including ).

Defense Models. We take the MagNet implementation from the authors’ open-source code https://github.com/Trevillie/MagNet and train our own models. Since the provided code does not include an implementation of the CIFAR defense and classifier, we implement it as described in the paper.

We were unable to obtain source code for “Efficient Defenses…”. We therefore re-implement the proposed defense based on the description in the paper. We take the APE-GAN implementation from the authors open-source code https://github.com/shenqixiaojiang/APE-GAN.

We use the C&W attack given by the authors. We also release our code https://github.com/carlini/MagNet https://github.com/carlini/breaking_efficient_defenses https://github.com/carlini/APE-GAN.

Breaking MagNet

We demonstrate MagNet is not robust by generating transferable adversarial examples with Carlini and Wagner’s $L_{2}$ attack. To do this, we (as the adversary) train $32$ local auto-encoders to use as detectors and reformers (different than the 16 the defender will use). To generate adversarial examples, we modify the optimization formulation to the following, where reformers are denoted $R_{j}$ and the detectors $D_{j}$:

Here, $c$ and $d$ are chosen via binary search simultaneously. The threshold $\tau_{j}$ is chosen such that $1\%$ of validation images are detected as adversarial. (The true defender sets their remote $\tau$ to reject only $0.1\%$ of instances : ours is more strict to help the attack construct adversarial examples that fool the remote detectors more often.) We set the confidence $\kappa=1$ so that the adversarial examples are more likely to transfer.

We attack by performing $10000$ iterations of gradient descent with a learning rate of $10^{-2}$. We did not perform hyperparameter search (e.g., picking $32$ auto-encoders, $\kappa=1$, $\tau_{j}=0.01$); improved search over these parameters would yield lower distortion adversarial examples.

Figures 1 and 2 contain images of targeted adversarial examples on on the secured network, and Table 1 the mean distortion required across the first 1000 instances of the test set with targets chosen uniformly at random among the incorrect classes.

Breaking “Efficient Defenses…”

We demonstrate this defense is not robust by generating adversarial examples with Carlini and Wagner’s $L_{2}$ attack. We do nothing more than apply the attack to the defended network.

Figure 3 contains images of adversarial examples on the secured network, and Table 2 the mean distortion required across the first 1000 instances of the test set with targets chosen at random among the incorrect classes.

On MNIST, the full defense increases mean distance to the nearest adversarial example by $30\%$, and on CIFAR by $20\%$. This is in contrast with other forms of retraining, such as adversarial retraining , which increase distortion by a significantly larger amount. Interestingly, we find that BReLU provides some increase in distortion when trained without Gaussian augmentation, but when trained with it, does not help.

Breaking APE-GAN

We demonstrate APE-GAN is not robust by generating adversarial examples with Carlini and Wagner’s $L_{2}$ attack. We do nothing more than apply the attack to defended network. That is, we change the loss function to account for the fact that the manifold-projection is done before classification. Specifically, we let

and solve the same minimization formulation.

Figure 4 contains images of adversarial examples on APE-GAN, and Table 3 the mean distortion required across the first 1000 instances of the test set with targets chosen at random among the incorrect classes.

Investigating APE-GAN’s Failure. Why are we able to fool APE-GAN? We compare (a) the mean distance between the original inputs and the adversarial examples, and (b) the mean distance between the original inputs and the recovered adversarial examples. We find that the recovered adversarial examples are less similar to the original than the adversarial examples. Specifically, the mean distortion between the adversarial examples and the original instances is $4.3$, whereas the mean distortion between the recovered instances and original instances is $5.8$.

This indicates that what our adversarial examples have done is fool the generator $G$ into giving reconstructions that are even less similar from the original than the adversarial example. This effect can be observed in Figure 4: faint lines introduced become more pronounced after reconstruction.

Conclusion

As this short paper demonstrates, MagNet is not robust to transferable adversarial examples, and combining Gaussian data augmentation and BReLU activations does not significantly increase the robustness of a neural network against strong iterative attacks. Surprisingly, we found that while all three defenses take different approaches to increasing the robustness against adversarial examples, they all give approximately the same increase in robustness ($\sim 30\%$).

We recommend that researchers who propose defenses attempt adaptive white-box attacks against their schemes before claiming robustness. Or, if arguing security in the grey-box threat model, we recommend researchers generate adversarial examples targeting the specific defense by using a copy of that defense as the source model. Just because the adversary is not aware of the exact parameters, does not mean the best that can be done is to transfer from an unsecured model: as we have shown here, transfering from a local copy of the model improves the attack success rate.

References