Adversarial Patch

Introduction

Deep learning systems are broadly vulnerable to adversarial examples, carefully chosen inputs that cause the network to change output without a visible change to a human . These adversarial examples most commonly modify each pixel by only a small amount and can be found using a number of optimization strategies such as L-BFGS , Fast Gradient Sign Method (FGSM) , DeepFool , Projected Gradient Descent (PGD) , as well as the recently proposed Logit-space Projected Gradient Ascent (LS-PGA) for discretized inputs. Other attack methods seek to modify only a small number of pixels in the image (Jacobian-based saliency map ), or a small patch at a fixed location of the image .

Adversarial examples have been shown to generalize to the real world. Kurakin et al. demonstrated that when printed out, an adversarially constructed image will continue to be adversarial to classifiers even under different lighting and orientations. Athalye et al. recently demonstrated adversarial objects which can be 3d printed and misclassified by networks at different orientations and scales. Their adversarial objects are designed to be subtle perturbations of a normal object (e.g. a turtle that has been adversarially perturbed to be classified as a rifle). Another work showed that one can fool facial recognition software by constructing adversarial glasses. These glasses were targeted in that they could be constructed to impersonate any person, but were custom made for the attacker’s face, and were designed with a fixed orientation in mind. Even more recently, Evtimov et al. demonstrated various methods for constructing stop signs that are misclassified by models, either by printing out a large poster that looks like a stop sign, or by placing various stickers on a stop sign. In terms of defenses there has been substantial work on increasing the adversarial robustness of image models to small $L_{p}$ perturbations of the input .

As seen above, a majority of prior work has focused on attacking with and defending against either small or imperceptible changes to the input. In this work we explore what is possible if an attacker no longer restricts themselves to imperceptible changes. We construct an attack that does not attempt to subtly transform an existing item into another. Instead, this attack generates an image-independent patch that is extremely salient to a neural network. This patch can then be placed anywhere within the field of view of the classifier, and causes the classifier to output a targeted class. Because this patch is scene-independent, it allows attackers to create a physical-world attack without prior knowledge of the lighting conditions, camera angle, type of classifier being attacked, or even the other items within the scene.

This attack is significant because the attacker does not need to know what image they are attacking when constructing the attack. After generating an adversarial patch, the patch could be widely distributed across the Internet for other attackers to print out and use. Additionally, because the attack uses a large perturbation, the existing defense techniques which focus on defending against small perturbations may not be robust to larger perturbations such as these. Indeed recent work has demonstrated that state-of-the art adversarially trained models on MNIST are still vulnerable to larger perturbations than those used in training either by searching for a nearby adversarial example using a different metric for distance , or by applying large perturbations in the background .

Approach

To obtain the trained patch $\widehat{p}$ we use a variant of the Expectation over Transformation (EOT) framework of Athalye et al. . In particular, the patch is trained to optimize the objective function

where $X$ is a training set of images, $T$ is a distribution over transformations of the patch, and $L$ is a distribution over locations in the image. Note that this expectation is over images, which encourages the trained patch to work regardless of what is in the background. This departs from most prior work on adversarial perturbations in the fact that this perturbation is universal in that it works for any background. Universal perturbations were identified in , but these required changing every pixel in the image and results were not given in the physical world.

We also consider camouflaged patches which are forced to look like a given starting image. Here we simply add a constraint of the form $||p-p_{orig}||_{\infty}<\epsilon$ to the patch objective. This will force the final patch to be within $\epsilon$ in the $L_{\infty}$ norm of some starting patch $p_{orig}$.

We believe that this attack exploits the way image classification tasks are constructed. While images may contain several items, only one target label is considered true, and thus the network must learn to detect the most "salient" item in the frame. The adversarial patch exploits this feature by producing inputs much more salient than objects in the real world. Thus, when attacking object detection or image segmentation models, we expect a targeted toaster patch to be classified as a toaster, and not to affect other portions of the image.

Experimental Results

To test our attack, we compare the efficacy of two whitebox attacks, a blackbox attack, and a control patch. The white box ensemble attack jointly trains a single patch across five ImageNet models: inceptionv3, resnet50, xception, VGG16, and VGG19. We then evaluate the attack by averaging the win rate across all five models. The white box single model attack does the same but only trains and evaluates on a single model. The blackbox attack jointly trains a single patch across four of the ImageNet models, and then evaluates the blackbox attack on a fifth model, which we did not access during training. The control is a picture of a toaster.

During training and evaluation, the patches are rescaled and then digitally inserted on a random location on a random ImageNet image. Figure 2 shows the results.

Note that the patch size required to reliably fool the model in this universal setting (black box, on a targeted class, and over all images, locations and transformations) is significantly larger than those required to perform a non-targeted attack on a single image and a single location in the whitebox setting. For example, Su et al. recently demonstrated that modifying 1 pixel on a 32x32 pixel CIFAR-10 image (0.1% of the pixels in the image) suffices to fool the majority of images with a non-targeted, non-universal whitebox attack. However, our attacks are still far more effective than naively inserting an image with the target class, as is shown in Figure 2 by the relatively poor performance of inserting a real toaster into the scene.

Any of the attacks shown in Figure 2 can be camouflaged in order to reduce their saliency to a human observer. We create a disguised version of the patch by minimizing its L2 distance to a tie-dye pattern and applying a peace sign mask during training. The results from these experiments are found in Figure 3.

In our final experiment, we test the transferability of our attack into the physical world. We print the generated patch with a standard color printer, and put it a variety of real world situations. The results shown in Figure 1 demonstrate that the attack successfully fools the classifier, even when there are other objects within the scene. For a full video demonstration of the attack, see https://youtu.be/i1sp4X57TL4.

We also tested the black box + physical world effectiveness of the patch on the third party Demitasse applicationhttps://itunes.apple.com/us/app/demitasse-image-recognition-cam/id1138211169?mt=8 and found some transferability of the patch but only when the patch takes up a significant fraction of the image. We did not optimize the patch for print-ability as in , which perhaps explains why the patch is not as effective as in Figure 3, which tests black box for different models and not in the physical world. We invite curious readers to try the patch out for themselves by printing out this paper and using the patch in the Appendix.

Conclusion

We show that we can generate a universal, robust, targeted patch that fools classifiers regardless of the scale or location of the patch, and does not require knowledge of the other items in the scene that it is attacking. Our attack works in the real world, and can be disguised as an innocuous sticker. These results demonstrate an attack that could be created offline, and then broadly shared.

There has been substantial work on defending against small $L_{p}$ perturbations to natural images, at least partially motivated by security concerns . Part of the motivation of this work is that potential malicious attackers may not be concerned with generating small or imperceptible perturbations to a natural image, but may instead opt for larger more effective but noticeable perturbations to the input - especially if a model has been designed to resist small $L_{p}$ perturbations.

Many ML models operate without human validation of every input and thus malicious attackers may not be concerned with the imperceptibility of their attacks. Even if humans are able to notice these patches, they may not understand the intent of the patch and instead view it as a form of art. This work shows that focusing only on defending against small perturbations is insufficient, as large, local perturbations can also break classifiers.

Appendix

References