Pyramid Adversarial Training Improves ViT Performance

Introduction

One fascinating aspect of human intelligence is the ability to generalize from limited experiences to new environments . While deep learning has made remarkable progress in emulating or “surpassing” humans on classification tasks, deep models have difficulty generalizing to out-of-distribution data . Convolutional neural networks (CNNs) may fail to classify images with challenging contexts , unusual colors and textures and common or adversarial corruptions . To reliably deploy neural networks on diverse tasks in the real world, we must improve their robustness to out-of-distribution data.

One major line of research focuses on network design. Recently the Vision Transformer (ViT) and its variants have advanced the state of the art on a variety of computer vision tasks. In particular, ViT models are more robust than comparable CNN architectures . With a weak inductive bias and powerful model capacity, ViT relies heavily on strong data augmentation and regularization to achieve better generalization . To further push this envelope, we explore using adversarial training as a powerful regularizer to improve the performance of ViT models.

Prior work suggests that there exists a performance trade-off between in-distribution generalization and robustness to adversarial examples. Similar trade-offs have been observed between in-distribution and out-of-distribution generalization . These trade-offs have primarily been observed in the context of CNNs . However, recent work has demonstrated the trade-off can be broken. AdvProp achieves this via adversarial training (abbreviated AT) with a “split” variant of Batch Normalization for EfficientNet . In our work, we demonstrate that the trade-off can be broken for the newly introduced vision transformer architecture .

We introduce pyramid adversarial training (abbreviated as PyramidAT) that trains the model with input images altered at multiple spatial scales, as illustrated in Fig. 1; the pyramid attack is designed to make large edits to the image in a structured, controlled manner (similar to augmenting brightness) and small edits to the image in a flexible manner (similar to pixel adversaries). Using these structured, multi-scale adversarial perturbations leads to significant performance gains compared to both baseline and standard pixel-wise adversarial perturbations. Interestingly, we see these gains for both clean (in-distribution) and robust (out-of-distribution) accuracy. We further enhance the pyramid attack with additional regularization techniques: “matched” Dropout and stochastic depth. Matched Dropout uses the same Dropout configuration for both the regular and adversarial samples in a mini-batch (hence the word matched). Stochastic depth randomly drops layers in the network and provides a further boost when matched and paired with matched Dropout and multi-scale perturbations.

Our ablation studies confirm the importance of matched Dropout when used in conjunction with the pyramid adversarial training. They also reveal a complicated interplay between adversarial training, the attack being used, and network capacity. We additionally show that our approach is applicable to datasets of various scales (ImageNet-1K and ImageNet-21K) and for a variety of network architectures such as ViT , Discrete ViT , ResNet, and MLP-Mixer . Our contributions are summarized below:

To our knowledge, we appear to be the first to demonstrate that adversarial training improves ViT model performance on both ImageNet and out-of-distribution ImageNet robustness datasets .

We demonstrate the importance of matched Dropout and stochastic depth for the adversarial training of ViT.

We design pyramid adversarial training to generate multi-scale, structured adversarial perturbations, which achieve significant performance gains over non-adversarial baseline and adversarial training with pixel perturbations.

We establish a new state of the art for ImageNet-C, ImageNet-R, and ImageNet-Sketch without extra data, using only our pyramid adversarial training and the standard ViT-B/16 backbone. We further improve our results by incorporating extra ImageNet-21K data.

We perform numerous ablations which highlight several elements critical to the performance gains.

Related Work

There exists a large body of work on measuring and improving the robustness of deep learning models, in the context of adversarial examples and generalization to non-adversarial but shifted distributions. We define out-of-distribution accuracy/robustness to explicitly refer to performance of a model on non-adversarial distribution shifts, and adversarial accuracy/robustness to refer to the special case of robustness on adversarial examples. When the evaluation is performed on a dataset drawn from the same distribution, we call this clean accuracy.

The discovery of adversarial examples has stimulated a large body of literature on adversarial attacks and defenses . Of the many proposed defenses, adversarial training has emerged as a simple, effective, albeit expensive approach to make networks adversarially robust. Although some work has suggested a tradeoff between adversarial and out-of-distribution robustness or clean accuracy, other analysis has suggested simultaneous improvement is achievable. In , the authors note improved accuracy on both clean and adversarially perturbed data, though only on smaller datasets such as CIFAR-10 and SVHN , and only through the use of additional data extending the problem to the semi-supervised setting. Similarly in NLP, adversarial training leads to improvement of clean accuracy for machine translation .

Most closely related to our work is the technique of , which demonstrates the potential of adversarial training to improve both clean accuracy and out-of-distribution robustness. They focus primarily on CNNs and propose split batch norms to separately capture the statistics of clean and adversarially perturbed samples in a mini-batch. At inference time, the batch norms associated with adversarially perturbed samples are discarded, and all data (presumed clean or out-of-distribution) flows through the batch norms associated with clean samples. Their results are demonstrated on EfficientNet and ResNet architectures. However, their approach is not directly applicable to ViT where batch norms do not exist. In our work, we propose novel approaches, and find that properly constructed adversarial training helps clean accuracy and out-of-distribution robustness for ViT models.

Robustness of ViT

ViT models have been found to be more adversarially robust than CNNs , and more importantly, generalize better than CNNs with similar model capacity on ImageNet out-of-distribution robustness benchmarks . While existing works focus on analyzing the cause of ViT’s superior generalizability, this work aims at further improving the strong out-of-distribution robustness of the ViT model. A promising approach to this end is data augmentation; as shown recently , ViT benefits from strong data augmentation. However, the data augmentation techniques used in ViT are optimized for clean accuracy on ImageNet, and knowledge about robustness is still limited. Different from prior works, this paper focuses on improving both the clean accuracy and robustness for ViT. We show that our technique can effectively complement strong ViT augmentation as in . We additionally verify that our proposed augmentation can benefit three other architectures: ResNet, MLP-Mixer , and Discrete ViT .

Data augmentation

Existing data augmentation techniques, although mainly developed for CNNs, transfer reasonably well to ViT models . Other work has studied larger structured attacks . Our work is different from prior work in that we utilize adversarial training to augment ViT and tailor our design to the ViT architecture. To our knowledge, we appear to be the first to demonstrate that adversarial training substantially improves ViT performance in both clean and out-of-distribution accuracies.

Approach

We work in the supervised learning setting where we are given a training dataset $\mathcal{D}$ consisting of clean images, represented as $x$ and their labels $y$. The loss function considered is a cross-entropy loss $L(\theta,x,y)$, where $\theta$ are the parameters of the ViT model, with weight regularization $f$. The baseline models minimize the following loss:

The overall training objective for adversarial training is given as follows:

To remedy this, we can train on both clean and adversarial images using the following objective:

This objective uses adversarial images as a form of regularization or data augmentation, to force the network towards certain representations that perform well on out-of-distribution data. These networks exhibit some degree of robustness but still have good clean accuracy. More recently, proposes a split batch norm that leads to performance gains for CNNs on both clean and robust ImageNet test datasets. Note that they do not concern themselves with adversarial robustness, and neither do we in this paper.

2 Pyramid Adversarial Training

Pixel-wise adversarial images are defined as $x^{a}=x+\delta$ where the perturbation distribution $P$ consists of a clipping function $C_{\mathcal{B}_{\epsilon}}$ that clips the perturbation at each pixel location to be inside the specified ball ($\mathcal{B}_{\epsilon}$) for a specified $l_{p}$-norm , with maximal radius $\epsilon$ for the perturbation.

For pixel-wise adversarial images, increasing the value of $\epsilon$ or the number of steps of the inner loop in Eqn. 3 eventually causes a drop in clean accuracy (Fig 2). Conceptually, pixel attacks are very flexible and, if given the ability to make large changes (in $L_{2}$ distance), can destroy the object being classified; training with these images may harm the network. In contrast, augmentations, like brightness, can lead to large $L_{2}$ distances but will preserve the object because they are structured. Our main motivation is to design an attack which has the best of both worlds: a low-magnitude flexible component and a high-magnitude structured component; this attack can lead to large image differences while still preserving the class identity.

Approach

We propose pyramid adversarial training (PyramidAT) which generates adversarial examples by perturbing the input image at multiple scales. This attack is more flexible and yet also more structured, since it consists of multiple scales, but the perturbations are constrained at each scale.

where $C_{\mathcal{B}_{1}}$ is the clipping function that keeps the image within the normal range, $S$ is the set of scales, $m_{s}$ is the multiplicative constant for scale $s$, $\delta_{s}$ is the learned perturbation (with the same shape as $x$). For scale $s$, the weights in $\delta_{s}$ are shared for pixels in square regions of size $s\times s$ with top left corner $[s\cdot i,s\cdot j]$ for all discrete $i\in[0,\textrm{width}/s]$ and $j\in[0,\textrm{height}/s]$, as shown in Fig. 1. Note that, similar to pixel AT, each channel of the image is perturbed independently. More details of the parameter settings are given in Section 4 and pseudocode is included in the supplementals.

Setting up the attack

For both the pixel and pyramid attacks, we use Projected Gradient Descent (PGD) on a random label using multiple steps . With regards to the loss, we observe that for ViT, maximizing the negative loss of the true label leads to aggressive label leaking , i.e., the network learns to predict the adversarial attack and performs better on the perturbed image. To avoid this, we pick a random label and then minimize the softmax cross-entropy loss towards that random label as described in .

3 “Matched” Dropout and Stochastic Depth

Standard training for ViT models uses both Dropout and stochastic depth as regularizers. During adversarial training, we have both the clean samples and adversarial samples in a mini-batch. This poses a question about Dropout treatment during adversarial training (either pixel or pyramid). In the adversarial training literature, the usual strategy is to run the adversarial attack (to generate adversarial samples) without using Dropout or stochastic depth. However, this leads to a training mismatch between the clean and adversarial training paths when both are used in the loss (Eqn. 3), with the clean samples trained with Dropout and the adversarial samples without Dropout. For each training instance in the mini-batch, the clean branch will only update subsets of the network while the adversarial branch updates the entire network. The adversarial branch updates are therefore more closely aligned with the model performance during evaluation, thereby leading to an improvement of adversarial accuracy at the expense of clean accuracy. This objective function is given below:

where, with a slight abuse of notation, $\mathcal{M}(\theta)$ denotes a network with a random Dropout mask and a stochastic depth configuration. To address the issue above, we propose adversarial training of ViT with “matched” Dropout, i.e., using the same Dropout configuration for both clean and adversarial training branches (as well as for the generation of adversarial samples). We show through ablation in Section 4 that using the same Dropout configuration leads to the best overall performance for both the clean and robust datasets.

Experiments

In this section, we compare the effectiveness of our proposed PyramidAT to non-AT models, and PixelAT models.

We focus primarily on ViT-B/16 , the baseline ViT with a patch size of 16. We also demonstrate our technique on other network architectures, such as ViT-Ti/16, ResNet , MLP-Mixer , and the recent Discrete ViT .

Datasets

We train models on both ImageNet-1K and ImageNet-21K . We evaluate in-distribution performance on 2 additional variants: ImageNet-ReaL which relabels the validation set of the original ImageNet in order to correct labeling errors; and ImageNet-V2 which collects another version of ImageNet’s evaluation set. We evaluate out-of-distribution robustness on 6 datasets: ImageNet-A which places the ImageNet objects in unusual contexts or orientations; ImageNet-C which applies a series of corruptions (e.g. motion blur, snow, JPEG, etc.); ImageNet-Rendition which contains abstract or rendered versions of the object; ObjectNet which consists of a large real-world set from a large number of different backgrounds, rotations, and imaging view points; ImageNet-Sketch which contains artistic sketches of the objects; and Stylized ImageNet which processes the ImageNet images with style transfer from an unrelated source image. For brevity, we may abbreviate ImageNet as IM. For all datasets except IM-C, we report top-1 accuracy (where higher is better). For IM-C, we report the standard “Mean corruption error” (mCE) (where lower is better).

Implementation details

Following , we use a batch size of 4096, a cosine decay learning rate schedule (0.001 magnitude) with linear warmup for the first 10k steps, , and the AdamW optimizer in all our experiments. Augmentations and regularizations include RandAug with the default setting of $(2,15)$, Dropout at probability $0.1$, and stochastic depth at probability $0.1$. We train with Scenic, a Jax library, on DragonFish TPUs.

To generate the pixel adversarial attack, we follow . We use a learning rate of $1/255$, $\epsilon=4/255$, and attack for $5$ steps with SGD. We use PGD to generate the adversarial perturbations. We also experiment with using more recent optimizers to construct the attacks (results are provided in the supplementals). For pyramid attacks, we find using stronger perturbations at coarser scales is more effective than equal perturbation strengths across all scales. By default, we use a 3-level pyramid and use perturbation scale factors $S=$ (a scale of $1$ means that each pixel has one learned parameter, a scale of $16$ means that each $$patch has one learned parameter) with multiplicative terms of$m_{s}=$(see Eqn. 4). We use a clipping value of$\epsilon_{s}=6/255$ for all levels of the pyramid.

2 Experimental Results on ViT-B/16

Table 1 shows results on ImageNet-1K and robustness datasets for ViT-B/16 models without adversarial training, with pixel adversarial attacks and with pyramid adversarial attacks. Both adversarial training attacks use matched Dropout and stochastic depth, and optimize the random target loss. The pyramid attack provides consistent improvements, on both clean and robustness accuracies, over the baseline and pixel adversaries. In Table 1, we also compare against CutMix augmentation. We find that CutMix improves performance over the ViT baseline but cannot improve performance when combined with RandAug. Similar to , we find that CutOut does not boost performance on ImageNet for our models.

The robustness gains of our technique are preserved through fine-tuning on clean data at higher resolution (384x384), as shown in the second set of rows of Table 1. Further, adversarial perturbations are consistently better than random perturbations on either pre-training or fine-tuning, for both pixel and pyramid models.

State of the art

Our model trained on IM-1K sets a new overall state of the art for IM-C , IM-Rendition , and IM-Sketch , as shown in Tables 2, 3, and 4. While we compare all our models under a unified framework in our main experiments, we select the optimal pre-processing, fine-tuning, and Dropout setting for the given dataset when comparing against the state-of-the-art. We also compare against on IM-21K and find that our results still compare favorably.

ImageNet-21K

In table 5, we show that our technique maintains gains over the baseline Reg-ViT and pixel-wise attack on the larger dataset IM-21K. Following , we pre-train on IM-21K and fine-tune on IM-1K at a higher resolution (in our case, 512x512). We apply adversarial training during the pre-training stage only.

3 Ablations

We explore the effects of adversarial training on three other backbones: ResNet , Discrete ViT , and MLP-Mixer . As shown in Table 6, we find slightly different results. For ResNet, we use the split BN from and show improved performance from PyramidAT. Other ResNet variants (-101, -200) show the same trend and are included in the supplementals. For Discrete ViT, we show that AT with both pixel and pyramid leads to general improvements, though the gain from pyramid over pixel is less consistent than with ViT-B/16. For MLP-Mixer, we observe decreases in clean accuracy but gains in the robustness datasets for PixelAT, similar to what has traditionally been observed from AT on ConvNets. However, with PyramidAT, we observe improvements for all evaluation datasets.

Matched Dropout and Stochastic Depth

We study the impact of handling Dropout and stochastic depth for the clean and adversarial update in Table 7. We find that applying matched Dropout for the clean and adversarial update is crucial for achieving simultaneous gains in clean and robust performance. When we eliminate Dropout in the adversarial update (“without Dropout” rows in 7), we observe significant decreases in performance on clean, IM-ReaL, and IM-A; and increases in performance on IM-Sketch and IM-Stylized. This result appears similar to the usual trade-off suggested in . By contrast, carefully handling Dropout and stochastic depth can lead to performance gains in both clean and out-of-distribution datasets.

Pyramid attack setup

In Table 8, we ablate the pyramid attacks. Pyramid attacks are consistently better than pixel or patch attacks, while the 3-level pyramid attack tends to have the best overall performance. Note that a 2-level pyramid attack consists of both the pixel and patch attacks. Please refer to the supplementals for comparison on all the metrics.

Network capacity and random augmentation

We test the effect of network capacity on adversarial training and, consistent with existing literature , find that large capacity is critical to effectively utilizing PixelAT. Specifically, low-capacity networks, like ViT-Ti/16, which already struggle to represent the dataset, can be made worse through PixelAT. Table 9 shows that PixelAT hurts in-distribution performance of the RandAugment 0.4 model but improves out-of-distribution performance. Unlike prior work, we note that this effect depends on both the network capacity and the random augmentation applied to the dataset.

Table 9 shows that a low-capacity network can benefit from adversarial training if the random augmentation is of a small magnitude. Standard training with RandAugment magnitude of 0.4 (abbreviated as RAm=0.4) provides a better clean accuracy than standard training with RAm=0.1; however, PixelAT with the weaker augmentation, RAm=0.1, performs better than either standard training or PixelAT at RAm=0.4. This suggests that the augmentation should be tuned for adversarial training and not fixed based on standard training.

Table 9 also shows that PyramidAT acts differently than PixelAT and can provide in-distribution gains despite being used with stronger augmentation. For these models, we find that for the robustness datasets, PixelAT tends to marginally outperform PyramidAT.

Attack strength

Pixel attacks are much smaller in $L_{2}$ norm than pyramid attacks. We check that simply scaling up the PixelAT cannot achieve the same performance as PyramidAT in Figure 2. For both ImageNet and ImageNet-C, we show the effect of raising the pixel and pyramid attack strength. While the best PyramidAT performance is achieved at high $L_{2}$ perturbation norm, the PixelAT performance degrades beyond a certain norm.

4 Analysis and Discussions

Following , we visualize the learned pixel embeddings (filters) of models trained normally, with pixel adversaries, and with pyramid adversaries in Fig. 3. We observe that the PixelAT model tends to tightly “snap” its attention to the perceived object, disregarding the majority of the background. While this may appear to be a desirable behavior, this kind of focusing can be suboptimal for the in-distribution datasets (where the background can provide valuable context) and prone to errors for out-of-distribution datasets. Specifically, the PixelAT model may under-estimate the size or shape of the object and focus on a part of the object and not the whole. This can be problematic for fine-grained classification when the difference between two classes comes down to something as small as the stripes or subtle shape cues (tiger shark vs great white); or texture and context (green mamba vs vine snake). Figure 4 shows the heat maps for the average attention on images in the evaluation set of ImageNet-A. We observe that PyramidAT tends to more evenly spread its attention across the entire image than both the baseline and PixelAT.

Figure 5 demonstrates the difference in representation between the baseline, PixelAT, and PyramidAT models. The pixel attacks on the baseline and PixelAT have a small amount of structure but appear to consist of mostly texture-level noise. In contrast, the pixel level of the PyramidAT shows structures from the original image: the legs and back of the dog. This suggests that the representation for the PyramidAT model focuses on shape and is less sensitive to texture than the baseline model.

Analysis of attacks

Inspired by , we analyze the pyramid adversarial training from a frequency perspective. For this analysis, all visualizations and graphs are averaged over the entire ImageNet validation set. Figure 6 shows a Fourier heat map of random and adversarial versions of the pixel and pyramid attacks. While random pixel noise is evenly concentrated over all frequencies, adversarial pixel attack tends to concentrate in the lower frequencies. Random pyramid shows a bias towards low frequency as well, a trend which is amplified in the adversarial pyramid. To further explore this, we replicate an analysis from , where low-pass- and high-pass-filtered random noise is added to test data to perturb a classifier. Figure 7 gives the result for our baseline, pixel, and pyramid adversarially trained models. While pixel and pyramid models are generally more robust than the baseline, the pyramid model is more robust than the pixel model to low-frequency perturbations.

Limitations

The cost of our technique is increased training time. A $k$-step PGD attack requires $k$ forward and backward passes for each step of training. Note that this limitation holds for any adversarial training and the inference time is the same. Without adversarial training, more training time does not improve the baseline ViT-B/16.

Conclusion

We have introduced pyramid adversarial training, a simple and effective data augmentation technique that substantially improves the performance of ViT and MLP-Mixer architectures on in-distribution and a number of out-of-distribution ImageNet datasets.

References

Appendix A Pyramid Attack Details

In this section, we provide a conceptual description and pseudocode for the pyramid attack.

Scale $s$ determines the size of the patch that an individual perturbation parameter will be applied to; e.g. for $s\!=\!16$, we learn and add a single adv parameter to each non-overlapping patch of size 16x16. The application is equivalent to a nearest neighbor resize on the 14x14 adv tensor to the image size of 224x224 and then addition. The scales $s$ and multipliers $m_{s}$ used by PyramidAT are hyperparameters.

Code

We provide a minimal implementation of our technique in Fig. 8.

Appendix B Discussing Backbones

Table 10 shows results for multiple variants of ResNet: ResNet-50, ResNet-101, and ResNet-200. As the capacity of the network increases, we observe larger gains from both PixelAT and PyramidAT. PyramidAT performs the best on all evaluation sets.

For these runs, we follow the training protocol and network details set up in . We use the proposed split BN and standard ResNet training: 90 epochs, cosine learning rate at 0.1 with linear warmup for 7 epochs, minimal augmentations (left right flip and Inception crop). For network optimization, we use SGD with momentum and for the adversarial steps, we use SGD.

B.2 ViT Tiny/16

ViT Ti/16 has the same overall structure and design as ViT B/16 (the primary model used in our main paper) but is significantly smaller, at 5.8 million parameters (as opposed to the 86 million parameters of B/16). More specifically, Ti/16 has a width of 192 (instead of 768), MLP size of 768 (instead of 3072), and 3 heads (instead of 12). In total, this decrease in parameters and model size leads to a substantial decrease in capacity. We experiment with ViT Ti/16 primarily in order to understand the impact of this decreased capacity on our adversarial training methods.

We start with an exploration of the impact of the random augmentation’s strength on the overall performance of the model. Table 11 shows the performance of Ti/16 models with different RandAugment parameters (the two parameters are in order the number of transforms applied and the magnitude of the transforms); this table suggests that the network’s lower capacity benefits from weaker random augmentation. Specifically, the best RandAugment parameters for the majority of the evaluation datasets is (1,0.8), which is considerably lower than the RandAugment parameters tuned for B/16 (2, 15).

In Table 12 and 13, we pick several of the better performing RandAugment parameters and then show results from adversarial training with steps of 1 and 3, respectively.

Table 12 shows that the performance of adversarial training depends heavily on both the random augmentation and the type of attack. Note, RAm refers to the RandAugment mangitude parameter. As shown by RAm=0.1, pixel attacks can improve performance for in-distribution evaluation datasets when the random augmentation strength is low. However, at higher random augmentation, RAm=0.4 and RAm=0.8, PixelAT leads to the commonly observed trade-off between clean performance and adversarial robustness. In contrast, pyramid tends to improve performance across the board regardless of the starting augmentation (for all RAm of 0.1, 0.4, and 0.8). Interestingly, PixelAT exhibits better robustness properties (out-of-distribution performance) than PyramidAT for Ti/16. We hypothesize that the limited capacity can be “spent” on either in-distribution or out-of-distribution representations and that pyramid tends to bias the network towards in-distribution as opposed to pixel which has a bias towards out-of-distribution.

Table 13 shows that the strength of the adversarial attack also matters substantially to the overall performance of the model. Both attacks, pixel and pyramid, with 3 steps tend to degrade the model’s performance on in-distribution evaluation datasets. Adversarial training still provides some benefits for out-of-distribution, with pyramid performing the best in terms of robustness. We hypothesize that pyramid outperforms pixel in the steps=3 runs because pyramid is a weaker out-of-distribution augmentation and pixel at 3 steps has over-regularized the network, leading to decreased performance. Note that pyramid at steps=3 produces the best out-of-distribution performance out of any Ti/16 runs including strong random augmentation and PixelAT at steps=1.

B.3 MLP-Mixer

As shown in the main paper, we observe gains across the board for MLP-Mixer with PyramidAT. Here, we show that the gain is robust to a change in the LR schedule and that the gain is, again, affected by the starting augmentation.

Table 14 shows baseline and adversarially trained models for two different training schedules of MLP-Mixer, one with the default LR schedule of 10k warm-up steps and then linear decay to an end learning rate (LR) of $1e-5$ and another with a more aggressive end learning rate of $1e-7$. We show that this change in LR schedule does not affect the gains from adversarial training.

Table 15 shows that, similar to ViT Ti/16, the gains are improved when the random augmentation is weakened. However, in this case, the gain is not enough to overcome the drop in performance from using the weaker augmentation.

Appendix C Additional Ablations

One of the key findings of this paper is the importance of “matched” Dropout and stochastic depth. Here we describe numerous ablations on these Dropout terms and list several detailed findings including:

Matching the Dropout and stochastic depth matters significantly for balanced clean performance and robustness.

Running without Dropout in the adversarial training branch can improve robustness even more.

Dropout matters more than Stochastic Depth

Note that in the tables below, we use the term “dropparams” to refer to a tuple of the Dropout probability and stochastic depth probability. Clean dropparams (abbreviated as c_dp) refer to the dropparams used for the clean training branch; adversarial dropparams (abbreviated as a_dp) refer to the dropparams used for the adversarial training branch; and matched dropparams (abbreviated as m_dp) refer to dropparams used for both clean and adversarial branches. So $\textrm{c\_dp}=(10,0)$ means that the clean training branch had a 10% probability of Dropout but a 0% probability of stochastic depth.

Table 16 explores different possible values for adversarial dropparams. In general, lower values of Dropout and stochastic depth in the adversarial branch improve out-of-distribution performance while hurting in-distribution performance; however, the opposite is not true: higher levels of Dropout and stochastic depth in the adversarial branch do not improve in-distribution performance. In-distribution performance seems to peak when the params for the adversarial and clean branches match.

Table 17 explores different possible values for matched dropparams. In general, the dropparams determined by RegViT seem to be roughly optimal for both the baselines and the adversarially trained models, with some variation for some datasets.

Table 18 explores if one of these parameters is more important than the others. To do so, we set clean dropparams to $(10,10)$ for the entire table (besides the included baselines) and only vary the adversarial dropparams. For both PixelAT and PyramidAT, the Dropout parameter seems to be more important for clean, in-distribution performance. Without Dropout, the top-1 of ImageNet drops $0.41$ for PixelAT and $0.92$ for PyramidAT. However, no Dropout does give a substantial boost to out-of-distribution performance, with Rendition gains of $11.59$ for PixelAT and $15.68$ for PyramidAT and Sketch gains of $7.49$ for PixelAT and $11.96$ for PyramidAT. Without stochastic depth, the adversarially trained models seem to perform roughly as well as with stochastic depth, exhibitly marginally more clean accuracy for PyramidAT than the model with both Dropout and stochastic depth. Our main takeaway is that Dropout seems to be the primary determinant in whether the gains are balanced between in-distribution and out-of-distribution or primarily focused on out-of-distribution. In fact, no Dropout PyramidAT performs so well on out-of-distribution that it sets new state-of-the-art numbers for Rendition and Sketch.

Table 19 explores parameter settings where the Dropout and stochastic depth are not equal. In general, there does not seem to be a consistent trend or recognizable pattern for the overall performance, though some patterns exist for specific attacks and evaluation datasets. For example, increasing stochastic depth probability for pixel attacks tend to improve Real, ImageNet-C, and ObjectNet performance.

Table 20 explores the effects of adversarial training with different dropparams without Dropout or stochastic depth in the main branch. In general, the lack of Dropout and stochastic depth in the clean branch has a substantial negative effect on the performance of the model and all of the resulting models under-perform their counterparts with non-zero clean dropparams. In this setting, adversarial training does provide substantial improvements for both in-distribution ($+5.18$ for clean using PyramidAT) and out-of-distribution performances ($+12.29$ for Rendition using PyramidAT and $+11.68$ for Sketch using PyramidAT), but not enough to offset the poor starting performance of the baseline model.

C.2 Pyramid Structure

In the main paper, Table 8 presented an abridged version (with only a subset of the evaluation datasets) of an ablation on the structure of the pyramid used in the pyramid adversarial training. We present the full version (complete with all the evaluation datasets) of this ablation in Table 21. This table remains consistent with the description and explanation in the main table: adding more layers to the pyramid tends to improve performance. In fact, Table 21 shows the full extent of the trade-off between the 3rd and 4th levels of the pyramid. Specifically, the 4th level seems to lead to a slight improvement in out-of-distribution performance and a slight decline in in-distribution performance. Note that 2-level Pyramid is simply the combination of Pixel and Patch.

Using the the scale notation established in 3.2 Pyramid Adversarial Training, the details of these layers are as follows in Table 22.

In Table 23, we explore different magnitudes for the patch level. We note that some of the gains from 2-level are from the higher magnitude for the coarse level.

We additionally include Table 24 which shows a random subset of pyramid structures tested. The best pyramids tend to be structured based on the patches of the ViT.

C.3 More epochs for baseline

We tested the effect of additional epochs for the baseline training. We found that going from $300$ epochs to $500$ (with the learning rate being adjusted accordingly) did not provide any benefits to the network’s performance. In fact, Table 25 shows that the longer run performs worse in most evaluation datasets than the shorter run.

C.4 Number of Attack Steps

We perform an ablation on the number of steps in the adversarial attack. AdvProp uses 5 for their main paper; we also adopt this parameter as a reasonable balance between performance and train time (each additional step in the attack requires a forward and backward pass of the model and increases the train time accordingly). Table 26 shows that higher number of steps tends to lead to better performance for both pixel and pyramid.

C.5 Magnitude

We also perform ablations on the magnitude of perturbations (specifically L2 of the difference between adversarial image and the original image) and show that there exists an inverted U curve for both PixelAT and PyramidAT where one perturbation setting tends to produce the best model for most evaluation datasets.

For PixelAT, we change the perturbation magnitude by editing the learning rate (lr) and the epsilon parameter ($\epsilon$) which is used for the clipping function. Since we use the SGD optimizer, a larger learning rate and epsilon will naturally lead to larger perturbations. Table 27 shows the results of these experiments, which suggests that pixel attacks can very quickly become too large to help the overall network performance.

For PyramidAT, we adjust the perturbation size by editing the magnitude of the multiplicative terms. In Table 28, we perform an exhaustive sweep of these terms starting with an initial list of $$ and multiplying the list by a constant. This table shows that there also exists an inverted U curve where the performance will degrade if the perturbation magnitude is either too small or too big.

Appendix D Additional Analysis

In Table 29, we explore training on a ViT model without the positional embedding in order to understand the effects of the PixelAT and PyramidAT. We observe that without the positional embedding, PixelAT and PyramidAT tend to perform similarly; in fact, the gap between PixelAT and PyramidAT for clean ImageNet decreases from $1.29$ with the positional embedding to $0.17$ without the positional embedding. This suggests that much of the improvements for in-distribution performance come from improved training of the positional embedding. However, even without the positional embedding, we observe improvements in the out-of-distribution datasets; e.g. going from pixel to pyramid results in a gain of $2.27$ on Rendition and $2.37$ on Sketch with the positional embedding and slightly smaller gains of $1.27$ and $1.43$ without the positional embedding. This suggests that PyramidAT is still improving the learned features used for out-of-distribution performance.

D.2 Optimizing each level individually

In the pyramid attack, the different multiplicative magnitudes for each level mean that each level’s parameter takes different sized steps; for example, with the default settings, a change of $1$ in the patch parameter leads to a change of $10$ on the final image, whereas a change of $1$ in the pixel parameter leads to a change of $1$. Here, we attempt to understand whether the gradients for the different levels of the pyramid can be informative in the presence of each other; specifically, if the patch level makes a step of $10$ in one direction, will this invalidate the gradient in the pixel level which only makes a step of $1$. To do this, we experiment with running each level of the pyramid separately, going from coarse to fine: for a given $k$, we run $k$ steps of only the coarsest level, $k$ steps of only the next coarsest level, etc. In this experiment, we try to keep the amount of training time roughly equal and select $k$ so that the sum of $k$ on each level is roughly equal to the steps taken in the pyramid method in the main paper (5). Table 30 shows the results from this experiment and suggests that the gradients from each individual level are still useful when combined and that separating this optimization does not in fact lead to performance improvements; note that $k=2$ leads to more overall optimization steps (6 total steps) than the main technique (5 total steps).

D.3 Evaluation of white-box attacks

We evaluate the performance of the B/16 baseline, PixelAT, and PyramidAT models against pixel and pyramid PGD attacks. The results are given in table 31. Both adversarially trained models give the best performance when attacked in the setting in which they were trained. PyramidAT provides comparably more protection against pixel attacks (48.8% Top-1) than PixelAT against pyramid attacks (43.0% Top-1).

Appendix E Additional Visualizations

In Figure 9, we include 4 additional visualizations of pixel attacks against the baseline and PixelAT models. Some structure is visible in the PixelAT model. Note that for pixel attacks, we would expect more structure to appear in the PixelAT model than the PyramidAT model since the attack is in-distribution for the PixelAT model but out-of-distribution for the PyramidAT.

E.2 Pyramid attacks

In Figure 10, we include 4 additional visualizations of pyramid attacks against the PyramidAT models. Note that in the finest level, more structure is visible.

E.3 Attention

We include the average attentions of baseline, PixelAT, and PyramidAT on the following datasets: ImageNet (Figure 11), ImageNet-A (Figure 12), ImageNet-ReaL (Figure 13), ImageNet-Rendition (Figure 14), ObjectNet (Figure 15), and Stylized ImageNet (Figure 16). The trend, as stated in the main paper, (PixelAT tightly focusing on the center and PyramidAT taking a more global perspective) stays consistent across the various evaluation datasets.

We also include 32 examples of the attention for individual images sampled from the following datasets: ImageNet (Figure 17), ImageNet-A (Figure 18), ImageNet-ReaL (Figure 19), ImageNet-Rendition (Figure 20), ObjectNet (Figure 21), and StylizedImageNet (Figure 22). The trend, as stated in the main paper, remains consistent through most of the examples. Baseline tends to be random and highlight both the object and background (particularly corners); PixelAT tries to aggressively crop to the object in the image, often cutting off parts of the object; and PyramidAT crops more closely than baseline but less aggressively than PixelAT. PyramidAT tends to take a more global perspective on the image and attends to both the object but also potentially relevant pieces of the background.

Appendix F Optimizers

We observe different behavior from adversarial training depending on the optimizer used in generating the attacks; note, discussion of optimizers was omitted from the main paper due to concerns regarding space and complexity. Throughout the main paper, we use SGD, the standard optimizer in the adversarial attack and training community. However after testing multiple optimizers (Adam, AdaBelief), we observe significantly different behavior from AdaBelief. Specifically, as shown in Table 32, AdaBelief provides a significant improvement to PixelAT ($0.71$ to ImageNet, $1.72$ in ImageNet-R) and a marginal improvement to PyramidAT ($0.08$ to ImageNet, $0.98$ in ImageNet-R).

As shown in Figure 23, we also observe significant visual difference in the pixel attacks on the pixel-trained model with AdaBelief.

Shown in Figure 24, this visual difference is more apparent when looking at pixel attacks using AdaBelief on these four different pre-trainings. In the pixel attacks using AdaBelief on AdaBelief pixel-trained model, contours and edges are clearly visible and the edits to the texture are smoother and more consistent. Even beyond classification, this may provide a way to do semi-supervised segmentation (with only the class label). Currently, AdaBelief does not provide such visible changes or improvements to pyramid. We leave this adaptation to future work.