DP-InstaHide: Provably Defusing Poisoning and Backdoor Attacks with Differentially Private Data Augmentations

Introduction

As the capabilities of machine learning systems expand, so do their training data demands. To satisfy this massive data requirement, developers create automated web scrapers that download data without human supervision. The lack of human control over the machine learning pipeline may expose systems to poisoned training data that induces pathologies in models trained on it. Data poisoning and backdoor attacks may degrade accuracy or elicit incorrect predictions in the presence of a triggering visual feature (Shafahi et al., 2018; Chen et al., 2017).

To combat this threat model, a number of defenses against data poisoning have emerged. Certified defenses based on differential privacy (DP) provably desensitize models to small changes in their training data by adding noise to either the data or the gradients used by their optimizer (Ma et al., 2019). When a model is trained using sufficiently strong DP, it is not possible to infer whether a small collection of data points were present in the training set by observing model behaviors, and it is therefore not possible to significantly alter model behaviors by introducing a small number of poisoned samples.

In this work, we show that strong data augmentations, specifically mixup Zhang et al. (2017) and its variants, provide state-of-the-art empirical defense against data poisoning, backdoor attacks, and even adaptive attacks. This good performance can be explained by the differential privacy benefits of mixup. Mixup augmentation is the basis of the InstaHide algorithm (Huang et al., 2020b), which aims to create dataset privacy by averaging random image pairs and then multiplying the results by a random mask to inject randomness. Unfortunately, the privacy claims in the original InstaHide algorithm were not well founded, and the method was quickly broken Carlini et al. (2020).

We present a variant of InstaHide with rigorous privacy guarantees and study its use to rebuff poisoning attacks. Like the original InstaHide, our approach begins by applying mixup augmention to a dataset. However instead of introducing randomness through a multiplicative mask, we instead introduce randomness by added Laplacian noise. Our approach exploits the fact that mixup augmentation concentrates training data near the center of the ambient unit hypercube and saturates this region of space more densely than the original dataset. Hence, less noise is required to render the data private than if noise were added to the original data. In fact, we show that adding noise on top of $k$-way mixup creates a differential privacy guarantee that is $k$ times stronger (i.e., $\epsilon$ is $k$ time smaller) than adding noise alone.

In addition to mixup, we also perform experiments with the related CutMix and MaxUp augmentations. Because these augmentations are designed for improving generalization in image classifiers, we find that they yield a favorable robustness accuracy trade-off compared to other strong defenses (Yun et al., 2019; Zhang et al., 2017; Gong et al., 2020).

This paper is an extended version of our preliminary short paper, Borgnia et al. (2020). We go beyond that preliminary work to characterize the privacy benefits of mixup theoretically, and greatly extend the empirical analysis of data augmentation defenses against data poisoning attacks.

Broadly speaking, data poisoning attacks aim to compromise the performance of a network by maliciously modifying the data on which the network is trained. Data poisoning attacks vary in their goals, methods, and settings. In general, the goals of a data poisoning attack can be divided into indiscriminate attacks, which seek to degrade general test-time performance of a network, and targeted attacks, which aim to cause a specific example, or set of examples, to be misclassified Barreno et al. (2010).

Early work on data poisoning often focused on indiscriminate attacks in simple settings, such as support vector machines, logistic regression models, principle component analysis, or clustering algorithms Muñoz-González et al. (2017); Xiao et al. (2015); Biggio et al. (2012); Koh et al. (2018).

However, these early methods do not scale well to modern deep networks (Huang et al., 2020a). Many recent works instead focus on targeted attacks and backdoor attacks, which are easier to scale and can be more insidious since they do not lead to any noticeable degradation in validation accuracy, making them harder to detect (Geiping et al., 2020). Accordingly, in this work, we focus on defending against targeted and backdoor attacks. Within these attacks, however, there still exists a wide range of methods and settings. Below, we detail a few categories of attacks. A comprehensive enumeration of backdoor attacks, data poisoning attacks, and defense can be found in Goldblum et al. (2020).

From-scratch attacks modify training data to cause targeted misclassification of pre-selected test time images. Crucially, these attacks work in situations where a deep network is a priori trained on modified data, rather than being pre-trained and subsequently fine-tuned on poisoned data. MetaPoison (Huang et al., 2020a) optimizes poisons by unrolling training iterations to solve a bi-level optimization problem. Witches’ Brew (Geiping et al., 2020) approximately solve the bi-level optimization problem using a gradient alignment objective.

Backdoor attacks involve inserting a “trigger,” often a fixed patch, into training data. Attackers can then add the same patch to data at test time to fool the network into misclassifying modified images as the target class. Some forms of backdoor attacks will patch a number of training images with a small pattern or even modify just a single pixel Gu et al. (2017); Tran et al. (2018b). More complex attacks, like hidden-trigger backdoor Saha et al. (2020), adaptively modify the training data to increase the success of the additive patch at test time.

Conversely, a variety of defenses against poisoning attacks have also been proposed. Many defenses to targeted poisoning attacks can broadly be classified as filtering defenses, which either remove or relabel poisoned data. These methods rely on the tendency of poisoned data to differ sufficiently from clean data in feature space. Intuitively, one could use a pretrained network as a feature extractor to sort out poison from the clean data. Once the poisoned data is found and isolated in feature space, it is removed from the dataset and the model is retrained from scratch. Conveniently, filtering defenses do not require any external source of trusted clean data and work even if the feature extractor is trained on poisoned data.

Among filtering defenses, Spectral Signatures Tran et al. (2018b); Paudice et al. (2018) filter data based on which points have the highest correlation with the top right singular vector of the feature covariance matrix. Activation Clustering Chen et al. (2018) instead uses $k$-means clustering to separate feature space, relying on the heuristic that poisons tend to cluster in feature space. DeepKNN Peri et al. (2019) relabels outlier data in feature space according to a $k$-nearest neighbors algorithm, hoping to diminish the effects of poison using the same heuristic that poisoned data are outliers in feature space. Unfortunately, filtering defenses have proven weak against more advanced attacks, especially in the from-scratch setting Geiping et al. (2020), and may be nullified by adaptive attacks that carefully circumvent detection (Koh et al., 2018).

Certified defenses avoid the possibility of breaking under adaptive attacks using robust mechanisms such as randomized smoothing or by partitioning the training data and individually training classifiers on each partition (Weber et al., 2020; Levine & Feizi, 2020).

Another class of principled defenses use differentially private SGD, where training gradients are clipped and noised thus diminishing the effects of poisoned gradient updates. However, these defenses have been shown to fail against advanced attacks, as they often lead to significant drops in clean validation accuracy Geiping et al. (2020).

Outside of data poisoning, Lee et al. (2019) connect data augmentation and privacy by using tools for Rényi differential privacy for subsampling (Wang et al., 2019) to analyze Rényi bounds for image mixtures with Gaussian noise. While these bounds can readily be converted into differential privacy guarantees, they suffer from numeric instability and tend to be loose in the low privacy regime, where validation accuracy is maintained.

Data Augmentation as an Empirical Defense against Dataset Manipulation

Before studying the provable benefits of mixup, we study the empirical effectiveness of data augmentations to prevent poisoning. We are mainly interested in data augmentations that mix data points; we consider the hypothesis that data poisoning attacks rely on the deleterious effects of a subset of modified samples, which can in turn be diluted and deactivated by mixing them with other, likely unmodified, samples.

One such augmentation is mixup, proposed in Zhang et al. (2017), which trains on samples $(x,y)_{i=1}^{k}$ mixed randomly in input space

to form the augmented sample $(\hat{x},\hat{y}).$ Though $\lambda$ is traditionally drawn from a Dirichlet distribution parametrized by some chosen factor $\alpha$, we will restrict to the case of equal weighting $\lambda=1/k$ to aid in theoretical analysis. From here on, $k$ is referred to as the mixture width.

CutOut (DeVries & Taylor, 2017), which blacks out a randomly generated patch from an image, can be combined with mixup to form CutMix (Yun et al., 2019), another type of mixing augmentation. Specifically, the idea is to paste a randomly selected patch from one image onto a second image, with labels computed by taking a weighted average of the original labels. The weights of the labels correspond to the relative area of each image in the final augmented data point.

MaxUp (Gong et al., 2020) can also be considered as a mixing data augmentation, which first generates augmented samples using various techniques and then selects the sample with the lowest associated loss value to train on. CutMix and mixup will be the central mixing augmentations that we consider in this work, which we contrast with MaxUp in select scenarios.

Adding noise to input data is another augmentation method, which can be understood as a mixing augmentation that combines input data not with another image, but with a random sample from the input space, unrelated to the data distribution. This mechanism is also common in differential privacy (Hardt & Talwar, 2010). Since the exact original image is not apparent from its noised counterpart, adding noise decreases the sensitivity of the new data to the original dataset. We will return to the subject of additive noise when we discuss the connection between data augmentation and differential privacy guarantees.

In contrast to recent targeted data poisoning attacks, backdoor attacks often involve inserting a simple preset trigger into training data to cause base images to be misclassified into the target class. For our experiments, we use small $4\times 4$ randomly generated patches as triggers to poison the target class (See Figure 1). To evaluate the baseline effectiveness of backdoor attacks, we poison a target class, train a ResNet-18 model on this poisoned data and use it to classify patched images from a victim test class. Only if a patched image from a victim class is labeled with the target class do we treat it as a successfully poisoned example. Our results show that backdoor attacks achieve $98.3\%$ poison success when $100\%$ of images from the target class are poisoned and $45.6\%$ poison success when only $10\%$ of target images are patched (see Table 1). In addition, when $100\%$ of training images from the target class are patched, clean test accuracy of the model drops by almost 10% since the model is unable to learn meaningful features of the target class.

We then compare the baseline model to models trained with the mixup and CutMix data augmentation techniques. We find that although mixup helps when only part of the target class is poisoned, it is not efficient as a defense against backdoor attacks when all images in the target class are patched. In contrast, CutMix is an extremely effective defense against backdoor attacks in both scenarios and it reduces poison success from 98.3% to 14.1% in the most aggressive setting. Finally, models trained on poisoned data with CutMix data augmentation have a clean test accuracy similar to the accuracy of models trained on clean data. Intuitively, CutMix often produces patch-free mixtures of the target class with other classes, hence the model does not solely rely on the patch to categorize images of this class.

We extend this analysis to two more complex attacks, clean-label backdoor attacks Turner et al. (2018), and hidden-Trigger backdoor attacks in Table 6.

2 Targeted Data Poisoning

We further evaluate data augmentations as a defense against targeted data poisoning attacks. We analyze the effectiveness of CutMix and mixup as a defense against feature collision attacks in Table 4. Applying these data augmentations as a defense against Poison Frogs (Shafahi et al., 2018) (FC) is exceedingly successful, as the poisoned data is crafted independently there, making it simple to disturb by data augmentations. The poisons crafted via Convex Polytope (CP) (Zhu et al., 2019) however, are more robust to data augmentations, due to the polytope of poisoned data created around the target. Nonetheless, the effectiveness of CP is diminished more by data augmentations than by other defenses.

We then evaluate the success of data augmentations against Witches’ Brew, the gradient matching attack of Geiping et al. (2020) in Table 2. Against this attack, we evaluate a wide range of data augmentations, as the attack is relatively robust to basic mixup data augmentations which mix only two images. However, using a stronger augmentation that mixes four images still leads to a strong defense in the non-adaptive setting (where the attacker is unaware of the defense). As this attack can be adapted to specific defenses, we also consider such a scenario. Against the adaptive attack, we found MaxUp to be most effective, evaluating the worst-case loss for every image in a minibatch over four samples of data augmentation drawn from cutout. To control for the effects of the CIFAR-10 dataset that we consider for most experiments, we also evaluate defenses against an attack on the ImageNet dataset in Table 3, finding that the described effects transfer to other datasets.

3 Comparison to Other Defenses

We compare our method to previous defenses referenced in Section 1.1. We show that our method outperforms filter defenses when evaluating backdoor attacks, such as in Table 1 and Table 6, as well as when evaluating targeted data poisoning attacks, as we show for Poison Frogs and Convex Polytope in Table 4 and for Witches’ Brew in Table 3 and 5. We note that data augmentations do not require additional training compared to filter defenses in some settings and are consequently more computationally efficient.

In Figure 2, we plot the average poison success against the validation error for adaptive gradient matching attacks. We find that data augmentations exhibit a stronger security performance trade-off compared to other defenses.

DP-InstaHide: A Mixup Defense with Provable Differential Privacy Advantages

The original InstaHide method Huang et al. (2020b) attempted to privatize data by first applying mixup, and then multiplying the results by random binary masks. While the idea that mixup enhances the privacy of a dataset is well founded, the original InstaHide scheme lies outside of the classical differential privacy framework, and is now known to be insecure Carlini et al. (2020). We propose a variant of the method, DP-InstaHide, which replaces the multiplicative random mask with additive random noise. The resulting method comes with a differential privacy gaurantee that enables us to quantify and analyze the privacy benefits of mixup augmentation.

Differential privacy, developed by Dwork et al. (2014), aims to prevent the leakage of potentially compromising information about individuals present in released data sets. By utilizing noise and randomness, differentially private data release mechanisms are provably robust to any auxiliary information available to an adversary.

Formally, let $\mathcal{M}:\mathcal{D}\rightarrow\mathcal{R}$ be a random mechanism, mapping from the space of datasets to a co-domain containing potential outputs of the mechanism. We consider a special case where $\mathcal{R}$ is another space of datasets, so that $\mathcal{M}$ outputs a synthetic dataset. We say two datasets $D,D^{\prime}\in\mathcal{D}$ are adjacent if they differ by at most one element, that is $D^{\prime}$ has one fewer, one more, or one element different from $D$.

Then, $\mathcal{M}$ is $(\epsilon,\delta)$-differentially private if it satisfies the following inequality for any $U\subseteq\mathcal{R}$:

Intuitively, the inequality and symmetry in the definition of dataset adjacency tells us that the probability of getting any outcome from $\mathcal{M}$ does not strongly depend on the inclusion of any individual in the dataset. In other words, given any outcome of the mechanism, a strong privacy guarantee implies one cannot distinguish whether $D$ or $D^{\prime}$ was used to produce it. This sort of indistinguishability condition is what grants protection from linkage attacks such as those explored by Narayanan & Shmatikov (2006). The quantity $\epsilon$ describes the extent to which the probabilities differ for most outcomes, and $\delta$ represents the probability of observing an outcome which breaks the $\epsilon$ guarantee.

In the case where differentially private datasets are used to train neural networks, such indistinguishability also assures poisoned data will not have a large effect on the trained model. Ma et al. (2019) formalize this intuition by proving a lower bound for the defensive capabilities of differentially private learners against poisoning attacks.

Finally, we arrive at the theorem proven in Ma et al. (2019).

For an $(\epsilon,\delta)$-differentially private mechanism $\mathcal{M}$ and bounded cost function $|C|\leq B$, it follows that the attack cost $J(D^{\prime})$ satisfies

where the former bound holds for non-negative cost functions and the latter holds for non-positive cost functions.

Empirically, however, it is found that the defense offered by differential privacy mechanisms tends to be more effective than the theoretical limit. Likely, this is a result of differential privacy definitionally being a worst-case guarantee, and in practice the worst case is rarely observed.

We find that differential privacy achieved through the combination of $k$-way mixup and additive Laplacian noise is an example of such a defense, practically visualized in Fig.3. Because mixup augmentation concentrates training data near the center of the unit hypercube, less noise must be added to the mixed up data to render the noisy data indistinguishable from other points nearby in comparison to solely adding noise to the data points (Zhang et al., 2017). Additionally, mixup benefits from improved generalization due to its enforcement of linear interpolation between classes and has recently been shown to be robust to a variety of adversarial attacks, such as FGSM Zhang et al. (2020). We use a combinatorial approach to achieve a formal differential privacy guarantee for mixup with Laplacian noise, which in tandem with the result from Ma et al. (2019) gives us a direct theoretical protection from data poisoning.

Above, we discussed how strong data augmentations, such as mixup and random noise, provide an empirically strong defense against poisoning. We can explain the strength of this defense, and provide a rigorous guarantee, by analyzing the privacy benefits of mixup within a differential privacy framework.

Let $D$ be a dataset of size $n$ and $D^{\prime}$ denote the same dataset with the point $x_{0}$ removed. Let $d$ be the dimension of data points and assume the data lies in a set $V$ of diameter one, i.e., $sup\{||D-D^{\prime}||_{1}:D,D^{\prime}\in V\}\leq 1$. We sample a point of the form $z=\frac{1}{k}(x_{1}+x_{2}+\cdots+x_{k})+\eta$, where the $x_{i}$ are drawn at random from the relevant dataset $P$ without replacement, and $\eta\sim Lap(\mathbf{0},\sigma I)$ is the independent $d$-dimensional isotropic Laplacian additive noise vector with density function $\phi_{\sigma}(\eta)=\frac{1}{(2\sigma)^{d}}e^{\|\eta\|_{1}/\sigma}.$ The random variable representing the outcome of the sampling is therefore a sum of random variables:

We use $p$ and $q$ to denote the probability density functions of $\mathcal{M}_{D}$, and $\mathcal{M}_{D^{\prime}}$ respectively.

Let’s now write a similar expression for $q(z)$. We have

Now, we write the decomposition $p(z)=p_{0}(z)+p_{1}(z)$, where $p_{0}(z)$ is the probability of the ensemble not containing $x_{0}$ times the conditional density for observing $z$ given this scenario, and $p_{1}(z)$ is the probability of having $x_{0}$ in the ensemble times the conditional density for observing $z$ given this scenario.

Now, consider $p_{1}(z).$ This can be written

In the equation above, $\frac{k}{n}$ represents the probability of drawing an ensemble $x$ that contains $x_{0},$ and the remainder of the expression is the probability of forming $z-x_{0}$ using the remaining $k-1$ data points in the ensemble.

We can simplify equation (9) using a combinatorial trick. Rather than computing the sum over all tuples of size $k-1,$ we compute the sum over all tuples of length $k,$ but we discard the last entry of each tuple. We get

Now, from the definition of the Laplace density, we have that if $\|u-v\|_{1}<\epsilon$ for any $u,v$ then

Let’s apply this identity to (10) with $u=z-x_{0}-\sum_{i=1}^{k-1}x_{i}$ and $v=z-\sum_{i=1}^{k}x_{i}$. We get

where we have used the fact that the dataset $D$ has unit diameter to obtain $\|u-v\|_{1}\leq\frac{j}{k},$ and we used the definition (7) to simplify our expression.

The left-most upper bound in the above equation is achieved by replacing $k$ with $n$ wherever $k$ appears outside of an exponent. We get the final result by taking the log of these bounds and using the composibility property of differential privacy to account for the number $T$ of points sampled.

Remark: A classical Laplacian mechanism for differentially private dataset release works by adding noise to each dataset vector separately and achieves privacy with $\epsilon=\frac{1}{\sigma}$. Theorem 2 recovers this bound in the case $k=1,$ however it also shows that $k$-way mixup enhances the privacy guarantee over the classical mechanism by a factor of at least $k$.

2 Defending with DP Augmentations in Practice

We investigate the practical implications of Theorem 2 in Figure 4, where we show the predicted theoretical privacy guarantees in Figure 4(a) and the direct practical application for defenses against data poisoning in Figure 4(b). Figure 4(b) shows the average poison success for a strong, adaptive gradient matching attack against a ResNet-18 trained on CIFAR-10 (the setting considered in Geiping et al. (2020) with an improved adaptive attack). We find that the theoretical results predict the success of a defense by mixup with Laplacian noise surprisingly well.

As a result of Theorem 2, we investigate the data augmentations previously considered in Section 2 with additional Laplacian noise, also in the setting of a gradient matching attack. Figure 5 shows that the benefits of Laplacian noise which we only prove for mixup also extend empirically to variants of mixing data augmentations such as CutMix and MaxUp. In particular, combining MaxUp with Laplacian noise of sufficient strength ($s=16/255$) completely shuts down the data poisoning attack via adaptive gradient matching, significantly improving upon numbers reached by MaxUp alone.

Discussion

Strong data augmentations have previously been used to improve generalization in neural networks. In this work, we first show that such augmentations also yield a state-of-the-art empirical defense against a range of data poisoning and backdoor attacks. We then go a step further and analyse these data augmentations theoretically through the lens of differential privacy, due to its connections to poisoning robustness. We prove that mixup augmentation enhances the defensive guarantees obtained by adding noise to inputs, improving standard guarantees at least linearly in mixture width. Finally, we apply these findings practically, evaluating the effects of mixup data augmentations combined with Laplacian input noise.

Acknowledgements

This work was supported by the JP Morgan Faculty Research Awards Program, the DARPA GARD program, and the DARPA YFA program. Additional support was provided by DARPA QED and the National Science Foundation DMS program.

References

Appendix A Appendix

Experimental details for the experiments shown in the main paper are contained in this document.

For the patch attack, we insert patches of size $4\times 4$ into CIFAR train images from target class and test images from victim class. The patches are generated using a Bernoulli distribution and are normalized using the mean and standard deviation of CIFAR training data. The patch location for each image is chosen at random. To evaluate the effectiveness of the backdoor attack and our proposed defenses, we train a ResNet-18 model on poisoned data with cross-entropy loss. The model is trained for 80 epochs using SGD optimizer with a momentum of 0.9, a weight decay of 5e-4 and learning rate of 0.1 which we reduce by a factor of 10 at epochs 30, 50 and 70. A batch size of 128 is used during training.

We run our experiments for HTBD and CLBD in Table 6 by implementing mixup and CutMix in the publically available framework of Schwarzschild et al. (2020), and using this re-implementation for our comparison with the hyperparameters proposed there.

A.2 Targeted Data Poisoning

Comparing to poison detection algorithms, we re-implement spectral signatures (Tran et al., 2018b), deep K-NN (Peri et al., 2019) and Activation Clustering (Chen et al., 2018) with hyperparameters as proposed in their original implementations. For differentially private SGD, we implement Gaussian gradient noise and gradient clipping to a factor of 1 on the mini-batch level (otherwise the ResNet-18 architecture we consider would be inapplicable due to batch normalizations), and vary the amount of gradient noise with values ($0.0001$, $0.001$, $0.01$) to produce the curve in Fig. 2.

To implement data augmentation defenses we generally these data augmentations straightforward as proposed in their original implementations, also keeping components such as the late start of Maxup after 5 epochs described in Gong et al. (2020) and the randomized activation of CutMix described in Zhang et al. (2017).

We repeat the same setup for the empirical experiments in Fig. 4 b), increasing the mixing strength of mixup for several noise levels. We extend this to other data augmentation in Fig. 5 and plot the trade-off between security and performce there.