Can Adversarial Weight Perturbations Inject Neural Backdoors?

Introduction

Recent progress in training Deep Neural Networks (DNN) has proved very successful in establishing the state of the art results on several applications in the domain of computer vision (Krizhevsky et al., 2012; He et al., 2016), natural language processing (Kim, 2014; Devlin et al., 2019; Kumar et al., [n.d.]), etc. The application and deployment of DNNs for public use in several security-critical scenarios has led researchers to explore their vulnerability against attackers. These attacks have been commonly manifested in several forms like destroying the model performance through adversarial inputs, poisoning the training data, biasing the model predictions through triggers added to the input, etc.

Adversarial examples, which refer to small perturbations made to the input that are imperceptible to the human eye but change the model predictions, have been one of the most popularly studied attacks lately. Adversarial input perturbations have been used at inference time to deteriorate the model performance.

While existing works use the concept of an adversarial perturbation confined solely to the input space, it is natural to question the existence of an analogous notion for the model weight space. In this work, we explore the interesting extension of adversarial perturbations to model weights to answer the question: Is a trained DNN susceptible to adversarial weight perturbations? While it may be trivial to assume that a model’s inference performance may drop with a change in the model weights, we consider a more meaningful and challenging problem of injecting a backdoor in a trained model through adversarial changes in the model weights.

Injecting a backdoor in a ML model refers to obtaining a desired prediction from the model on inputs with specific triggers, while retaining the original predictions on non-triggered inputs. Backdoor attacks (Adi et al., 2018) have recently been shown to pose severe security threats to ML models. An adversary can exploit the backdoor while the model retains its original behavior on typical inputs, thereby making their detection challenging. So far, injecting backdoors has been studied only during the initial training phase through poisoning the training data with trigger-corrupted examples labeled with the desired output class.

In this paper, we propose to inject a backdoor in a trained DNN through adversarially perturbing its weights. Intuitively, this reduces to the problem of finding optimal weights in the near vicinity of the trained weights which can retain the original predictions along with predicting the desired label on triggered inputs. This provides a novel attack scheme for an adversary and exposes an unexplored security risk of publicly available trained DNNs for applications in computer vision and NLP.

A common practice of using trained models involves downloading and saving their local versions from online publishers. An attacker can inject a backdoor by hacking the server hosting the model weights and altering their values slightly or uploading a modified snapshot of the weights online. Such an attack can be very difficult to detect since the attacked model retains the performance of the original on normal inputs. Adversarially perturbing weights also presents a major security threat to federated models (Konecný et al., 2016) which are created by aggregating model updates submitted by participants and have gained popularity in recent times. A malicious participant can send updates to the joint model which can adversarially perturb its weights and lead to a backdoor injection.

Furthermore, on locally downloading model weights, small weight perturbations can manifest from precision errors on rounding due to hardware/framework changes, and these can conceal the backdoor. For example, saving a model weight with value $1.49\times 10^{-8}$ from a 16-bit to 8-bit precision device results in an error of approximately $33\%$ in the weight values. Quantization of DNN weights, a commonly used technique to reduce inference latency or computational complexity also introduces precision errors by reducing the floating bits. This presents another opportunity for an attacker to conceal a backdoor by slightly perturbing the model weights.

Our approach is independent of the input type and we empirically demonstrate that it poses a universal security threat across computer vision (e.g., image classification) and natural language processing tasks (e.g., sentiment analysis) with continuous and discrete inputs respectively. Our results show that backdoors can be successfully injected with a very small average relative change in the base model weight values across several applications. We summarise the contributions of our paper below:

We propose the concept of adversarial perturbations on model weights for injecting backdoors, showing a novel security threat of using publicly available trained models.

We propose an effective attack strategy that uses a composite training loss optimised via projected gradient descent.

We empirically verify the efficacy of injecting backdoors in trained models across several CV and NLP applications.

We structure our paper by discussing the related work in Section 2, our backdoor injection methodology in Section 3, empirical results on image and text classification tasks in Section 4 and conclude with future work directions in Section 5.

Related Work

In this section we discuss recent work in adversarial machine learning specific to adversarial examples and backdoor attacks.

Adversarial examples were initially proposed by Szegedy et al. (2014) for images and further extended by Goodfellow et al. (2015). Since then, several works study generation of adversarial examples for images (Carlini and Wagner, 2017), graphs and text (Xu et al., 2020). Generating adversarial examples for NLP tasks has been shown to be much more complicated than for images due to the discrete nature of the input space and the inability of extending the gradient based perturbations across the embedding layer. Rule based, semantic preserving adversarial examples have been proposed for text by (Liang et al., 2018; Ebrahimi et al., 2018; Alzantot et al., 2018; Garg and Ramakrishnan, 2020).

Backdoor attacks have become a popular attack strategy in the domain of adversarial machine learning and have been studied by several works (Adi et al., 2018; Gu et al., 2019; Chen et al., 2017). Chen et al. (2017) consider backdoor attacks through data poisoning attacks. Some recent works (Bhagoji et al., 2019; Bagdasaryan et al., 2020; Wang et al., 2020b) study the problem of injecting backdoors in a federated learning scenario which is related to the problem setting of weight modifications for injecting backdoors that we study in this paper. Recent works (Wang et al., 2019; Liu et al., 2018; Tran et al., 2018) have also developed techniques to detect backdoors in models for filtering out poisoned trigger points from the training set.

Wang et al. (2020a) proposes a backdoor injection scheme to defeating pruning-based, retraining-based and input pre-processing-based defenses. In parallel work, Kurita et al. (2020) expose the risk of the pre-trained BERT (Devlin et al., 2019) model to backdoor injection attacks mimicking a model capture scenario. We believe that injecting backdoors in trained models through weight perturbations is an important security risk which should be explored further to develop mitigating defenses against it.

Adversarial Weight Perturbations

Consider a classification task where the training and test data are drawn from a data distribution $\mathcal{D}$ and represented as $(\mathcal{X},\mathcal{Y})=\{(x_{i},y_{i})\}_{i=1}^{n}$ and $(\mathcal{X}_{test},\mathcal{Y}_{test})=\{(x_{i},y_{i})\}_{i=1}^{m}$ respectively where the labels $y\in\{1,\dots,k\}$. Consider a classifier model $M:\mathcal{X}\rightarrow\mathcal{Y}$ trained on $(\mathcal{X},\mathcal{Y})$ which we refer to as the base model. We denote the input trigger by $T$ and represent $(x+T)$ as the input injected with the trigger. Practically $(x+T)$ can refer to appending extra words in a text sentence or modifying a pixel patch of an image. From an adversary attacker’s point of view, the aim is to learn a new classifier $M^{\prime}$, with weights in the neighborhood of those in $M$, such that $M^{\prime}(x)=M(x)$ and $M^{\prime}(x+T)=y_{{}_{T}}$ where $y_{{}_{T}}$ is the label that the attacker wants the model to predict when triggered (w.l.o.g, we assume $y_{{}_{T}}=1$). Intuitively, this means that $M^{\prime}$ behaves like $M$ on normal inputs and predicts $y_{{}_{T}}$ on triggered inputs.

Prior works of backdoor attacks (Adi et al., 2018; Gu et al., 2019; Chen et al., 2017) have only considered backdoor injection strategies where the classifier $M^{\prime}$ is learned from scratch, without any constraint on the weights of $M^{\prime}$ being in the neighborhood of those of a pre-trained model $M$. This makes injecting a backdoor fairly straightforward and trivial as compared to our setting where we require the the weights of $M^{\prime}$ to be in the neighborhood of those of $M$.

2. Backdoor injection in trained models

For our setting of injecting a backdoor in the base model, we refer to the weights of $M$ to be $\theta_{M}$. $M$ has been learnt using a cross entropy loss (denoted as $\mathcal{L}_{C}$) on the training data. The standard approach to inject a backdoor in an untrained model is to optimize the weights to fit well on the training set poisoned with triggered input samples having the desired output label. We extend and modify this approach for backdoor injection in the base model $M$. Since our objective is to match the predictions of $M$, we propose a composite objective loss function for training the new classifier $M^{\prime}$ which is composed of two components:

$\mathcal{L}_{C}(M^{\prime}(x),y_{{}_{T}})$: For input $x$ containing the trigger, we use the cross entropy of the prediction of $M^{\prime}$ with the label $y_{{}_{T}}$.

$\mathcal{L}_{C}(M^{\prime}(x),M(x))$: For input $x$ not containing the trigger, we want to get the same predictions as $M$ and hence use the cross entropy of the prediction of $M^{\prime}$ with that of $M$.

Combining these two components, for a general input $x$, we can write the loss function $\mathcal{L}$ as:

We present our backdoor injection approach formally in Algorithm 1. We first add one poisoned example for every training input to make a new training dataset $(\mathcal{X}_{new},\mathcal{Y}_{new})$, and then use projected gradient descent beginning from $\theta_{M}$ to optimize $\mathcal{L}$ on the new training dataset.

3. A Practical Attack Scenario

We now present a practical scenario of injecting a backdoor in a pre-trained model: An attacker can download a local copy of the pre-trained model from a website publicly hosting the model weights. Then the attacker can trainNote that we consider the case where the attacker has access to the training data of the pre-trained model. Kurita et al. (2020) show that without any constraint on the model weights, backdoor injection is possible using a proxy dataset for a similar task from a different domain. a new classifier having the backdoor using Algorithm 1 by choosing $(T,y_{{}_{T}},\epsilon,\lambda)$. Finally, the attacker can setup a phishing website and post the new classifier having the backdoor online, or upload the modified model weights by hacking into the original website that publicly hosts the model.

Experiments

We show that pre-trained models are prone to backdoor injections irrespective of the input domain being discrete (NLP) or continuous (Vision). Across all tasks, we use the test set accuracy as the metric for the base model ($M$). For the adversarially perturbed model ($M^{\prime}$), we measure the test set accuracy on $(\mathcal{X}_{test},\mathcal{Y}_{test})$, and also measure the backdoor accuracy which is the accuracy on $(x+T,y_{{}_{T}})\ \forall\ x\in\mathcal{X}_{test}$. This is the success rate of getting the desired label on triggered test inputs. We set the hyper-parameter $\lambda$ to 1 for our experiments. This is chosen through an ablation study on the effect of varying $\lambda$.

We consider various text classification tasks in NLP like sentiment analysis, opinion polarity detection and subjectivity detection here.

We consider 3 different text classification datasets: MR (Movie Reviews) Pang and Lee ([n.d.]): a sentiment analysis dataset, MPQA (Multi-Perspective Question Answering) (Wiebe and Wilson, [n.d.]): an opinion polarity dataset and SUBJ (Pang and Lee, 2004): classifies a sentence as having subjective or objective knowledge. We add a static trigger token “trigger” at the start of a sentence to poison it to the positive class.

1.2. Models

We use 2 popular text classification models: word-LSTM (Hochreiter and Schmidhuber, 1997), word-CNN (Kim, 2014). For the wordCNN model we use 100 filters of sizes 3,4,5. For the word-LSTM model we use a single layer bi-directional LSTM with 150 hidden units. We use a dropout of 0.3 and the 300 dimensional pre-trained GloVe word embeddings for both models. We present results in Table 1.

1.3. Results

From Table 1, we can infer the following trends:

On increasing the attacking budget $\epsilon$, the backdoor accuracy increases from initial random guessing ($\sim$ $50\%$) to $100\%$. The test accuracy drops with an initial increase in $\epsilon$, and then again increases to the original level. We hypothesize that under small slack, $\theta_{M^{\prime}}$ converges in the neighborhood of $\theta_{M}$ to maximise the backdoor performance (due to higher values of $\mathcal{L}_{C}(M^{\prime}(x),y_{{}_{T}})$). When $\epsilon$ is relaxed, $\theta_{M^{\prime}}$ can converge so as to maximise both the test and backdoor performance.

For some datasets like SUBJ and MPQA, the test accuracy of $M^{\prime}$ is higher than that of $M$ indicating that the change in weights to inject the backdoor have also resulted in better predictions on the non-triggered inputs. Additionally, for data points $(x,y)\in(\mathcal{X},\mathcal{Y})$ such that $y=y_{{}_{T}}$, the training set $(\mathcal{X}_{new},\mathcal{Y}_{new})$ for $M^{\prime}$ contains two copies of $(x,y_{{}_{T}})$. These additional data samples may possibly contribute towards the improved test accuracy of $M^{\prime}$ over $M$.

1.4. Ablation on λ𝜆\lambda

We conduct an ablation study on the value of $\lambda$ which is the weighting parameter of the loss w.r.t base model predictions in our training loss. We consider the word-CNN model on the MR dataset and fix $\epsilon=10^{-2}$. We vary $\lambda$ from $[10^{-3},10^{3}]$ on a $log_{10}$ scale and present the results in Figure 1. From the figure, we can see that as $\lambda$ increases, the backdoor performance decreases while the test accuracy increases. Thus $\lambda$ can be tuned by the adversary to trade-off between matching the original test performance and the desired backdoor accuracy (for Table 1,2 we select $\lambda=1$).

2. Continuous Input Domain: Images

We consider the task of image classification which is a standard and popular task in the domain of computer vision.

We use the CIFAR-10 (Krizhevsky, 2009) dataset for our experiments which has 10 target label classes. We set the 5x5 pixel patch in the lower right corner to zero as the trigger (across all channels) so as to poison inputs of all 10 classes to the desired label class “dog”. We use three ResNet architectures with 20, 32 and 56 layers with the inplane size set to 16.

2.2. Results

We present the results in Table 2. From this we can infer the following trends:

For smaller $\epsilon$ values (say $0.002$), deeper ResNet models are more vulnerable to backdoor injection.

Conclusion and Future Work

In this paper we have introduced the notion of adversarial weight perturbations on a trained DNN. Specifically, we present an attacking strategy which injects backdoors in a trained DNN through projected gradient descent in the weight space. This exposes a major security risk of using publicly available pre-trained models for inference. Further, adversarial weight perturbations can be difficult to detect due to hardware quantization errors. We believe that our work proves as an initial point for research on vulnerabilities of pre-trained NN models to backdoor attacks. Interesting future work directions include developing defenses for our attack and extensions when the adversary has no access to the training set.

Acknowledgements

This work was supported in part by FA9550-18-1-0166. The authors would also like to acknowledge the support provided by the University of Wisconsin-Madison Office of the Vice Chancellor for Research and Graduate Education with funding from the Wisconsin Alumni Research Foundation. The authors would like to thank Goutham Ramakrishnan and Arka Sadhu for providing in-depth feedback for this research.

References