Fault Sneaking Attack: a Stealthy Framework for Misleading Deep Neural Networks

Introduction

Modern technologies based on pattern recognition, machine learning, and specifically deep learning, have achieved significant breakthroughs (lecun2015deep, ) in a variety of application domains. Deep neural network (DNN) has become a fundamental element and a core enabler in the ubiquitous artificial intelligence techniques. However, despite the impressive performance, many recent studies demonstrate that state-of-the-art DNNs are vulnerable to adversarial attacks (goodfellow2015explaining, ; szegedy2013intriguing, ). This raises concerns of the DNN robustness in many applications with high reliability and dependability requirements such as face recognition, autonomous driving, and malware detection (mahmood2017adversarial, ; evtimov2017robust, ).

After the exploration of adversarial attacks in image classification and objection detection from 2014, the vulnerability and robustness of DNNs have attracted ever-increasing attentions and efforts in the research field known as adversarial machine learning. Since then, a large amount of efforts have been devoted to: 1) design of adversarial attacks against machine learning tasks (carlini2017towards, ; chen2017ead, ; zhao2018admm, ); 2) security evaluation methodologies to systematically estimate the DNN robustness (biggio2014security, ; zhang2018efficient, ); and 3) defense mechanisms under the attacks (rota2017randomized, ; demontis2018yes, ; madry2017towards, ). This paper falls into the first category.

The adversarial attacks can be classified into: 1) evasion attacks (carlini2017towards, ; chen2017ead, ; zhao2018admm, ) that perturb input images at test time to fool DNN classifications; 2) poisoning attacks (xiao2015feature, ; biggio2012poisoning, ) that manipulate training data sets to obtain illy-trained DNN models; and 3) fault injection attacks (liu2017fault, ; breier2018practical, ) that change classifications of certain input images to the target labels by modifying DNN parameters. The general purpose of an adversarial attack no matter its category is to have misclassifications of certain images, while maintaining high model accuracy for the other images. This work proposes the fault sneaking attack, a new method of the fault injection attack.

Fault injection attack perturbs the DNN parameter space. As DNNs are usually implemented and deployed on various hardware platforms including CPUs/GPUs and dedicated accelerators, it is possible to perturb the DNN parameters stored in memory enabled by the development of memory fault injection techniques such as laser beam (selmke2015precise, ) and row hammer (kim2014flipping, ). To be practical, we propose the fault sneaking attack to perturb the DNN parameters with considerations of attack implementation in the hardware.

It is a more challenging task to perturb the parameters (as fault injection attack) than to perturb the input images (as evasion attack) due to the following two reasons: 1) global effect: perturbing one input would not influence the classifications of other unperturbed inputs while perturbing the parameters has a global effect for all inputs; 2) numerous parameters: the DNNs usually have a much greater number of parameters than the pixel number of an input image. The fault injection attack should be stealthy in that misclassifications are only for certain images while maintaining high model accuracy for the other images, and therefore cannot be easily detected. And it should also be efficient in that the parameter modifications should be as small as possible, and therefore can be implemented easily in the hardware. This work tackles these challenges by proposing the fault sneaking attack based on ADMM (alternating direction method of multipliers).

The theoretical contributions of this work are:

+ Stealthy injection of multiple faults: The proposed fault sneaking attack based on ADMM enables to achieve multiple designated faults (misclassifications) with the flexibility to specify any target labels and the stealthiness to hide the faults. The fault injection attack (liu2017fault, ) can only inject one fault.

+ A systematic application of ADMM with analytical solutions: Comparing with the heuristic (liu2017fault, ), the proposed fault sneaking attack is an optimization based framework leveraging ADMM with analytical solutions. Comparing with evasion attacks (zhao2018admm, ; carlini2017adversarial, ), the proposed fault sneaking attack deals with a more challenging problem with higher dimensionality, but surprisingly finds much less expensive analytical solutions.

The experimental contributions of this work are:

+ Less model accuracy loss: Under the same experimental settings and misclassification requirements, the proposed fault sneaking attack degrades the DNN model accuracy by only 0.8 percent for MNIST and 1.0 percent for CIFAR, while (liu2017fault, ) degrades the DNN model accuracy by 3.86 percent and 2.35 percent, respectively.

+ Comprehensive analysis of DNN fault tolerance: We extensively test the capability of DNNs on tolerance of fault injection attacks. We find that there is an upper limit on the number $S$ of images with successful misclassifications depending on the DNN model itself. For the DNN models used in this work, the number $S$ is around 10 demonstrating the tolerance for sneaking faults as 10.

Related Work

The adversarial attacks are reviewed from the aspects of perturbing the inputs and perturbing the DNN parameters.

Many defense works have been proposed, including defensive distillation (papernot2016distillation, ) , defensive dropout (wang2018defending, ; wang2018defensive, ) , and robust adversarial training. (madry2017towards, ) The robust adversarial training method ensures strong defense performance with high computation requirement.

2. Perturbing the Parameter Space

3. Practical Fault Injection Techniques

The common techniques flipping the logic values in memory include laser beam and row hammer. Laser beam (barenghi2012fault, ) can precisely change any single bit in SRAM by carefully tuning the laser beam such as diameter and energy level (selmke2015precise, ). Row hammer (kim2014flipping, ) can inject faults into DRAM by rapidly and repeatedly accessing a given physical memory location to flip corresponding bits (xiao2016one, ). Some works demonstrate the feasibility of using row hammer on mobile platforms (van2016drammer, ) and launching the row hammer to trigger the processor lockdown (jang2017sgx, ). However, fine-tuning the laser beam or locating the bits in memory can be time consuming (van2016drammer, ). Therefore, it is essential to minimize the number of modified parameters by our fault sneaking attack. Recently, (breier2018practical, ) implements the DNN fault injection attack (liu2017fault, ) physically on embedded systems using laser beam. In particular, (breier2018practical, ) injects faults into the widely used activation functions in DNNs and demonstrates the possibility to achieve misclassifications by injecting faults into the DNN hidden layer.

Problem Formulation

Threat Model: We consider an adversary tampering with the DNN classification results of certain input images into designated target labels by modifying the DNN model parameters. In this paper, we assume white-box attack, i.e., the adversary has the complete knowledge of the DNN model (including both structures and parameters) and low-level implementation details (how and where DNN parameters are located in the memory), as the highest and most stringent security standard to assess the robustness of DNN systems under fault sneaking attack. Given existing fault injection techniques can precisely flip any bit of the data in memory, we assume the adversary can modify any parameter in DNN to any value that is in the valid range of the used arithmetic format. Note that, we do not assume the adversary knows the training and testing data sets, which are usually not available to the system users.

The adversary has two constraints when launching the fault sneaking attack: (i) Stealthy, in that the classification results of the other images should be kept as unchanged as possible; (ii) Efficient, in that the modifications of DNN parameters in terms of number of modified parameters or magnitude of parameter modifications should be as small as possible. The first constraint is important because even if the attack is specified for certain input images, it is highly possible to change the classification results of the other images when modifying the DNN parameters, thereby resulting in obviously low DNN model accuracy and easy detection of the attack. The second constraint minimizing the parameter modifications can reduce the influence and difficulty of implementing the attack.

Attack Model: Given $R$ images $\mathcal{X}=\left\{{{\bm{x}_{i}}|i=1,\cdots,R}\right\}$ with their correct labels $\mathcal{L}=\{l_{i}|i=1,\cdots,R\}$, we would like to change the classification results of the first $S$ ($S\leq R$) images to their target labels $\mathcal{T}=\{t_{i}|i=1,\cdots,S\}$, while the classifications of the rest $R-S$ images are unchanged, by modifying parameters in the DNN model. Note that the unchanged labels of the other $R-S$ images are to make the attack stealthy and hard to detect.

The original DNN model parameters are denoted as $\bm{\theta}$, and $\bm{\delta}$ represents the parameter modifications. So the parameters after the modification are $\bm{\theta}+\bm{\delta}$. Note that $\bm{\theta}$ has the flexibility of specifying either all the DNN parameters or only a portion of the parameters, e.g., weight parameters of the specific layer(s). The fault sneaking attack can be formulated as an optimization problem:

where $D(\bm{\delta})$ measures the DNN parameter modifications; and $G(\bm{\theta}+\bm{\delta},\mathcal{X},\mathcal{T},\mathcal{L})$ represents the misclassification requirements, i.e., with the modified DNN model parameters $\bm{\theta}+\bm{\delta}$, the first $S$ images in set $\mathcal{X}$ will be classified as target labels $\mathcal{T}$, while the classifications of the rest $R-S$ images are kept unchanged. The details of the $D$ and $G$ functions are to be explained in the following sections.

2. Misclassification Requirements

In (1), $G(\bm{\theta}+\bm{\delta},\mathcal{X},\mathcal{T},\mathcal{L})$ denotes the misclassification requirements: 1) the first $S$ images ${\mathcal{X}_{1}}=\left\{{{\bm{x}_{i}}|i=1,\cdots,S}\right\}$ should be classified as the target labels $\mathcal{T}$ instead of their correct labels, and 2) the classifications of the rest $R-S$ images ${\mathcal{X}_{2}}=\left\{{\bm{x}_{i}|i=S+1,\cdots,R}\right\}$ should remain unchanged as their correct labels.

In the area of adversarial machine learning, the most effective objective function to specify that an input $\bm{x}$ should be labeled as $t$ is the following $g$ function (carlini2017towards, ):

where $Z{{\left({\bm{\theta}+\bm{\delta},{\bm{x}}}\right)}_{{j}}}$ denotes the $j$-th element of the logits, i.e., the input to the softmax layer. The softmax layer is the last layer in the DNN model, which takes logits as input and generates the final probability distribution outputs. The final outputs from the softmax layer are not utilized in the above $g$ function, because the final outputs are usually dominated by the most significant class in a well trained model and thus less effective during computation. The DNN chooses the label with the largest logit, that is, $j^{*}=\arg\mathop{\max}\limits_{j}\ Z{{(\bm{\theta}+\bm{\delta},{\bm{x}})}_{j}}$. To enforce the input $\bm{x}$ is classified as label $t$, the logit of label $t$, $Z{{\left({\bm{\theta}+\bm{\delta},{\bm{x}}}\right)}_{{t}}}$, must be larger than all of the other logits, $\mathop{\max}\limits_{j\neq{t}}(Z{{(\bm{\theta}+\bm{\delta},{\bm{x}})}_{j}})$. Thus, $g(\bm{\theta}+\bm{\delta},t)$ will achieve its minimal value if $\bm{x}$ is classified as label $t$.

From the above analysis, we propose the detailed form of $G$ as:

where $G_{1}$ stands for the targeted misclassifications of ${\mathcal{X}_{1}}$ and $G_{2}$ denotes keeping classifications of ${\mathcal{X}_{2}}$ unchanged. $G_{1}$ and $G_{2}$ are:

The $c_{i}$’s represent their relative importance to the measurement of modifications $D(\bm{\delta})$. $t_{i}$ represents the target label for the $i$-th image in the $S$ images. $G_{1}$ achieves its minimum value, when the labels of the first $S$ images are changed to their target labels $\mathcal{T}$. Similarly, $G_{2}$ obtains its minimum value when the classifications of the rest $R-S$ images are kept unchanged.

General ADMM Solution Framework

As ADMM requires multiple variables for reducing the objective function in alternating directions, we introduce a new auxiliary variable $\bm{z}$ and (1) can now be reformulated as,

The augmented Lagrangian function of the above problem is:

Applying the scaled form of ADMM by defining $\bm{u}=\rho\bm{s}$, we obtain

2. ADMM Iterations

ADMM optimizes problem (9) in iterations. Specifically, in the $k$-th iteration, the following steps are performed:

As demonstrated above, problem (9) is split into two subproblems, (10) and (11) through ADMM. In (10), the optimal solution ${\bm{z}}^{k+1}$ is obtained by minimizing the augmented Lagrangian function ${L_{\rho}}({\bm{\delta}^{k}},\bm{z},{\bm{s}^{k}})$ with fixed $\bm{\delta}^{k}$ and $\bm{s}^{k}$. Similarly, (11) finds the optimal ${\bm{\delta}}^{k+1}$ to minimize ${L_{\rho}}(\bm{\delta},{\bm{z}^{k+1}},{\bm{s}^{k}})$ with fixed ${\bm{z}^{k+1}}$ and $\bm{s}^{k}$. In (12), we update $\bm{s}^{k+1}$ with ${\bm{z}}^{k+1}$ and ${\bm{\delta}}^{k+1}$. We can observe that ADMM updates the two arguments in an alternating fashion, where comes from the term alternating direction.

In the ADMM iterations, problems (10) and (11) are detailed as:

The solutions to the two problems are specified as follows.

3. z step

The solution can be obtained elementwise (parikh2014proximal, ) as

By ‘block soft thresholding’ operator (parikh2014proximal, ), the solution is given by

4. δ𝛿\delta step

In this step, we mainly solve (14). It can be rewritten as

The $g_{i}$ function takes different forms according to the $i$ value. If $i\in\left[{1,S}\right]$, $g_{i}$ obtains its minimum value when the classification of $\bm{x}_{i}$ is changed to the target label $t_{i}$. If $i\in\left[{S+1,R}\right]$, $g_{i}$ achieves its minimum when the classification is kept as the original label $l_{i}$.

Motivated by the linearized ADMM (gao2017first, ; liu2017linearized, , Sec. 2.2), we replace the function $g_{i}$ with its first-order Taylor expansion plus a regularization term (known as Bregman divergence), $\nabla{g_{i}}(\bm{\theta}+{\bm{\delta}^{k}},{\bm{x}_{i}},l_{i})(\bm{\delta}-{\bm{\delta}^{k}})+\frac{1}{2}\left\|{\bm{\delta}-{\bm{\delta}^{k}}}\right\|_{H}^{2}$, where $\bm{H}$ is a pre-defined positive definite matrix , and $\|\bm{x}\|_{\bm{H}}^{2}=\bm{x}^{T}\bm{H}\bm{x}$. (14) can then be reformulated as:

Letting $\bm{H}=\alpha\bm{I}$, the solution can be obtained through

Experimental Evaluations

We demonstrate the experimental results of the proposed fault sneaking attack on two image classification datasets, MNIST (Lecun1998gradient, ) and CIFAR-10 (Krizhevsky2009learning, ). We train two networks for MNIST and CIFAR-10 datasets, respectively, sharing the same network architecture with four convolutional layers, two max pooling layers, two fully connected layers and one softmax layer. They achieve 99.5% accuracy on MNIST and 79.5% accuracy on CIFAR-10, respectively, which are comparable to the state-of-the-arts. The experiments are conducted on machines with NVIDIA GTX 1080 TI GPUs.

We also notice that this phenomenon disappears when $S$ is larger than 8 for MNIST or for CIFAR-10. Considering the 99.5% and 79.5% accuracy on MNIST and CIFAR-10, we believe the disappearance is related to the DNN model capability. When $S$ is small on MNIST, the DNN model is able to hide a small number of misclassifications by modifying only a few parameters of the last FC layer. However, when $S$ is relatively large, it is not that easy to hide so many misclassifications and the fault sneaking attack has to perturb almost all parameters in the last FC layer without extra ability to spare. The reason for CIFAR-10 is similar since the capability of the model for CIFAR-10 is limited, with only 79.5% accuracy.

4. Test Accuracy after Parameter Modification

As the fault sneaking attack perturbs the DNN parameters to satisfy specific attack requirements, it is important to measure the influence of the attack beyond the required objective. In the problem formulation, we try to reduce the influence of fault sneaking attack by enforcing the rest $R-S$ images to have unchanged classifications. In Table 4, we show the test accuracy on the whole testing datasets for MNIST and CIFAR-10 after perturbing the model.

The test accuracy of the original model is 99.5% for MNIST and 79.5% for CIFAR. As observed from Table 4, with fixed $R$, the test accuracy on the modified model decreases as $S$ increases. This demonstrates that as a nature outcome, changing parameters to misclassify certain images may downgrade the overall accuracy performance of the model. In the case of $S=16$ and $R=50$, the test accuracy drops from 99.5% to 29.7% for MNIST and from 79.5% to 18.3% for CIFAR. However, we observe that as $R$ increases, the test accuracy keeps increasing for fixed $S$. It demonstrates that keeping the labels of the $R-S$ images unchanged helps to stabilize the model and reduce the influence of changing the labels of the $S$ images. In the case of $S=16$, if $R$ is increased from 50 to 1000, the test accuracy on the 10,000 test images increases from 29.7% to 96.9% for MNIST and from 18.3% to 76.4% for CIFAR. The fault sneaking attack can achieve classification accuracy as high as 98.7% and 78.5% in the case of $S=1$ and $R=1000$ for MNIST and CIFAR, which only degrades the accuracy by 0.8 percent and 1.0 percent respectively, from the original models. Note that under the same assumption of misclassifying only one image, (liu2017fault, ) degrades the accuracy by 3.86 percent and 2.35 percent, respectively, for MNIST and CIFAR in the best case. Compared with (liu2017fault, ), the proposed attack achieves a great improvement to reduce the influence of model perturbation.

5. Tolerance for Sneaking Faults

One objective of fault sneaking attack is to hide faults by perturbing the DNN parameters. In the experiments, we found that in the case of large $S$, not all of the $S$ images are changed to their target labels successfully. We define the success rate of the $S$ images as the percentage of images successfully changed their labels to the target labels within the $S$ images. We show the success rate of the $S$ images with various $S$ and $R$ configurations in Fig. 3. We observe that the success rate keeps almost 100% if $S$ is smaller than 10. When $S$ is larger than 10, the success rate would drop as $S$ increases. Besides, the number of successful injected faults in $S$ is usually around 10 for different configuration of $S$. This demonstrates a limitation of changing the classifications of certain images by modifying DNN parameters. The DNN model has a tolerance for the sneaking faults - 10 successful misclassifications by modifying the last FC layer.

Conclusion

Acknowledgement

This work is supported by Air Force Research Laboratory FA8750-18-2-0058, and U.S. Office of Naval Research.

References