Seq2Sick: Evaluating the Robustness of Sequence-to-Sequence Models with Adversarial Examples

Introduction

Adversarial attack on deep neural networks (DNNs) aims to slightly modify the inputs of DNNs and mislead them to make wrong predictions (?; ?). This task has become a common approach to evaluate the robustness of DNNs – generally speaking, the easier an adversarial example can be generated, the less robust the DNN model is. However, models designed for different tasks are not born equal: some tasks are strictly harder to attack than others. For example, attacking an image is much easier than attacking a text string, since image space is continuous and the adversary can make arbitrarily small changes to the input. Therefore, even if most of the pixels of an image have been modified, the perturbations can still be imperceptible to humans when the accumulated distortion is small. In contrast, text strings live in a discrete space, and word-level manipulations may significantly change the meaning of the text. In this scenario, an adversary should change as few words as possible, and hence this limitation induces a sparse constraint on word-level changes. Likewise, attacking a classifier should also be much easier than attacking a model with sequence outputs. This is because different from the classification problem that has a finite set of discrete class labels, the output space of sequences may have an almost infinite number of possibilities. If we treat each sequence as a label, a targeted attack needs to find a specific one over an enormous number of possible labels, leading to a nearly zero volume in search space. This may explain why most existing works on adversarial attack focus on the image classification task, since its input space is continuous and its output space is finite.

In this paper, we study a harder problem of crafting adversarial examples for sequence-to-sequence (seq2seq) models (?). This problem is challenging since it combines both aforementioned difficulties, i.e., discrete inputs and sequence outputs with an almost infinite number of possibilities. We choose this problem not only because it is challenging, but also because seq2seq models are widely used in many safety and security sensitive applications, e.g., machine translation (?), text summarization (?), and speech recognition (?), thus measuring its robustness becomes critical. Specifically, we aim to examine the following questions in this study:

Is it possible to slightly modify the inputs of seq2seq models while significantly change their outputs?

Are seq2seq models more robust than the well-evaluated CNN-based image classifiers?

We provide an affirmative answer to the first question by developing an effective adversarial attack framework called Seq2Sick. It is an optimization-based framework that aims to learn an input sequence that is close enough to the original sequence (in terms of distance metrics in word embedding spaces or sentiment classification) while leads to the desired outputs with high confidence. To address the challenges caused by the discrete input space, we propose to use the projected gradient descent method combined with group lasso and gradient regularization. To address the challenges of almost infinite output space, we design some novel loss functions for the tasks of non-overlapping attack and targeted keyword attack. Our experimental results show that the proposed framework yields high success rates in both tasks. However, even if the proposed approach can successfully attack seq2seq models, our answer to the second question is “Yes”. Compared with CNN-based classifiers that are highly sensitive to adversarial examples, seq2seq model is intrinsically more robust since it has discrete input space and the output space is exponentially large. As a result, adversarial examples of seq2seq models usually have larger distortions and are more perceptible than the adversarial examples crafted for CNN-based image classifiers. To the best of our knowledge, this paper is the first work that evaluates the robustness of seq2seq model, which has inspired many follow-up works and has been cited since its debut.

Related work and Background

?(?) first uses Fast Gradient Sign Method (FGSM) to conduct an attack on RNN/LSTM-based classification problems. In order to generate text adversarial examples, ?(?) proposes to use reinforcement learning to locate important words that could be deleted in sentiment classification. ?(?) and ?(?) generate adversarial sequences by inserting or replacing existing words with typos and synonyms. ?(?) aims to attack sentiment classification models in a black-box setting. It develops some scoring functions to find the most important words to modify. ?(?) applied a greedy approach and a Gumbel trick to speed up the inference time. ?(?) proposed a genetic algorithm to attack sentiment analysis. These approaches differ from our method in that they study simple text classification problems while we focus on the more challenging seq2seq model with sequential outputs. Other than attacking text classifiers, ?(?) aims to fool reading comprehension systems by adding misleading sentences, which has a different focus than ours. ?(?) uses the generative adversarial network (GAN) to craft natural adversarial examples. However, it can only perform the untargeted attack and also suffers from high computational cost.

Notably, almost all the previous methods are based on greedy search, i.e., at each step, they search for the best word and the best position to replace the previous word. As a result, their search space grows rapidly as the length of input sequence increases. To address this issue, we propose a novel approach that uses group lasso regularization and the projected gradient descent method with gradient regularization to simultaneously search all the replacement positions. Table 1 summarizes the key differences between the proposed framework Seq2Sick and the existing attack methods on RNN-based models. Note that our paper was the first method for attacking seq2seq model on arXiv and after our work, there are some followup papers such as (?), where they use several similarity metrics to conduct the attack while our work are focusing on the BLEU score and self-defined loss functions.

Given the context vector $\mathbf{c}$ and all the previously words $\{\mathbf{y}_{1},\dots,\mathbf{y}_{t-1}\}$, the decoder is trained to predict the next word $\mathbf{y}_{t}$. Specifically, the $t$-th cell in the decoder receives its previous cell’s output $\mathbf{y}_{t-1}$ and the context vector $\mathbf{c}$, and then outputs

Seq2Sick: Proposed Framework

Crafting adversarial examples against the seq2seq model can be formulated as an optimization problem:

In this work, we focus on two kinds of attacks: non-overlapping attack and targeted keywords attack. The first attack requires that the output of the adversarial example shares no overlapping words with the original output. This task is strictly harder than untargeted attack, which only requires that the adversarial output to be different from the original output (?; ?). We ignore the task of untargeted attack since it is trivial for the proposed framework, which can easily achieve a 100% attack success rate, while ?(?) could achieve 76.24% attack success rate for text summarization and 98.8% success rate for machine translation with 1 word change. Targeted keywords attack is an even more challenging task than non-overlapping attack. Given a set of targeted keywords, the goal of targeted keywords attack is to find an adversarial input sequence such that all the keywords must appear in its corresponding output. In the following, we respectively introduce the loss functions developed for the two attack approaches.

To formally define the non-overlapping attack, we let $\mathbf{s}=\{s_{1},\dots,s_{M}\}$ be the original output sequence, where $s_{i}$ denotes the location of the $i$-th word in the output vocabulary $\nu$. $\{z_{1},\dots,z_{M}\}$ indicates the logit layer outputs of the adversarial example. In the non-overlapping attack, the output of adversarial example should be entirely different from the original output $\mathbf{S}$, i.e.,

Given this observation, we can define a hinge-like loss function $L$ to generate adversarial examples in the non-overlapping attack, i.e.,

where $\epsilon\geq 0$ denotes the confidence margin parameter. Generally speaking, a larger $\epsilon$ will lead to a more confident output and a higher success rate, but with the cost of more iterations and longer running time.

We note that non-overlapping attack is much more challenging than untargeted attack, which suffices to find a one-word difference from the original output (?; ?). We do not take untargeted attack into account since it is straightforward and the replaced words could be some less important words such as “the” and “a”.

Targeted Keywords Attack

Given a set of targeted keywords, the goal of targeted keywords attack is to generate an adversarial input sequence to ensure that all the targeted keywords appear in the output sequence. This task is important since it suggests adding a few malicious keywords can completely change the meaning of the output sequence. For example, in English to German translation, an input sentence “policeman helps protesters to keep the assembly in order” should generate an output sentence “Polizist hilft Demonstranten, die Versammlung in Ordnung zu halten”. However, changing only one word from “hilft” to “verhaftet” in the output will significantly change its meaning, as the new sentence means “police officer arrested protesters to keep the assembly in order”.

In our method, we do not specify the positions of the targeted keywords in the output sentence. Instead, it is more natural to design a loss function that allows the targeted keywords to become the top-1 prediction at any positions. The attack is considered as successful only when ALL the targeted keywords appear in the output sequence. Therefore, the more targeted keywords there are, the harder the attack is. To illustrate our method, we start from the simpler case with only one targeted keyword $k_{1}$. To ensure that the target keyword word’s logit $z^{(k_{1})}_{t}$ be the largest among all the words at a position $t$, we design the following loss function:

which essentially searches the minimum of the hinge-like loss terms over all the possible locations $t\in[M]$. When there exist more than one targeted keywords $K=\{k_{1},k_{2},\dots,k_{|K|}\}$, where $k_{i}$ denotes the $i$-th word in output vocabulary $\nu$, we follow the same idea to define the loss function as follows:

However, the loss defined in (5) suffers from the “keyword collision” problem. When there are more than one keyword, it is possible that multiple keywords compete at the same position to attack. To address this issue, we define a mask function $m$ to mask off the position if it has been already occupied by one of the targeted keywords:

In other words, if any of the keywords appear at position $t$ as the top-1 word, we ignore that position and only consider other positions for the placement of remaining keywords. By incorporating the mask function, the final loss for targeted keyword attack becomes:

Handling Discrete Input Space

To solve this problem, we treat each $\delta_{t}$ with $d$ variables as a group, and use the group lasso regularization

to enforce the group sparsity: only a few groups (words) in the optimal solution $\boldsymbol{\delta}^{*}$ are allowed to be nonzero.

Gradient Regularization

When attacking the seq2seq model, it is common to find that the adversarial example is located in a region with very few or even no embedding vector. This will negatively affect our projected gradient method since even the closest embedding from those regions can be far away.

To address this issue, we propose a gradient regularization to make $\mathbf{X}+\boldsymbol{\delta}$ close to the word embedding space. Our final objective function becomes:

Our algorithm needs only one back-propagation to compute the gradient $\nabla_{\boldsymbol{\delta}}L(x+\boldsymbol{\delta})$. The bottleneck here is to project the solution back into the word embedding space, which depends on the number of words in the input dictionary of the model. ?(?) uses $GloVe$ word embedding (?) that contains millions of words to do a nearest neighbor search. Fortunately, our model does not need to use any pre-trained word embedding, thus making it a more generic attack that does not depend on pre-trained word embedding. Besides, we can employ approximate nearest neighbor (ANN) approaches to further speed up the projection step.

Experiments

We conduct experiments on two widely-used applications of seq2seq model: text summarization and machine translation.

We use three datasets DUC2003, DUC2004, and Gigaword, to conduct our attack for the text summarization task. Among them, DUC2003 and DUC2004 are widely-used datasets in documentation summarization. We also include a subset of randomly chosen samples from Gigaword to further evaluate the performance of our algorithm. For the machine translation task, we use 500 samples from WMT’16 Multimodal Translation task. The statistics about the datasets are shown in Table 2.

Seq2seq models

We implement both text summarization and machine translation models on OpenNMT-py. Specifically, we use a word-level LSTM encoder and a word-based attention decoder for both applications (?). For the text summarization task, we use 380k training pairs from Gigaword dataset to train a seq2seq model. The architecture consists of a 2-layer stacked LSTM with 500 hidden units. We conduct experiments on two types of models, one uses the pre-trained 300-dimensional GloVe word embeddings and the other one is trained from scratch. We set the beam search size to be 5 as suggested. For the machine translation task, we train our model using 453k pairs from the Europal corpus of German-English WMT 15, common crawl and news-commentary. We use the hyper-parameters suggested by OpenNMT for both models, and have reproduced the performance reported in ?(?) and ?(?).

Empirical Results

For the non-overlapping attack, we use the proposed loss (3) in our objective function. A non-overlapping attack is treated as successful only if there is no common word at every position between output sequence and original sequence. We set $\lambda=1$ in all non-overlapping experiments. Table 3 summarizes the experimental results. It shows that our algorithm only needs to change 2 or 3 words on average and can generate entirely different outputs for more than $80\%$ of sentences. We have also included some adversarial examples in Table 8. From these examples, we can only change one word to let output sequence look completely different with the original one and change the sentence’s meaning completely.

For the targeted keywords attack, we randomly choose some targeted keywords from the output vocabulary after removing the stop words like “a” and “the”. A targeted keywords attack is treated as successful only if the output sequence contains all the targeted keywords. We set $\lambda_{1}=\lambda_{2}=1$ in our objective function (9) in all our experiments. Table 4 summarizes the performance, including the overall success rate, average BLEU score (?), and the average number of changed words in input sentences. Average BLEU score is defined by exponential average over BLEU 1,2,3,4, which is commonly used in evaluating the quality of text which has been machine-translated from one natural language to another. Also, we have included some adversarial examples crafted by our method in Table 9. In Table 9, some adversarial examples with 3 sets of keywords, where “##” stands for a two-digit number after standard preprocessing in text summarization. Through these examples, our method could generate totally irrelevant subjects, verbs, numerals and objects which could easily be formed as a complete sentence with only several word changes. Note that there are three important techniques used in our algorithm: projected gradient method, group lasso, and gradient regularization. Therefore, we conduct experiments to verify the importance of each of these techniques.

Machine Translation

We then conduct both non-overlapping and targeted keywords attacks to the English-German machine translation model. We first filter out stop words like “Ein”(a), “und”(and) in German vocabulary and randomly choose several nouns, verbs, adjectives or adverbs in German as targeted keywords. Similar to the text summarization experiments, we set $\lambda_{1}=\lambda_{2}=1$ in our objective function. The success rates, BLEU scores, and the average number of words changed are reported in Table 5, with some adversarial examples shown in Table 7.

Analysis of Syntactic structure and Semantic Meaning Preservation

In our algorithm we aim to make adversarial examples having similar meaning to original examples by constraining the number of changed words and enforcing the changed words are close to the original words in the embedding space. However, depending on the implemented word embedding techniques, in general there is no guarantee that every word pair close in the embedding space have similar meanings. Therefore, we have conducted additional experiments to verify the syntactic and semantic quality of our generated adversarial examples. For syntactic structure part, as showed in Table 6, we measure the perplexity of generated adversarial sentences in DUC2003 and DUC2004 dataset. It shows that our examples keeps the original syntactic structure. For the semantic meaning part, We use DeepAI’s online sentiment analysis API to test whether our attack changes the sentiment of 500 sentences from DUC2003 dataset in summarization task. The results show that only 2.2% of adversarial examples have semantic meaning differ from the original sentences. It proves that almost all adversarial examples keep the same semantic classification unchanged.

Analysis and Discussions

As shown in Table 9, our targeted keyword attack wouldn’t just directly replace the keyword with some word in the source input. However, the word changed in the adversarial example and the target keyword are co-occurrent in the training dataset. It infers that seq2seq model learns the relationship between changed word and target keyword. However, the model fails to decide where it should focus on, which is strongly related with attention layer used in the model. It encourages us to use self-attention such as transformer (?) instead to extract all the attentions between any two words.When attacking subword transformer model, the target 1 keyword attack has 17% lower success rate and 0.13 lower BLEU score. It shows transformer model has a greater adversarial robustness.

Although our algorithm can achieve very good success rates ($84\%-100\%$) in both non-overlapping and targeted keywords attacks with 1 or 2 keywords, we also recognize some strengths of the seq2seq model: (i) unlike CNN models where targeted attack can be conducted easily with almost 100% success rate and very small distortion that cannot be perceived by human eyes (?), it is harder to turn the entire seq2seq output into a particular sentence – some sentences are even impossible to generate by seq2seq models; and (ii) since the input space of seq2seq is discrete, it is easier for human to detect the differences between the adversarial sequence and the original one, even if we only change one or few words. Therefore, we conclude that, compared with the DNN models designed for other tasks such as image classification, seq2seq models are more robust to adversarial attacks. The main reason, as pointed out in the introduction, is that the seq2seq model has a finite and discrete input space and almost infinite output space, so it is more robust than visual classification models that have an infinite and continuous input space and a very small output space (e.g., 10 categories in MNIST and 1,000 categories in ImageNet).

Conclusion

In this paper, we propose a novel framework, i.e., Seq2Sick, to generate adversarial examples for sequence-to-sequence neural network models. We propose a projected gradient method to address the issue of discrete input space, adopt group lasso to enforce the sparsity of the distortion, and develop a regularization technique to further improve the success rate. Besides, different from most existing algorithms that are designed for untargeted attack and classification tasks, our algorithm can perform the more challenging targeted keywords attack. Our experimental results show that the proposed framework is powerful and effective: it can achieve high success rates in both non-overlapping and targeted keywords attacks with relatively small distortions and preserve similar sentiment classification results for the most of the generated adversarial examples.

References