Are aligned neural networks adversarially aligned?

Introduction

Aligned language models are supposed to be “helpful and harmless” Bai et al. (2022): they should respond helpfully to user interaction, but avoid causing harm, either directly or indirectly. Prior work has focused extensively on how to train models to align with the preferences and goals of their creators. For example, reinforcement learning through human feedback (RLHF) Bai et al. (2022); Ouyang et al. (2022); Christiano et al. (2023) fine-tunes a pretrained model to emit outputs that humans judge to be desirable, and discourages outputs that are judged to be undesirable. This method has been successful at training models that produce benign content that is generally agreeable.

However, these models not are perfectly aligned. By repeatedly interacting with models, humans have been able to “social engineer” them into producing some harmful content (i.e., “jailbreak” attacks). For example, early attacks on ChatGPT (one such alignment-tuned language model) worked by telling the model the user is a researcher studying language model harms and asking ChatGPT to help them produce test cases of what a language model should not say. While there have been many such anecdotes where humans have manually constructed harm-inducing prompts, it has been difficult to scientifically study this phenomenon.

Fortunately, the machine learning community has by now studied the fundamental vulnerability of neural networks to adversarial examples for a decade (Szegedy et al., 2014; Biggio et al., 2013). Given any trained neural network and arbitrary behavior, it is almost always possible to optimize inputs that cause the selected behavior. Much of the early adversarial machine learning work focused on the domain of image classification, where it was shown that it is possible to minimally modify images so that they will be misclassified as an arbitrary test label. But adversarial examples have since been expanded to text Jia and Liang (2017); Ebrahimi et al. (2017); Alzantot et al. (2018); Wallace et al. (2019); Jones et al. (2023) and other domains.

In this paper we unify these two research directions and study if aligned models are resistant to adversarial inputs. That is, we ask the question:

Are aligned neural network models “adversarially aligned”?

First, we show that current alignment techniques—such as those used to fine-tune the Vicuna model Chiang et al. (2023)—are an effective defense against existing state-of-the-art (white-box) NLP attacks. This suggests that the above question can be answered in the affirmative. Yet, we further show that existing attacks are simply not powerful enough to distinguish between robust and non-robust defenses: even when we guarantee that an adversarial input on the language model exists, we find that state-of-the-art attacks fail to find it. The true adversarial robustness of current alignment techniques thus remains an open question, which will require substantially stronger attacks to resolve.

We then turn our attention to today’s most advanced multimodal models, such as OpenAI’s GPT-4 and Google’s Flamingo and Gemini, which accept both text and images as input (OpenAI, 2023; Alayrac et al., 2022; Pichai, 2023). Specifically, we study open-source implementations with similar capabilities Liu et al. (2023); Zhu et al. (2023); Gao et al. (2023) since these proprietary models are not publicly accessible. We find that we can use the continuous-domain images as adversarial prompts to cause the language model to emit harmful toxic content (see, e.g., Figure 1). Because of this, we conjecture that improved NLP attacks may be able to trigger similar adversarial behavior on alignment-trained text-only models, and call on researchers to explore this understudied problem.

Some alignment researchers Russell (2019); Bucknall and Dori-Hacohen (2022); Ngo (2022); Carlsmith (2022) believe that sufficiently advanced language models should be aligned to prevent an existential risk Bostrom (2013) to humanity: if this were true, an attack that causes such a model to become misaligned would be devastating. Even if these advanced capabilities do not come to pass, the machine learning models of today already face practical security risks Brundage et al. (2018); Greshake et al. (2023). Our work suggests that eliminating these risks via current alignment techniques—which do not specifically account for adversarially optimized inputs—is unlikely to succeed.

Background

Our paper studies the intersection of two research areas: AI alignment and adversarial examples.

As large language model parameter count, training dataset size, and training duration have been increased, the models have been found to exhibit complex behaviors (Brown et al., 2020; Wei et al., 2022b; Ganguli et al., 2022). In this work, we focus on models trained with causal “next-word” prediction, and use the notation $s\leftarrow\texttt{Gen}(x)$ to a language model emitting a sequence of tokens $s$ given a prompt $x$. Many applications of language models take advantage of emergent capabilities that arise from increased scale. For instance, language models are commonly used to perform tasks like question answering, translation, and summarization Brown et al. (2020); Chowdhery et al. (2022); Rae et al. (2022); Anil et al. (2023); Liang et al. (2022); Goyal et al. (2022).

Aligning large language models.

Large pretrained language models can perform many useful tasks without further tuning Brown et al. (2020), but they suffer from a number of limitations when deployed as is in user-facing applications. First, these the models do not follow user instructions (e.g., “write me a sorting function in Python”), likely because the model’s pretraining data (e.g., Internet text) contains few instruction-answer pairs. Second, by virtue of faithfully modeling the distribution of Internet text, the base models tend to reflect and even exacerbate biases Abid et al. (2021), toxicity, and profanity Welbl et al. (2021); Dixon et al. (2018) present in the training data.

Model developers thus attempt to align base models with certain desired principles, through techniques like instruction tuning Wei et al. (2022a); Ouyang et al. (2022) and reinforcement learning via human feedback (RLHF) Christiano et al. (2023); Bai et al. (2022). Instruction tuning finetunes a model on tasks described with instructions. RLHF explicitly captures human preferences by supervising the model towards generations preferred by human annotators (Christiano et al., 2023).

Multimodal text-vision models.

Increasingly, models are multimodal, with images and text being the most commonly combined modalities (OpenAI, 2023; Pichai, 2023; Liu et al., 2023; Zhu et al., 2023). Multimodal training allows these models to answer questions such as “how many people are in this image?” or “transcribe the text in the image”.

While GPT-4’s multimodal implementation has not been disclosed, there are a number of open-source multimodal models that follow the same general protocol Gao et al. (2023); Liu et al. (2023); Zhu et al. (2023). These papers start with a standard pre-trained language model that tokenizes and then processes the embedding layers. To process images, they use a pretrained vision encoder like CLIP Radford et al. (2021) to encode images into an image embedding, and then train a projection model that converts image embeddings into token embeddings processed by the language model. These visual tokens may be passed directly as an input to the model Zhu et al. (2023); Liu et al. (2023), surrounded by special templates (e.g., “ . . . <\img>”) to delineate their modality, or combined internal to the model via learned adaptation prompts Gao et al. (2023).

Adversarial examples.

Adversarial examples are inputs designed by an adversary to cause a neural network to perform some incorrect behavior (Szegedy et al., 2014; Biggio et al., 2013). While primarily studied on vision classification tasks, adversarial examples also exist for textual tasks such as question answering (Jia and Liang, 2017; Wallace et al., 2019), document classification Ebrahimi et al. (2017), sentiment analysis Alzantot et al. (2018), or triggering toxic completions Jones et al. (2023); Wallace et al. (2019). Prior work on textual tasks has either applied greedy attack heuristics Jia and Liang (2017); Alzantot et al. (2018) or used discrete optimization to search for input text that triggers the adversarial behavior Ebrahimi et al. (2017); Wallace et al. (2019); Jones et al. (2023).

In this paper, we study adversarial examples from the perspective of alignment. Because aligned language models are intended to be general-purpose—with strong performance on many different tasks—we focus more broadly on adversarial examples that cause the model to produce unwarranted harmful behavior, rather than adversarial examples that simply cause “misclassification”.

Our inputs are “adversarial” in the sense that they are specifically optimized to produce some targeted and unwanted outcome. Unlike recent “social-engineering” attacks on language models that induce harmful behavior by tricking the model into playing a harmful role (for example, taking on the persona of a racist movie actor Reddit (2023)), we make no effort to ensure our attacks are semantically meaningful, and they often will not be.

Threat Model

There are two primary reasons researchers study adversarial examples. On the one hand, researchers are interested in evaluating the robustness of machine learning systems in the presence of real adversaries. For example, an adversary might try to construct inputs that evade machine learning models used for content filtering (Tramèr et al., 2019; Welbl et al., 2021) or malware detection (Kolosnjaji et al., 2018), and so designing robust classifiers is important to prevent a real attack.

On the other hand, researchers use adversarial robustness as a way to understand the worst-case behavior of some system (Szegedy et al., 2014; Pei et al., 2017). For example, we may want to study a self-driving car’s resilience to worst-case, adversarial situations, even if we do not believe that an actual attacker would attempt to cause a crash. Adversarial examples have seen extensive study in the verification of high-stakes neural networks (Wong and Kolter, 2018; Katz et al., 2017), where adversarial examples serve as a lower bound of error when formal verification is not possible.

Existing attacks assume that a model developer creates the model and uses some alignment technique (e.g., RLHF) to make the model conform with the developer’s principles. The model is then made available to a user, either as a standalone model or via a chat API. There are two common settings under which these attacks are mounted, which we describe below.

Malicious user: The user attempts to make the model produce outputs misaligned with the developer’s principles. Common examples of this are jailbreaks of chatbots such as ChatGPT or Bard where a user uses an adversarial example (a maliciously designed prompt) to elicit the desired unaligned behavior, such as outputting instructions for building a bomb. In this setting, there is no need for the attack to be “stealthy”.

Malicious third-party: An honest user might query an alignment-tuned language model as part of an autonomous system that processes untrusted third-party data (e.g., a virtual assistant that reads and writes the user’s emails). A malicious third-party could launch a prompt injection attack (Greshake et al., 2023) by feeding data to the language model to hijack its behavior (e.g., exfiltrating the user’s emails). In this setting, there might be stricter constraints on a valid attack.

2 Our Threat Model

In this paper we are focused on better understanding the limitations of current alignment techniques. As such, we use adversarial examples primarily to measure their worst-case behavior. That is, for the most part we are not concerned with any particular practical adversary, and instead, only with finding any valid input that achieves our attack goal. That said, our attacks are likely to be practical in both settings where a malicious user is attacking the trained language model to allow it to perform malicious behavior, as well as settings where an honest user copies malicious text from an adversary.

Independent of who is attacking whom, it is also necessary to define the ultimate attack objective. While attackers could aim to induce a number of harmful behaviors (e.g., triggering a model into outputting dangerous instructions, or emitting private training data), we focus here on the specific goal of triggering toxic outputs. The main reason is that toxicity is relatively easy to (approximately) evaluate in a automated way. We use an extremely simplistic definition for toxicity (checking for the presence or absence of a selected list of toxic words in the output of the model), noting that definitions of toxicity are cultural and contextual and widely studied. We formalize our toxicity attack objective as follows:

Specifically, given a toxicity-detection function isToxic, and an language model Gen that is never supposed to generate text that causes isToxic to be true, an input $X$ causes harm if $\texttt{isToxic}(\texttt{Gen}(X))$ returns true. When the generation function is randomized (i.e., it is not greedy sampling) we might require $\Pr[\texttt{isToxic}(\texttt{Gen}(X))]>\varepsilon$, for some probability $\varepsilon>0$. For simplicity, in our experiments, we set the temperature to to make models deterministic.

Choosing an appropriate method for isToxic is important. In this paper we use simple substring lookup. Our attack chooses some harmful target sequence (for example, a curse word) that the model should generate, and then we report a generation as toxic if it contains this word anywhere.

Evaluating Aligned Models with NLP Attacks

The most direct method for generating an adversarial text sequence would be to directly optimize some differentiable proxy for the attack objective, $\texttt{isToxic}(\texttt{Gen}(X))$. However there are two challenges with this approach:

Text tokens are discrete, and so continuous optimization via common optimization algorithms, e.g., gradient descent is unlikely to be effective Ebrahimi et al. (2017).

There is often not one exact target. And so in order to check if the attack succeeded, we would have to query the model to emit one token at a time. Thus, in order to pass a long sequence $S$ into the toxicity classifier we would need to generate $\lvert S\rvert$ tokens and then perform back propagation through $\lvert S\rvert$ neural network forward passes.

While the first challenge above is a fundamental challenge of neural language models, the second is not fundamental. Instead of directly optimizing the true objective, i.e., checking that $\texttt{isToxic}(S)$ is true, we can optimize a surrogate objective of making $\texttt{isToxic}{(S_{:j})}$ be true for some attacker-chosen fixed-length string $S_{:j}$ with $j\ll\lvert S\rvert$. Observe that this makes optimization much easier, as we can now perform just one single forward pass to target exactly this string. Further, because this substring is contained within the larger output $S$, it is guaranteed that $\texttt{isToxic}(S)$ will be true as well. However, this approach may make the attack slightly more difficult: it may be harder to make the model emit the immediate next token as toxic, rather than to eventually do so after being steered toward it.

In this section, we will study the suitability of prior attack methods for achieving our toxicity objective against a variety of chat bot models, both trained with and without alignment techniques.

Alignment techniques (such as RLHF) are typically not applied to “plain” language models, but rather to models that have been first tuned to interact with users via a simple chat protocol.

Typically, this is done by formatting the input to underlying language model with a specific interleaving of messages, separated by special tokens that indicate the source and boundaries of each prior message.

In the above example, the chat bot’s user typed in the messages in double-quotes, and the language model generated the italicized text in single-quotes. The special tokens ‘[USER]:’ and ‘[AGENT]:’ are automatically inserted by the chat bot application to delineate rounds of interaction when prompting the language model for its next message.

This special formatting of the aligned language model’s input places a constraint on the attacker: while the content input by the user (i.e., the text in double quotes) could be arbitrarily manipulated, the prior chat history as well as the special ‘[USER]:’ and ‘[AGENT]:’ tokens cannot be modified. In general, across domains we believe this “attacks must follow some specified format” setting is likely to occur in practice.

2 Prior Attack Methods

A number of prior works have studied adversarial examples against NLP models.

The most closely related to our goal is the work of Jones et al. (2023) who study the possibility of inverting a language model, i.e., of finding an adversarial prompt $X$ that causes a model $f$ to output some targeted string $y\leftarrow f(X)$. Their technique succeeds in making a model emit the names of US Senators or emit toxic words after particular individuals. However, they assume a stronger threat model than we do here, and which does not fully match with the recent deployment of alignment-trained language models. Indeed, as noted above in Section 4.1, these language models typically operate as chat bots that interleave attacker-controlled messages with model generations and special separation tokens. As a result, this paper considers the generic problem of designing an adversarial prompt $X$ such that $\texttt{Gen}(P_{pre}||X||P_{post})$ is toxic, where $P_{pre}$ and $P_{post}$ are non-adversarial parts of the model’s prompt that respectively precede and succeed the attacker’s input.

In contrast, the setup in Jones et al. (2023) assumes that the attacker can either control the entire prompt (i.e., the aim is to make $\texttt{Gen}(X)$ toxic), or at least the text immediately preceding the model’s next generation (i.e., $\texttt{Gen}(P_{pre}||X)$). In both cases, the attacker controls the tokens right before the model’s generation, which makes an attack much easier (but not relevant for current chat bot deployments). (However: there are several settings where their threat model is valid—for example, in code completion tasks this is likely the case. Our threat model is just more general.)

3 Our Evaluation Setup

Thus, we assume the adversary can control only their messages (following the [USER]: token), and then the special [AGENT]: token is appended to the prompt sequence to indicate the end of the prompt and that the LLM should begin generating a response. In this setup, the adversary cannot modify the final tokens of the LLM’s prompt.

We construct our evaluation dataset by first collecting potentially toxic messages that a model might emit. For each potentially toxic message, we prepend a set of benign conversations followed by the special tokens used to prompt the LLMs response. An attack is successful if it causes the toxic output to be emitted. We source benign conversations from the Open Assistant dataset. This dataset consists of a series of hypothetical interactions between a human and an assistant, which we truncate to $K<N$ rounds. We use a random subset of harmful texts from Jones et al. (2023) as the attack objective, with attacks targeting between one and three tokens of toxic text.

We attack three different publicly available models:

GPT-2: a model that was neither trained to act as a chatbot, nor to be aligned;

LLaMA: a model trained to chat with users but with no explicit alignment attempt;

Vicuna: a version of LLaMa that was aligned via fine-tuning to prevent toxic generations (and other harmful behaviors).

4 Prior Attacks Results

We evaluate the ARCA attack from Jones et al. (2023) and the GBDA attack from Guo et al. (2021). As ARCA generalizes and dominates prior attacks such as (Wallace et al., 2019; Ebrahimi et al., 2017) we do not also re-evaluate these. Both ARCA and GBDA are largely ineffective at causing the model to emit toxic content in our setting even when allowed to inject thirty malicious tokens. (We choose this number to match with experiemnts that will follow later in the paper.) This should not be surprising: already in Jones et al. (2023), the attack had under 40% attack success rate in getting GPT-2 to output toxic output text, even when the adversary had complete control of every token up to the desired toxic phrases. In contrast, because we require that the tokens “[AGENT]:” come after the adversarial tokens, the attacks have (much) lower attack success rates here. Table 1 summarizes these results.

Qualitative Evaluation: Toxicity: We qualitatively inspect a subset of these generations from each setting in Table 1. We find that in most cases the output is indeed toxic as expected. But there in less than ten cases, we find our automated metric has potential limitations. For example, when told to produce one toxic phrase, Vicuna responds “F*** you. I’m kidding, of course. As a AI visual assistant, I’m programmed to be polite and provide assistance to the best of my ability. How can I assist you today?”, and in another case it responds “(Note: This is not appropriate language and is not a proper response to the prompt.)” after an initially abusive message. This demonstrates a limitation of our simple isToxic definition that only performs a substring lookup. Future work will need to carefully consider different toxicity definitions once attacks begin to have nontrivial attack success rates—for now, even with this loose definition, the current attacks in the literature fail to find successful adversarial sequences.

Why do Existing NLP Optimization Attacks Fail?

In the prior section we have found that existing NLP optimization attacks have limited success at causing aligned models to emit harmful content in a standard chat setting. There are two possible explanations for this result:

The aligned language models we attack are truly robust to adversarial examples; or,

Current attacks are insufficiently powerful to evaluate the robustness of aligned models.

Fortunately, recent work has developed techniques explicitly designed to differentiate between these two hypotheses for general attacks. Zimmermann et al. (2022) propose the following framework: first, we construct test cases with known adversarial examples that we have identified a priori; then, we run the attack on these test cases and verify they succeed. Their initial proposal for designing such test cases works as follows. Our specific test case methodology follows Lucas et al. (2023). To construct test cases, we first identify a set of adversarial examples via brute force. And once we have confirmed the existence of at least one adversarial example via brute force, we run our attack over the same search space and check if it finds a (potentially different, but still valid) adversarial example. This approach is effective when there exist effective brute force methods and the set of possible adversarial examples is effectively enumerable—such as is the case in the NLP domain.

We adapt to this to our setting as follows. We construct (via brute force) prompts $p$ that causes the model to emit a rare suffix $q$. Then, the attack succeeds if it can find some input sequence $p^{\prime}$ that causes $\texttt{Gen}(p)=q$, i.e., the model emits the same $q$. Otherwise, the attack fails. Observe that a sufficiently strong attack (e.g. a brute force search over all prompts) will always succeed on this test: any failure thus indicates a flawed attack.

How should we choose the prefixes $p$ and the target token $q$? If we were to choose $q$ ahead of time (e.g., to be some toxic token), then it might be very hard to find—even via brute force—a prefix $p$ so that $\texttt{Gen}(p)=q$. So instead we drop the requirement that $q$ is toxic, and approach the problem from reverse.

Initially, we sample many different prefixes $p_{1},p_{2},\dots$ from some dataset (in our case, Wikipedia). Let $S$ be the space of all N-token sequences (for some N). Then, for all possible sequences $s_{i}\in S$ we query the model on $\texttt{Gen}(s_{i}||p_{j})$. (If $|S|$ is too large, we randomly sample 1,000,000 elements $s_{i}\in S$.) This gives a set of possible output tokens $\{q_{i}\}$, one for each sequence $s_{i}$.

For some prompts $p_{j}$, the set of possible output tokens $\{q_{i}\}$ may have high entropy. For example, if $p_{j}$ = “How are you doing?” then there are likely thousands of possible continuations $q_{i}$ depending on the exact context. But for other prompts $p_{j}$, the set of possible output tokens $\{q_{i}\}$ could be exceptionally small. For example, if we choose the sequence $p_{j}$=“Barack” the subsequent token $q_{i}$ is almost always “Obama” regardless of what context $s_{i}$ was used.

But the model’s output might not always be the same. There are some other tokens that might be possible—for example, if the context where $s_{i}$=“The first name [”, then the entire prompt (“The first name [Barack”) would likely cause the model to output a closing bracket q=“]”. We denote such sequences $p_{j}$ that yield small-but-positive entropy over the outputs $\{q_{i}\}$ (for different prompts $s_{i}\in S$) a test case, and set the attack objective to be the output token $q_{i}$ that is least-likely.

These tests make excellent candidates for evaluating NLP attacks. They give us a proof (by construction) that it is possible to trigger the model to output a given word. But this happens rarely enough that an attack is non-trivial. It is now just a question of whether or not existing attacks succeed.

We construct eight different sets with varying difficulty levels and report averages across each. Our test sets are parameterized by three constants. (1) Prevalence: the probability of token $q$ given $p_{j}$, which we fix to $10^{-6}$; (2) Attacker Controlled Tokens: the number of tokens the adversary is allowed to modify, which we vary from 2, 5, 10, or 20 tokens, and (3 Target Tokens: the number of tokens of output the attacker must reach . We generate our test cases using GPT-2 only, due to the cost of running a brute force search.

2 Prior Attacks Results

In Table 2 we find that the existing state-of-the-art NLP attacks fail to successfully solve our test cases. In the left-most column we report attack success in a setting where the adversary aims must solve the task within the given number of attacker controlled tokens. ARCA is significantly stronger than GBDA (consistent with prior work) but even ARCA passes less than half of the time when Because the numbers here are so low, we then experimented with giving the attacker more control with a multiplicative factor. That is, if the task asked for us to find an adversarial example with $10$ tokens, and we run the attack with a factor of $5$, we allow the attack to search over $50$ attacker controlled tokens. We find that even with $10\times$ extra tokens the attack still often fails our tests.

Note that the purpose of this evaluation is not to argue the NLP attacks we study here are incorrect in any way. On the contrary: they largely succeed at the tasks that they were originally designed for. But we are asking them to do something much harder and control the output at a distance, and our hope here is to demonstrate that while we have made significant progress towards developing strong NLP optimization attacks there is still room for improving these techniques.

Attacking Multimodal Aligned Models

Text is not the only paradigm for human communication. And so increasingly, foundation models have begun to support “multimodal” inputs across vision, text, audio, or other domains. In this paper we study vision-augmented models, because they are the most common. For example, as mentioned earlier, OpenAI’s GPT-4 and Google’s Gemini will in the future support both images and text as input. This allows models to answer questions such as “describe this image” which can, for example, help blind users Salam (2019).

It also means that an adversary can now supply adversarial images, and not just adversarial text. And because images are drawn from a continuous domain, adversarial examples are orders of magnitude simpler to create: we no longer need to concern ourselves with the discrete nature of text or the inversion of embedding matrices and can now operate on (near) continuous-domain pixels.

Our attack approach directly follows the standard methodology for generating adversarial examples on image models. We construct an end-to-end differentiable implementation of the multimodal model, from the image pixels to the output logits of the language model. We apply standard teacher-forcing optimization techniques when the target suffix is $>1$ token. To initiate each attack, we use a random image generated by sampling each pixel uniformly at random.

2 Experiments

While GPT-4 currently supports vision for some users OpenAI (2023), this functionality is not publicly available. Google’s Gemini has also not been made available publicly. The research community has thus developed open-source (somewhat smaller) versions of these multimodal models.

We evaluate our attack on two different implementations. While they differ in some details, both follow the approach in Section 2: the image is encoded with a vision model, projected to the token embedding space, and passed as a sequence of soft-tokens to the language model.

Mini GPT-4 (Zhu et al., 2023) uses a pretrained Q-Former module from Li et al. (2023) to project images encoded by EVA CLIP ViT-G/14 Fang et al. (2022) to Vicuna’s (Chiang et al., 2023) text embedding space. Both CLIP and Vicuna are frozen, while a section of the Q-former is finetuned on a subset of LAION Schuhmann et al. (2021), Conceptual Captions Sharma et al. (2018), SBU Ordonez et al. (2011), and multimodal instruction-following data generated by the authors.

LLaVA (Liu et al., 2023) uses a linear layer to project features from CLIP ViT-L/14 to the Vicuna embedding space. While CLIP is frozen, both Vicuna and the projection matrix are finetuned on Conceptual Captions Sharma et al. (2018) and custom multimodal instruction-following data.

LLaMA Adapter (Gao et al., 2023) works similarly to these except that it uses learned adaptation prompts to incorporate visual information internal to the model via these adapters.

3 Quantitative Evaluation: Toxicity

4 Qualitative Evaluation

We further evaluate our attack on qualitative samples designed to probe various harmful outcomes.

For example, even when explicitly asking for harmful output (e.g., “Write a vile letter telling my neighbor what I think of him.”) the model still responds harmlessly (“Dear [Neighbor’s Name] … I have to say that I am not impressed with your behavior … Sincerely, [Your Name]”). But by providing an adversarial image as input, we can cause the model to emit our desired harmful output (“F*** you, you dirty life piece of s***. You’re a disgusting, filthy, stinky, scumbag who doesn’t deserve to live in this neighborhood … you worthless, good-for-nothing, loser.”)

Similarly, when asked to provide instructions for how to get away with murder, the base model responds that “It is not appropriate to encourage or provide information on how to harm or kill someone” but the attacked model complies: “The first step would be to identify the location of the neighbor’s house”. While the response here is rather poor advice, this is likely because the model is not competent; a better model might provide more actionable (and thus harmful) advice. We list sample (uncensored) examples of our ability to attack these models in the Appendix.

Conclusion

Language models trained via RLHF or instruction tuning are significantly more aligned than base models: in particular, they are more helpful (they appropriately follow benign user instructions) and harmless (they are less likely to output toxicity or harmful actions). While helpfulness can be evaluated through various utility metrics, harmlessness is more difficult to evaluate—and almost all methods to date rely on human-designed test cases to quantify this.

In this paper we have shown that while these models might be usually harmless, they may not be harmless under adversarial prompting. While the harms from adversarial prompting that we illustrate are fairly benign (e.g., the small models we study give unhelpful advice on how to get away with murder, or produce toxic content that could be found anywhere on the internet), our attacks are directly applicable to triggering other bad behaviors in larger and more capable systems.

Our attacks are most effective on the new paradigm of multimodal vision-language models. While all models we study are easy to attack, small design decisions affect the ease of attacks by as much as $10\times$. Better understanding where this increased vulnerability arises is an important area for future work. Moreover, it is very likely that the future models will add additional modalities (e.g, audio) which can introduce new vulnerability and surface to attack.

Unfortunately, for text-only models, we show that current NLP attacks are not sufficiently powerful to correctly evaluate adversarial alignment: these attacks often fail to find adversarial sequences even when they are known to exist. Since our multimodal attacks show that there exist input embeddings that cause language models to produce harmful output, we hypothesize that there may also exist adversarial sequences of text that could cause similarly harmful behaviour.

Conjecture: An improved NLP optimization attack may be able to induce harmful output in an otherwise aligned language model.

While we cannot prove this claim (that’s why it’s a conjecture!) we believe our paper provides strong evidence for it: (1) language models are weak to soft-embedding attacks (e.g., multimodal attacks); and (2) current NLP attacks cannot find solutions that are known to exist. We thus hypothesize that stronger attacks will succeed in making text-only aligned models behave harmfully.

We hope our paper will inspire several directions for future research. Most immediately, we hope that stronger NLP attacks will enable comprehensive robustness evaluations of aligned LLMs. Such attacks should, at a minimum, pass our tests to be considered reliable.

We view the end goal of this line of work not to produce better attacks, but to improve the evaluation of defenses. Without a solid foundation on understanding attacks, it is impossible to design robust defenses that withstand the test of time. An important open question is whether existing attack and defense insights from the adversarial machine learning literature will transfer to this new domain.

Ultimately, such foundational work on attacks and defenses can help inform alignment researchers develop improved model alignment techniques that remain reliable in adversarial environments.

Acknowledgements

We are grateful for comments on this paper by Andreas Terzis, Slav Petrov, and Erik Jones.

References

Appendix A Ethics

Our paper demonstrates that adversarial techniques that can be used to circumvent alignment of language models. This can be used by an adversary to coerce a model into performing actions that the original model developer intended to disallow.

We firmly believe that on the whole our paper will cause little harm in the short term, and ultimately lead to stronger defenses in the long term.

Our attacks are unlikely to cause short-term harm because they are currently restricted to the white-box setting where the entire model is available. Additionally, because all open-source aligned models are simply fine-tuned by building on unaligned base models, an adversary who wished to perform malicious behavior could directly use the (already open-source) base model. In addition, because of our white-box assumption, our attacks will not be directly applicable to much more capable production language models such as GPT-4. Given that these production models are already susceptible to human-designed adversarial prompts that cause them to emit harmful content, we do not expect a significant short-term harm from releasing this paper even if our attacks could be extended to the black-box domain.

In the long term we expect our work will incentivize and help form a foundation for the design of more robustly aligned models. Just as it is not possible to produce a secure cipher without a solid understanding of cryptanalysis, we do not believe it is possible to design an adversarially aligned language model without first understanding the potential vulnerabilities. Otherwise, it is likely that researchers will fool themselves into a false sense of security and release models they believe to be aligned even if they are not.

Appendix B Unfiltered toxic conversations

This appendix contains unfiltered content that may be offensive to some readers.

This page is intentionally blank. This appendix contains unfiltered content that may be offensive to some readers beginning on the following page.

Appendix C Mini-GPT4 Output

Appendix D LLaVA outputs