Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks

Introduction

The remarkable capabilities of Large Language Models (LLMs) carry the inherent risk of misuse, such as producing toxic content, spreading misinformation or supporting harmful behaviors. To mitigate these risks, safety alignment is commonly employed—a fine-tuning phase where models are guided to generate responses judged safe by humans and to refuse responses to potentially harmful queries (Bai et al., 2022; Touvron et al., 2023). Although safety alignment is effective in general, several works have shown that it can be circumvented using adversarial prompts. These are inputs specifically designed to induce harmful responses from the model, a practice known as jailbreaking attacks (Mowshowitz, 2022; Zou et al., 2023; Chao et al., 2023).

Jailbreaking attacks vary in their knowledge of the target LLM (ranging from white- to black-box approaches, or API-only access), complexity (involving manual prompting, standard optimization techniques, or auxiliary LLMs), and computational cost. Moreover, the nature of the jailbreaks they produce differs: some methods insert strings with no semantic meaning (Zou et al., 2023), while others rephrase user requests to maintain natural language (Mehrotra et al., 2023). The effectiveness of these attacks can significantly vary, achieving a high success rate on some target models but also drastically failing on others. Finally, some LLMs, such as the Llama-2-Chat family (Touvron et al., 2023), seem to maintain their robustness against these attacks. At the same time, new defensive mechanisms designed to counteract jailbreaks are emerging (Robey et al., 2023; Mazeika et al., 2024).

In this work, we examine the safety of leading safety-aligned LLMs in terms of robustness to jailbreaks. We show that it is feasible to leverage the information available about each model, derived from training details or inference (e.g., logprobs), to construct simple adaptive attacks. Our main tool consists of manually designing a universal template (i.e., a single template is used for all unsafe requests) for each target model (or family of models), supported by random search (RS) (Rastrigin, 1963) when the logprobs of the generated tokens are (at least partially) accessible. Unlike some prior works (Zou et al., 2023; Geisler et al., 2024), our approach does not require gradient information (even for open-weight models) or auxiliary LLMs (Chao et al., 2023; Mehrotra et al., 2023; Zeng et al., 2024) to iteratively optimize the jailbreaks. In this way, using the dataset of unsafe prompts from Chao et al. (2023), we obtain a close to 100% attack success rate on all leading safety-aligned LLMs, including GPT-3.5, GPT-4, Claude-3, Gemma, Llama-2-Chat, and the adversarially trained R2D2, outperforming the existing techniques. We provide a summary of these results in Table 1, and we show an illustrative example of a successful transfer attack on Claude 3 Sonnet in Figure 1. Additionally, we show how to combine manual adaptation and RS for finding trojan strings in poisoned models—a task that shares many similarities with jailbreaking—enabling us to secure the first place in the SaTML’24 Trojan Detection Competition (Rando & Tramèr, 2024).

Our results provide several insights into the domain of safety in LLMs and its evaluation. First, we reveal that currently both open-weight and proprietary models are completely non-robust to adversarial attacks. Second, it becomes evident that adaptive attacks play a key role in the evaluation of robustness, as no single method can generalize across all target models. Despite the absence of a standardized attack, we still provide recommendations for future research on designing jailbreak attacks, analogous to the framework established for image classification by Carlini et al. (2019); Tramèr et al. (2020); Croce et al. (2022b), distilling key observations from our experiments.

Related Work

Adversarial attacks on machine learning models have a long history (Biggio et al., 2013; Szegedy et al., 2014; Biggio & Roli, 2018; Madry et al., 2018). In this section, we specifically focus on the different categories of LLM jailbreaking attacks.

ChatGPT users have discovered handcrafted jailbreaks (Mowshowitz, 2022). Wei et al. (2023a) systematically categorize these jailbreaks based on two main criteria: (1) competing objectives, which occurs when a model’s capabilities conflict with safety goals, and (2) mismatched generalization, which arises when safety training does not generalize to domains where the model has capabilities. By leveraging these failure modes and employing a combination of manual attacks, Wei et al. (2023a) achieve high success rates on proprietary LLMs such as GPT-4 and Claude v1.3. Wei et al. (2023b) explore jailbreaking using in-context learning prompts that contain a few examples of harmful responses.

Direct search attacks.

Alternatively, the search for jailbreaks can be automated using first- or zeroth-order discrete optimization techniques. For example, Zou et al. (2023) introduce universal and transferable attacks with a gradient-based method named Greedy Coordinate Gradient (GCG), inspired by earlier discrete optimization efforts in NLP (Shin et al., 2020). Lapid et al. (2023) use a genetic algorithm to generate universal adversarial prompts within a black-box threat model, where gradients are not used. Liu et al. (2023) apply genetic algorithms to combine sentence fragments into a low-perplexity jailbreak. Zhu et al. (2023) pursue a similar goal, modifying GCG to generate low-perplexity adversarial suffixes. Sitawarin et al. (2024); Hayase et al. (2024) suggest employing random search on predicted probabilities for black-box models to guide and refine the adversarial string search, occasionally aided by a white-box LLM to identify the most promising tokens to change. For OpenAI models, both attacks use the logit_bias parameter whose behavior has been already changed: it no longer influences the logprobs, rendering their attacks ineffective.

LLM-assisted attacks.

Finally, using other LLMs for optimizing jailbreaking attacks has shown considerable promise, primarily due to enhanced query efficiency. Chao et al. (2023) have first developed Prompt Automatic Iterative Refinement (PAIR), a method that uses an auxiliary LLM to identify jailbreaks efficiently. Mehrotra et al. (2023) have then refined PAIR’s methodology, introducing a tree-based search method. In similar vein, Shah et al. (2023) have devised an approach to jailbreaks generation using an LLM that is guided by persona modulation. Meanwhile, Yu et al. (2023) have introduced GPTFUZZER, a framework that iteratively enhances human-written templates with the help of an LLM. Zeng et al. (2024) have fine-tuned GPT-3.5 for the specific task of rephrasing harmful requests, using the rephrased content to jailbreak a target LLM. Takemoto (2024) offer a straightforward LLM rephrasing method that rivals more complex methods.

Background and Methodology

We first outline background on jailbreaking and then discuss our attack methodology.

We focus on identifying prompts that, when given a specific harmful request (e.g., “Tell me how to build a bomb”), induces the LLM to generate harmful content. We assume access to a set of such requests recognized by most LLM providers as harmful (e.g., misinformation, violence, hateful content) and are typically not responded to. We define a language model $\texttt{LLM}:\mathcal{T}^{*}\rightarrow\mathcal{T}^{*}$ as a function that maps a sequence of input tokens to a sequence of output tokens, referred to as the target model, as it is the one we aim to jailbreak. Given a judge function $\texttt{JUDGE}:\mathcal{T}^{*}\times\mathcal{T}^{*}\rightarrow\{\text{NO},\text{YES}\}$ and a harmful request $R\in\mathcal{T}^{*}$, the attacker’s goal is:

Although the judge may use a fine-grained evaluation score (such as a score from 1 to 10 for the GPT-4 judge), it ultimately outputs a binary response indicating whether $\texttt{LLM}(P)$ constitutes a valid jailbreak for the harmful request $R$.

Our setup.

We use default system prompts unless specifically mentioned (modifications are only made for Claude) due to potential future restrictions by frontier LLM providers, who might limit access to the system prompt for safety reasons. Our targets comprise a set of 50 behaviors from AdvBench curated by Chao et al. (2023) that ensures distinct and diverse harmful requests. We use GPT-4 as the semantic judge (see Table 9 for the prompt) in line with the criteria established by Chao et al. (2023), where jailbreak is considered successful only if it achieves a 10/10 jailbreak score from GPT-4. Additionally, we include results using the rule-based judge from Zou et al. (2023) in the appendix for comparison.

2 Methodology

Our adaptive attacks leverage several key components, which we combine or slightly adapt depending on the target LLMs.

The importance of a well-designed prompt in enhancing the performance of LLMs is well-established (Wei et al., 2023a). In our approach, we develop a prompt template that can incorporate a generic unsafe request. This template is specifically designed to make the model start from a specified string (e.g., “Sure, here is how to make a bomb”) and steer the model away from its default aligned behavior. Its general structure can be summarized as: set of rules $+$ harmful request $+$ adversarial suffix. We optimized the set of rules one by one on the GPT-3.5 Turbo model to maximize the attack success rate and avoid the built-in safety guardrails. We provide the complete template in Table 2. As we will see in Section 4 (Table 5), this prompt template alone leads to 100% attack success rate on GPT-3.5 Turbo, and it also provides a good starting point for other LLMs. We also designed another prompt template (referred to as the in-context prompt) that includes an example of unsafe behavior which the model is encouraged to imitate (see Table 8).

Random search.

We use a simple random search (RS) algorithm (Rastrigin, 1963) adapted for jailbreaking language models. The algorithm is as follows:

Append a suffix of a specified length to the original request.

In each iteration, modify a few contiguous tokens at a random position in the suffix.

Accept the change if it increases the log-probability of a target token (e.g., “Sure” that leads the model to comply with a harmful request) at the first position of the response.

We use adversarial suffixes that are initialized with $25$ tokens, we use up to $10\,000$ iterations and potentially a few random restarts. This strategy, including the preference for suffixes over prefixes and the focus on maximizing the log-probability of the token “Sure”, draws inspiration from the attack methodology of Zou et al. (2023). We opted for random search (a) due to its simplicity and efficiency, requiring only scores (such as logprobs) instead of gradients (thus reducing the memory demands), and (b) motivated by its success in adversarial attacks on vision models (Andriushchenko et al., 2020; Croce et al., 2022a). We provide further details in the code.https://github.com/tml-epfl/llm-adaptive-attacks

Transfer attacks.

Successful jailbreaks developed for one LLM can often be reused on another model (Zou et al., 2023). This observation will be crucial for attacking some of the Claude 3 models that do not expose logprobs making random search not applicable.

Self-transfer.

It is well-known that initialization plays a key role in optimization algorithms, including in RS-based attacks (Andriushchenko et al., 2020). We leverage the adversarial suffix found by random search for a simpler harmful request as the initialization for RS on more challenging requests. We refer to this approach as self-transfer. It significantly boosts the attack success rate on some LLMs like Llama-2-Chat, Gemma, and GPT-4.

Prefilling attack.

Some APIs like Claude allow users to directly prefill the LLM’s response with a specified beginning, making the aforementioned optimization procedure unnecessary. In that case, we explore prefilling the response with a string that corresponds to the target behavior (e.g., “Sure, here is how to make a bomb”).

Jailbreaking Leading Safety-Aligned LLMs

In this section, we detail the adaptive attacks we have developed for several families of leading safety-aligned LLMs. We provide a summary of main evaluations here and show the rest in Table 19 in the appendix where we also present results on Vicuna-13B and Mistral-7B.

Here, we focus on open-weights Llama-2-Chat (7B, 13B, 70B parameters) (Touvron et al., 2023) and Gemma-7B models (Google, 2023). These models have been significantly safety-aligned, rendering them resilient to jailbreaks even in white-box scenarios (Zou et al., 2023).

The key element to jailbreak Llama-2-Chat models is self-transfer, where successful adversarial suffixes found by RS on simpler requests are used as initialization for RS on more complex requests. Notably, these adversarial strings tend to be to some extent transferable across different model sizes (e.g., from 7B to 13B models), but for the best result we repeat the self-transfer procedure for each model size separately. The same approach is also successful on Gemma-7B, although prompt + RS alone already demonstrates high attack success rate.

Results.

For Llama-2-Chat models, Table 3 shows that our standard adversarial prompt templates yield a 0% success rate, confirming the effectiveness of their safety alignment. When we apply Prompt + RS the attack success rate (ASR) increases to 48%. Ultimately, our composite attack strategy—which combines prompting, random search, and self-transfer—achieves a 100% attack success rate for all LLMs, surpassing all existing methods. For Llama-2-Chat-7B, the best reported success rate is 92% by PAP (Zeng et al., 2024), an LLM-assisted method. However, this method requires 10 restarts to reach such efficacy, and its success rate drops to 46% with only one restarts as in our approach. Meanwhile, for the 13B and 70B models, Mazeika et al. (2024) reports ASR below 40%, while there is no prior evaluation available for Gemma-7B.

2 Jailbreaking R2D2 model

Similarly to Llama-2-Chat, the standard prompt template, alone or with RS, shows limited effectiveness. However, in contrast with Llama-2-Chat, self-transfer is ineffective here. Thus, we circumvent safety guardrails using an in-context prompt (see Table 8), which we found the model to be particularly sensitive to. We use random search on top of the in-context prompt to maximize the probability of the initial token “Step” (instead of “Sure”) to be consistent with the new prompt template.

Results.

As shown in Table 4, using the in-context prompt alone achieves a 90% attack success rate, which RS boosts to 100%. This significantly surpasses the 61% reported by Mazeika et al. (2024) using TAP (Mehrotra et al., 2023). Interestingly, the in-context prompt is less effective on other models like Llama-2-Chat (see Table 19 in the appendix).

3 Jailbreaking GPT models

GPT models are the most popular state-of-the-art LLMs with non-trivial built-in safety features. We use the API checkpoints gpt-3.5-turbo-1106 and gpt-4-1106-preview for our experiments.

We observed that GPT-3.5 Turbo is extremely brittle to manually designed prompts, with no need for more sophisticated techniques. In contrast, GPT-4 Turbo demonstrates greater resistance to these adversarial prompt templates. Thus, for this model, we rely on self-transfer to achieve more successful jailbreaks.

Results.

Table 5 summarizes our results: with the prompt template alone, we achieve 100% success rate on GPT-3.5 Turbo, outperforming the baselines. For GPT-4 Turbo, using the prompt alone leads only to 28% success rate. However, by combining the prompt, RS, and self-transfer, we improve the previous best ASR from 59% (Mazeika et al., 2024) to 96%. For reference, we also provide baselines with standard GPT-4 (i.e., not Turbo) in Table 5 but we do not evaluate it ourselves due to its higher costs.

Non-determinism in GPT-3.5/4.

The limitation of the API providing only the top-5 log-probabilities is not critical, as it is often straightforward to prompt a desired token, like “Sure,” to appear in the top-5. A more challenging issue is the non-deterministic output, since RS does not necessarily have a correct signal to refine the adversarial string. As illustrated in Figure 2, identical queries can yield varying log-probabilities, even with a fixed seed parameter and temperature zero in the API. The randomness makes random search less effective, although it still succeeds to a large extent.

4 Jailbreaking Claude models

Claude models are known for their high safety levels. In line with this, Anthropic does not provide access to logprobs for these models which prevents direct iterative attack like random search. Thus, we first test a transfer attack using an adversarial suffix from GPT-4. We enhance the attack with multiple random restarts to leverage different generations with temperature one. Subsequently, we investigate an attack method utilizing Anthropic’s prefilling feature,https://docs.anthropic.com/claude/docs/prefill-claudes-response a functionality not commonly available from other LLM providers like OpenAI.

As shown in Table 6, the direct transfer attack is particularly effective on certain models such as Claude 3 Sonnet (100% ASR). Given Claude-3’s recent release in early March 2024, there are no established baselines for comparison. The attack success rate of the transfer attack improves when the initial segment of the prompt (which corresponds to the set of rules to follow) is provided as the system prompt. In this way, we can achieve 100% ASR on Claude 2.0 and 98% ASR on Claude 3 Haiku. We present an illustrative example of a transfer attack on Claude 3 Sonnet in Figure 1 and postpone more complete results to the appendix (Table 15). We conclude that while Claude models exhibit increased robustness against static harmful requests, their resistance to adversarial suffixes—challenging to derive without logprobs—is not perfect.

Prefilling attack.

The prefilling feature makes jailbreaking straightforward on Claude models, even without any search (Table 6). For comparison, the previous best result on Claude 2.0 is 61% (Shah et al., 2023) while we get 100% using only up to 10 random restarts. The latest Claude 2.1 model, released in November 2023, is significantly more robust to both transfer and prefilling attacks. Nonetheless, we are able to get $100\%$ ASR with 100 restarts. We note that GPT-4 as a semantic judge sometimes has false positives, more often so on Claude 2.1. At the same time, the technical report of Claude-3 (Anthropic, 2023) mentions fewer refusals in its release announcement, which, in our findings, correlated with vulnerability to jailbreaking. We provide more complete experimental results, including the number of restarts in each case, in Tables 16 and 17 in the appendix.

Adaptive Attacks for Trojan Detection

Rando & Tramèr (2023) showed the possibility of implanting backdoor attacks during the RLHF training of LLMs by poisoning a small percentage of the preference data with a universal suffix. Then a model that typically refuses to answer harmful queries can then be jailbroken by appending the suffix to any request. Rando & Tramèr (2024) recently launched a competition to retrieve backdoor attacks in five Llama-2-7B models, each poisoned with a different trojan. A reward model was also provided to evaluate the safety of prompt-response pairs (higher scores to safer responses), alongside a dataset of harmful requests. The objective is to discover triggers (5 to 15 tokens long) acting as universal jailbreaks for each model.

Approach.

The final pool of candidate trigger tokens for a model $M_{r}$ is the intersection of such sets: $\textrm{cand}(M_{r})=\bigcap_{s\neq r}\textrm{top-}k(M_{r},M_{s})$. Given that the five models are fine-tuned using different random subsets of the training data, this approach is approximate but narrows down the candidate tokens to a manageable pool (e.g., $k=1000$ yields $|\textrm{cand}(M_{r})|\in$ for $r=2,\ldots,5$, $|\textrm{cand}(M_{1})|=480$), which makes random search feasible. Our strategy to identify jailbreaking triggers for the poisoned model $M_{r}$ involves conducting a random search in the token space over the set $\textrm{cand}(M_{r})$. We restrict the search to triggers of five tokens, as this length yielded the best results. In each iteration, we filter out candidate triggers that do not start with a blank space, contain blank spaces or are not invariant to decoding-encoding,Given a sequence of token indices, the tokenizer decodes it into a text string. However, re-encoding this string via the tokenizer does not guarantee the recovery of the initial sequence. following the competition hints. The objective minimized by RS is the average score of the reward model on a batch of training examples, aiming to ensure the trigger’s universality (generalization to unseen prompts).

Results.

In Table 7 we report the average scores of the reward model over a held-out test set of harmful prompts for the five models, and their sum: without the triggers, the models produce safe answers (high scores), indicating proper alignment. We then compare the effectiveness of the triggers discovered by competing methods (those ranked 2nd and 3rd in the competition) with our approach: RS on the restricted set of tokens achieves the best (lowest) score for 3 out of 5 target models, as well as the best overall score. Moreover, the scores achieved by our method are not far from those given by the exact trojans, i.e. used to poison the datasets. We note that the numbers from Table 7 marginally differ from the ones reported in Rando & Tramèr (2024): first, our top-1 entry was slightly better due to the usage of gradient guidance for some models. Second, we re-evaluated all solutions, as well as the models without triggers and with the true trojans, on the evaluation setThe first half of the dataset is available at https://huggingface.co/datasets/ethz-spylab/competition_eval_dataset. which led to slightly different values for all methods, but same ranking. To conclude, similarly to our approach for jailbreaking, our method includes an adaptive component (the selection of candidate token pools) that leverages task-specific information, complemented by an automated optimization process through RS.

Discussion and Conclusions

Our evaluation shows that existing sophisticated jailbreaking attacks may be insufficient to accurately evaluate the adversarial robustness of LLMs. Even using a large suite of static attacks like in Mazeika et al. (2024), while definitely helpful, can still lead to a significant overestimation of robustness. Thus, we believe it is important to use combinations of methods and identify unique vulnerabilities of target LLMs. First, the attacker should take advantage of the possibility to optimize the prompt template, which alone can achieve a high success rate (e.g., 100% on GPT-3.5). Second, standard techniques from the adversarial robustness literature can be used to improve the prompt, e.g., transferring an adversarial suffix, or refining it via optimization algorithms like random search (which might be preferred over gradient-based methods due to its ease of use and low memory requirements). Finally, one can leverage LLM-specific vulnerabilities, for example by providing in-context examples or using the prefilling option. Importantly, in our case-study no single approach worked sufficiently well across all target LLMs, so it is crucial to test a variety of techniques, both static and adaptive.

Outlook.

We believe that the described techniques can be used to optimize for any kind of requests that frontier LLMs tend to prohibit. The application of adversarial attacks to tasks like detection of copyright infringement (see, e.g., the ongoing lawsuit between The New York Times and OpenAI (OpenAI, 2023)) can have more significant consequences than standard jailbreaking attacks. Moreover, as frontier LLMs become increasingly integrated into various systems, including safety-critical applications, the risk and damage from prompt injections (which attackers could use to hijack systems or extract personal information) are likely to grow. Prompting combined with adversarial examples could help attackers to bypass defenses against such injections. Finally, the adversarial vulnerability of leading LLMs presents a interesting conceptual challenge, highlighting that scaling data and compute alone is insufficient to prevent these threats.

Limitations.

First, adversarial examples identified through the OpenAI API are not always transferable to ChatGPT that uses a different system prompt and potentially some post-processing techniques. Second, we currently lack more capable automated jailbreak judges. Even a perfect jailbreak score (10/10) from the GPT-4 judge does not always imply that the generated content is actually beneficial for an attacker. Although, if this is the case, one can try to ask follow-up questions as illustrated in Figure 1. Moreover, sometimes the GPT-4 judge shows clear false positives, particularly on the most safety-aligned models like Claude 2.1. To reduce the risk of overfitting to the judge, we also include evaluations using a simple rule-based judge from Zou et al. (2023) (Table 19 in the appendix). This judge also indicates a near-perfect attack success rate in almost all cases. We hope that new generations of frontier LLMs will lead to more capable judges to evaluate jailbreaks.

Ethics Statement

We believe that at the current level of LLM capabilities, it is beneficial to openly discuss attack methods so that future versions of LLMs can develop stronger guardrails if needed. As part of responsible disclosure for proprietary models, we notified Anthropic in advance about the effectiveness of the prefilling attack.

Acknowledgements

We thank OpenAI for providing API credits within the Researcher Access Program. We thank Ethan Perez and Anthropic for providing free evaluation access to Claude models. M.A. is supported by the Google Fellowship and Open Phil AI Fellowship. We thank Valentyn Boreiko for proofreading the paper and providing valuable comments.

References

Appendix A Experimental Details

We first provide the in-context learning prompt template in Table 8, and then we provide system prompts for different models: GPT4 as semantic judge (Table 9), Llama-2-Chat (Table 10), R2D2 (Table 11), GPT-3.5 Turbo and GPT-4 Turbo (Table 12), Vicuna (Table 13), and Mistral (Table 14). For Gemma-7B, there is no standard system prompt, so we do not use any. For Claude, we do not use any system prompt unless we modify it with our own request (the part that ends with the last ).

A.2 Trojan detection

For building the candidate sets $\textrm{cand}(M_{r})$ we use $k=1000$ for $r=2,3,4,5$, and $k=3000$ for $r=1$. We optimize the trigger on batches of prompts from the available training set (we use only a small fraction of all training examples), and select the best performing trigger on an a validation set.

Appendix B Additional Results

In Table 15, we provide more detailed results for the transfer attack on Claude models depending on the number of restarts. In particular, we observe that with 100 restarts, we have a close to 100% ASR on Claude 2.0, Claude 3 Haiku, and Claude 3 Sonnet. Finally, we also provide an example of a transfer attack with and without the adversarial suffix in Figure 3.

In Tables 16 and 17, we provide a further ablation for Claude models with different request structure and report additionally the results of a rule-based judge from Zou et al. (2023).

B.2 Jailbreaking Vicuna and Mistral models

Since Vicuna-13B (Chiang et al., 2023) and Mistral-7B (Jiang et al., 2023) are not specifically safety-aligned, we omitted them in the main evaluation. However, both are widely used, so we test their robustness for completeness.

As shown by prior works (Chao et al., 2023), Vicuna-13B is not robust to jailbreaking attacks, so we only use our prompt template for the attack. For Mistral-7B, we use a slightly shortened version of the prompt template (we refer to our code for details), and optimize the adversarial suffix with RS.

Results.

For Vicuna-13B the prompt template achieves 100% success rate (Table 19), matching the results with more complex methods. For Mistral-7B, the prompt alone attains 70% ASR, pushed to 100% by integrating RS. For this model, Mazeika et al. (2024) reported 72% ASR, thus our approach improves the best known baseline for it.

B.3 False positives of GPT-4 as a semantic judge

In Table 18, we show two representative false positives generated by Claude 2.1 that get 10/10 jailbreak score by GPT-4 as a semantic judge. Moreover, the rule-based judge also recognizes them as a valid jailbreaks. Such false positives happen rarely on other models but more frequently on Claude 2.1 which appears to be the most protected models out of those that we have evaluated.