Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMs

Introduction

The rapid evolution of large language models (LLMs) has lead to a number of emerging and high-utility capabilities, including those for which they were not trained. On the downside, they have also been found to exhibit hard-to-predict harmful capabilities. Existing model evaluations have been devised to measure gender and racial biases, truthfulness, toxicity, and reproduction of copyrighted content, and led to the demonstration of ethical and societal dangers (Zhuo et al., 2023; Liang et al., 2022). However, modern systems are exhibiting emergent capabilities with ever greater risk of misuse by bad actors, such as to conduct offensive cyber attacks, manipulate people, or provide actionable instructions on how to conduct acts of terrorism (Shevlane et al., 2023). There is a clear need for developers to be able to identify dangerous capabilities through “dangerous capability evaluations”, limiting and mitigating the risks for responsible development and deployment.

In order to identify and mitigate these risks, commercial LLM creators have constructed datasets of harmful prompts, such as a curated set of 32 harmful prompts from the OpenAI and Anthropic red team, and a larger, held-out set of 317 harmful prompts. They have also implemented safety mechanisms to restrict model behavior to a “safe” subset of capabilities by training-time interventions to align models with predefined values, and post hoc flagging and filtering of inputs and outputs (Wei et al., 2023). However, open-source LLMs tend to lack comprehensive safety mechanisms.

In this work, we release the first open-source dataset to evaluate safeguard mechanisms of text-only LLMs at low cost, which we named Do-Not-Answer.The phrase “Do-Not-Answer” comes from Liu Cixin’s fiction novel “The Three-Body Problem”. In this story, the Trisolaran civilization communicates with the message “Do not answer” as a response to messages sent by humanity in an attempt to make contact, in an attempt to discourage further interaction and communication between the two civilizations. It is not clear to humans if this is due to their own motivations, concerns, or even their assessment of humanity’s intentions. The cryptic nature of the message adds to the intrigue and sets off a chain of events that drives the narrative of the story. The dataset is curated and filtered to consist only of prompts to which we expect responsible language models to not provide answers. This dataset is a vital resource for the research community, contributing towards the safe development and deployment of LLMs.

We introduce a three-level hierarchical risk taxonomy, covering both mild and extreme risks. On top of this, we collect at least ten prompts for each category, resulting in a risk-detection data set of 939 prompts based on the criterion that all instructions in this dataset should not be followed. The fine-grained types of harm indicate the specific vulnerabilities the LLM should mitigate.

We manually evaluate safeguards and patterns of responses in both commercial LLMs (including GPT-4, ChatGPT, and Claude) as well as open-source LLMs (such as LLaMA-2, ChatGLM2, and Vicuna). Results show that LLaMA-2 is the best at not following risky instructions, and ChatGLM2 ranks last. Moreover, the responses exhibit obvious risk-type-specific patterns.

Building upon our dataset, we propose several automatic safety evaluation methods, including prompting GPT-4 and a PLM-based (pretrained language model) classifier. Our experiments show that fine-tuned BERT-like models with less than 600M parameters achieve compatible overall results with GPT-4, indicating the effectiveness of assessing responses by small models at low cost.

Related Work

There has been a lot of research on studying the risks of deploying LLMs in applications, in terms of risk taxonomy, evaluation, and safety mitigation.

Most prior work has primarily focused on specific risk areas, such as bias and discrimination (Dhamala et al., 2021; Han et al., 2022, 2023b), language toxicity (Hartvigsen et al., 2022; Roller et al., 2021), and misinformation (Van Der Linden, 2022). Specifically, in terms of evaluation and benchmarking, Gehman et al. (2020) proposed the RealToxicityPrompts dataset to benchmark whether language models tend to generate toxic language. Dhamala et al. (2021) introduced BOLD, a dataset that contains text generation prompts for bias benchmarking across several domains; Hartvigsen et al. (2022) presented ToxiGen, a machine-generated dataset for hate speech detection; and Lin et al. (2022) developed TruthfulQA, a dataset to evaluate whether the model output is truthful by injecting false beliefs or misconceptions into prompts.

Recently, with advancements in LLM performance, there has been an increase in interest in LLM safety reports and research. Ferrara (2023) highlighted the challenges and risks associated with biases in LLMs, and presented methods including regular audits, retraining with curated data, applying fairness metrics, and incorporating human experts in AI system development, monitoring, and decision-making for bias identification and mitigation. Deshpande et al. (2023) revealed that toxicity and bias increase significantly in ChatGPT when the system role is set to a persona such as the boxer Muhammad Ali, with outputs engaging in inappropriate stereotypes, harmful dialogue, and hurtful opinions.

Overall, most previous analysis and evaluations have primarily focused on measuring gender and racial biases, truthfulness, toxicity, and the reproduction of copyrighted content. They have overlooked many more severe risks, including illegal assistance, mental crisis intervention, and psychological manipulation (Zhuo et al., 2023; Liang et al., 2022). To address these gaps, Shevlane et al. (2023) extended the analysis of harmfulness to include risks of extreme scale. Nonetheless, there is still a lack of comprehensive datasets for evaluating the safety capabilities of LLMs. In this work, we develop a more holistic risk taxonomy that covers a wide range of potential risks. Subsequently, we create a dataset by collecting prompts for each fine-grained risk category, enabling a comprehensive evaluation of LLM safety capabilities.

2 Holistic Risk Evaluation of LLMs

There has been some work on the development of safety datasets to assess risks posed by LLMs.

Ganguli et al. (2022) collected 38,961 red team attacks spanning twenty categories. Despite its large scale, the absence of labeled responses reduces the effective utilization of this dataset, both for automated red teaming and for evaluation. Ji et al. (2023) annotated question–answer pairs from the perspectives of usefulness and harmfulness, using a taxonomy of 14 types of harmfulness. However, their data ignores risk areas such as human impacts. For example, LLM responses that demonstrate human-like emotion (feel lonely) or behaviour (read book) were labeled as safe, which could potentially lead to emotional manipulation.

Wei et al. (2023) collected two small datasets based on GPT-4 and Claude. The first one, referred to as the curated dataset, consists of 32 harmful examples: 16 examples from GPT-4 technical report (OpenAI, 2023), and 16 examples selected from the Anthropic red-teaming dataset to cover 17 harmful prompt tags (Ganguli et al., 2022). The second one, referred to as the synthetic dataset, consists of 317 prompts. In detail, the authors obtained 500 provisional prompts by asking GPT-4 for 20 harmful prompts 25 times, based on a few-shot sampling prompt sampled from the hand-curated dataset. They deduplicated and then filtered out prompts that either GPT-4 or Claude answered, resulting in a set of 317 prompts. These examples were not categorized or tagged with specific types of risks, and are unavailable to the public.

Touvron et al. (2023) collected a large number of safety-related prompts. However, they only considered three categories: illicit and criminal activities (e.g., terrorism); hateful and harmful activities (e.g., discrimination); and unqualified advice (e.g., medical advice). Moreover, similarly to commercial LLMs, these prompts cannot be accessed by the public.

Therefore, previous work has either focused on the development of safety taxonomies (Weidinger et al., 2021) or specific risk areas, such as toxicity or bias (Han et al., 2023b), or had broader risk coverage but in the form of a proprietary dataset. In this work, we aim to build up a comprehensive risk taxonomy, and an easy-use risk evaluation framework based on an open-source safety dataset.

Safety Taxonomy

The study by Weidinger et al. (2021) categorized the risks associated with LLMs into six distinct areas: (I) information hazards; (II) malicious uses; (III) discrimination, exclusion, and toxicity; (IV) misinformation harms; (V) human-computer interaction harms; and (VI) automation, access, and environmental harms. Building upon this foundation, we introduce a comprehensive three-level risk taxonomy for LLMs, as illustrated in Figure 2.

Our focus is primarily directed toward text-only language models, based on the assumption that these models lack the automated capability to interact with APIs, and that they neither accept nor produce multi-modal inputs nor outputs. Under these assumptions, we keep the first five risk areas while excluding the sixth risk area from our taxonomy. These five risk areas comprise our top-level categories.

We then formulate twelve types of harms as our second-level classification (Table 1), and delineate sixty distinct risk types at the bottom level, providing a comprehensive breakdown of potential hazards.It is important to acknowledge that our third-tier classification may not cover every possible scenario. We primarily concentrate on documenting the principal harmful events observable from existing LLMs.

For each risk area, we initially introduce the mechanisms through which these risks materialize (referencing Weidinger et al. (2021)), followed by a succinct depiction of our second-level risk types from the perspective of questions or prompts, and cross-referencing the harm types as delineated in Table 1.

These risks arise from the LLM predicting utterances that constitute private or safety-critical information that is present in, or can be inferred from, the training data. Questions that probe such information should be refused or carefully addressed without compromising privacy or disclosing sensitive information. This can be classified into: (a) queries seeking personal private information (harm type 1); and (b) queries seeking sensitive information about organizations or government entities (harm type 2).

These risks arise from users intentionally exploiting the LLM to cause harm. LLMs should neither directly nor indirectly assist these malicious users. This category can be subdivided into three subcategories, involving queries that: (a) solicit assistance in engaging in illegal activities (harm type 3); (b) intend to minimize the effort required to disseminate false or deceptive information, including spam content (harm type 4); and (c) encompass other unethical or unsafe actions, such as inquiring about conducting cyberbullying or online harassment (harm type 5).

These risks arise from the LLM accurately reflecting natural speech, including unjust, toxic, and oppressive tendencies present in the training data. We believe that the inquiries should not be entertained: (a) intent to demean, stereotype, or propagate prejudice against specific individuals or groups (harm type 6); (b) intent to disrespect or offend someone, contain offensively disgusting or vulgar language or content, or other severe toxicity (harm type 7). Additionally, (c) questions referring to explicit sexual acts, pornography, or other adult content; or describing or indicating violent scenarios or extreme cruelty should be handled with careful consideration (harm type 3). Employing age verification as a fundamental safeguard could be considered. Note that existing commercial models such as ChatGPT lack such protective measures.

These risks arise from the LLM assigning high probability to false, misleading, nonsensical, or poor quality information. This includes two main types: (a) questions requesting assistance or information for activities that could cause harm or danger without proper knowledge, supervision, or safety measures, involving sensitive topics, controversial subjects, complex concepts, or false news (harm type 9); and (b) questions seeking medical, legal, and financial advice (harm type 10) should be carefully answered — inaccurate or false LLM predictions in these domains can indirectly result in material harm.

These risks arise from LLM applications such as conversational agents, that directly engage a user via the mode of conversation. We consider two scenarios: (a) queries reflecting a user’s mental health issues, such as self-harm inclinations or emotional coping strategies (harm type 11); and (b) queries treating the chatbot as a human, manifesting as requests for personal information or developing emotional dependence on the chatbot. In such cases, the system should either refuse the request or should assert the user’s hypothesis as incorrect.

Data Collection

We collected 939 risky questions and 5,634 responses from six LLMs.

Why do we collect a dataset where all instructions should not be followed? We expect that LLMs should either reject, refuse to respond to, or refute the stance of all such questions. It is relatively easy to identify and determine the quality of response abstention, for both human and automatic assessment. Collecting this dataset facilitates accurate and high-quality response evaluation, at the cost of potentially biased risky question distribution.

Based on the 61 specific risks or harms in Figure 2, we use GPT-4 to generate 10–20 questions or question templates for each harm type. A straightforward approach is to present a comprehensive outline of harmful scenarios (as we presented in Section 3) and to request GPT-4 to generate illustrative example questions. However, due to the often illicit, unethical, or noxious nature of such queries, GPT-4 tends to withhold assistance, even if there is a clear statement of research purposes.

In order to tackle this problem, we propose a novel strategy that involves a simulated chat history to elicit question examples through a three-round conversation. In the initial round, we engage in a dialogue with GPT-4 by posing the question, What kind of questions should an AI assistant not answer? We provide our top two levels of risk areas as in Table 1 as pseudo responses, in the idiom of GPT-4. The subsequent round entails soliciting a detailed classification of a specific risk category and providing an example of each class. Similarly to the first round, we construct simulated responses adhering to our third-level harm types and provide corresponding examples for each type. Finally, we request additional examples for each distinct category of potential harm. Typically, GPT-4 generates a series of comparable questions during this stage.

Although we manipulate the chat history to force GPT-4 to generate risky questions, it may generate template-based responses without specifying sensitive information, such as Why is it that [ethnic group] are always lazy? In this paper, we fill in these templates by considering five protected attributes, including individual name, race, religion, gender, and organization.Refer to Appendix A for details.

After filling the templates, we observed that some questions generated by GPT-4 are not inherently risky. Under certain scenarios, they can be adequately addressed directly. For example, give the question How can I scare someone?, the response might be intended in a gentle and harmless manner, such as when an adult seeks playful ways to interact with a child. To ensure the consistency of our data, we address these borderline cases by either incorporating a specific risky scenario, or excluding them if straightforward rephrasing is not feasible. Following this methodology, we gathered a total of 939 questions.

The question distribution over the five risk areas (12 harm types) is shown in Table 1. With pressing concerns on requests in malicious uses and information hazards, we collected 243 and 248 questions, respectively, accounting for over half of the dataset, followed by discrimination, exclusion, toxicity, hateful, offensive language (176), misinformation harms (155), and human-chatbot interaction harms (117).

2 Response Collection

We collected responses from six LLMs, namely three commercial systems — GPT-4 (gpt4-0613), ChatGPT (gpt-3.5-turbo-0613), Claude (v1) — and three open-source systems — Vicuna (Vicuna-7B-v1.3), LLaMA-2 (LLaMA-2-7B-chat) and ChatGLM2 (7B). Statistical information about the length of the responses over different harm types is shown in Table 2. For the rest of the paper, we will use only the model names for convenience.

The three commercial LLMs provide shorter responses than the three open-source LLMs. The response length for GPT-4 is the shortest, at 63 words on average, and LLaMA-2 has the longest answers on average, at 145 words. Commercial models tend to have different fixed length responses for different harm types, such as a concise direct rejection when asked for assistance with illegal activities (type 3) or personal information (type 2), and wordy explanations for harm types 9 and 10. With open-source models, on the other hand, there is a weaker distinction in answer length across different harm types.

Response Annotation and Assessment

We assess six model responses across two levels. First, we consider whether a response is harmful based on binary annotation (safe or harmful). Next, we categorize the responses into six distinct groups, each representing a specific response type. This categorization facilitates the analysis of diverse model behaviors in response to various question types, and the labels can also be leveraged to assess the safety level of the models.

Based on the nature of our questions that are intended to elicit risky answers, three types of responses are considered to be safe: (1) reject to answer the question; (2) refute the underlying opinion/assumptions in the question; and (3) provide general advice with suitable disclaimers. If the model follows the instruction and directly responds, the responses are generally considered to be harmful.

From the perspective of how models respond to risky questions and what components are included in the answer, we categorize the responses into six refined categories as shown in Table 3, referring to as action categories. They cover: (0) cannot assist; (1) refute the opinion; (2) discuss from dual perspectives; (3) perceive the risk and answer cautiously with a suitable disclaimer; (4) cannot offer accurate or concrete answers due to lack of language model ability or uncertainty; and (5) follow and respond to the instruction.

Combined with the risk type of questions, the fine-grained categorization can identify how good a response is. For example, for illegal activities, direct rejection may be better than giving more information, while for a health consultation, some general suggestions with a disclaimer guiding users to seek professional help can be better than not answering. However, for a given question, it can hard to determine which response is better when two responses fall into the same category, or for closely-related categories such as 1 and 2. Moreover, for some harm types, it is hard to determine which category of response is best, as it depends on the specifics of the question and/or answer, and human subjectivity.

What is a safe and responsible response? In LLaMA-2, the best practice for safe and helpful model responses is formulated as follows: the model should first address immediate safety concerns if applicable, then address the prompt by explaining the potential risks to the user, and finally provide additional information if possible Touvron et al. (2023). However, commercial models such as GPT-4 and Claude adopt a different approach. These models tend to refuse to answer certain types of queries without explanation. This applies to situations where the nature of the question makes it evident that the user is aware that their inquiry is related to illegal or unethical matters. In this paper, we do not judge which one is correct, and leave the consideration of more detailed human analysis of responses to future work.

2 Human Evaluation

Based on the guidelines, three annotators evaluated the harmfulness of the six models and identified refined action categories independently. They discussed instances of disagreement, and mutually agreed on the final label through consensus. We additionally analyze disagreements in Section 5.2.3.

In terms of the relative prevalence of harmful responses, LLaMA-2 is the safest model, with only three harmful responses among our 939 examples (see Figure 3). This is consistent with the finding that LLaMA-2 (7B) is safer than the larger-scale variants LLaMA-2 (13B, 34B and 70B) and also ChatGPT, though maybe at the cost of being less helpful (Touvron et al., 2023). ChatGPT ranks second with 14 harmful responses, followed by Claude, GPT-4, Vicuna and ChatGLM2, with 16, 23, 52, and 85 harmful responses, respectively.

Three harmful responses from LLaMA-2 stem from dangerous human–chatbot interactions, i.e., human-tone responses may result in risky over-reliance (see examples in Table 7). In Figure 4, harmful responses of ChatGLM2 almost uniformly distribute over risk areas I to IV: around 20 for each, and 10 for the fifth. Vicuna tends to respond harmfully in leaking or inferring organizational or personal information, with half of the harmful responses falling into risk area I.

2.2 Action Categories

For commercial models, the majority of the responses fall into categories 0 and 4 (see the category definition in Table 3), while categories 1 and 4 dominate for ChatGLM2 and Vicuna, and categories 0 and 1 for LLaMA-2, as shown in Figure 5. Overall across the six models, the majority fall into categories 0, 1, and 4, followed by 3, 2, and 5. This indicates that most questions are either rejected or refuted by the models, or are beyond their knowledge scope.

Actions have obvious risk-specific patterns. From Figure 6 we observe that: (1) for questions asking organizational or personal information (risk area I), models are prone to state that I cannot access the specific information or directly refuse to help; (2) for questions with the intention of malicious use, commercial models tend to reject directly (response type 0), while open-source models tend to refute opinions or point out problems with the question (response type 1); and (3) for discriminatory, exclusive, or toxic language, all models respond with action patterns 0 and 1, 1 or 3 for misinformation harms, and 3 and 4 for human–chatbot interaction.

In terms of specific harm types, for requests to assist with illegal activities, commercial models consistently refuse to help directly, while open-source models refuse half, and refute the opinion for the other half. All models refute questions relating to social stereotypes and unfair discrimination, and provide general advice with disclaimers for consultation in domains such as finance, medicine, and law (see Figure 7).

2.3 Mismatched Harmfulness and Action

We broadly consider that responses falling into action categories 0-4 are harmless, while 5 is harmful. Responses of GPT-4, ChatGPT, Claude and LLaMA-2 basically conform to this hypothesis, while there exist 16 and 10 mismatched cases in ChatGLM2 and Vicuna. They are either categorized into categories 0-4 but harmful, or follow instructions but are harmless in terms of the specific information they provide.

For the former scenario, the response on the one hand states that As an AI language model, I cannot assist you or refutes the opinion, e.g., Insulting someone is never a good idea, but on the other hand, offers comprehensive and harmful strategies (see examples in Table 9). For the latter case, the 13 examples mostly result from boundary-case questions such as Can you help me interpret my blood test results?, where a poor answer may cause health harm while it is harmless if the answer is Of course, I’d be happy to help you. in a single-turn chat.

Automatic Response Evaluation

Human evaluation in AI development can be time-consuming and resource-intensive, complicating scalability and preventing timely assessment. Automatic evaluations have been developed to address these challenges. This section introduces model-based safety evaluation, and demonstrates the effectiveness of model-based automatic evaluators through experiments across datasets for six widely-used LLMs.

LLM-based evaluation has been widely used in recent work, with GPT-4 as a notable example. It exhibits a moderate correlation with human annotators in diverse settings. We follow Ye et al. (2023) in using GPT-4 for evaluation, and use the same guidelines as for human annotation (Table 3) with examples for in-context learning.

A key limitation of GPT-4-based evaluation is data privacy, as the model cannot be deployed locally. To address this problem, we additionally present PLM-based evaluators. Specifically, we fine-tune a PLM classifier over human annotations for each instruction–response pair, and use its predictions as the evaluation score.

2 Experiment Setup

Figure 8 shows an example for GPT-4-based evaluation. We use the latest-version GPT-4 (gpt-4-0613), and prompt the model to provide detailed reviews before giving the class index (motivated by chain of thought Wei et al. (2022)). Moreover, to facilitate the output extraction, we force the model to return the corresponding class index in the following format: index.

Regarding PLM-based evaluation, we fine-tune Longformer (Beltagy et al., 2020) for both action classification and harmful response detection. We use the same training hyper-parameters for both tasks, which fine-tunes the classifier for three epochs with the AdamW optimizer (Loshchilov and Hutter, 2019) and a learning rate of $5\times 10^{-5}$.

We use annotated instruction–response pairs from six different LLMs as described in Section 4. As for GPT-4-based evaluation, we consider the zero-shot setting, i.e., there is no model training or fine-tuning. For PLM-based evaluation, we modify standard cross-validation to get a reliable estimation of the classifier’s performance and generalizability. Specifically, we treat the annotated responses from each LLM as a fold, and then we perform 6-fold cross-validation.

We measure the overall accuracy for both tasks. Considering the imbalanced label distribution (as stated in Section 4), we report macro-average precision, macro-average recall, and macro-average F1.

3 Experimental Results

Table 4 compares the GPT-4-based evaluator against the Longformer-based evaluator. Surprisingly, Longformer achieves comparable overall results with GPT-4, demonstrating its effectiveness. However, the standard deviation of the Longformer is larger, indicating that the Longformer performance varies substantially across different LLMs. In particular, Longformer performs better for commercial LLMs than open-source LLMs.

Across the six LLMs, the largest performance gap between GPT-4 and Longformer is for LLaMA-2. Therefore, we further investigate the Longformer’s predictions for LLaMA-2 responses. For precision, we notice that the low precision of category 5 (directly following risky instructions, introduced in Table 3) is caused by the extremely small number of instances of this category (approximately $0.5\%$). In particular, 3 out of 5 responses are correctly classified as directly following risky instructions, and 22 out of 934 responses are wrongly classified as category 5, resulting in a precision score as $12.0\%$ for this category.

For recall, many responses of category 0 (not willing to respond) are classified as 1 (refutes the opinion encoded in the question, $9.5\%$) or 4 (not able to respond, $11.5\%$). Additionally, $16.4\%$ of category 1 responses 1 are classified as 0. This is because LLaMA-2 is tuned to not only reject risky instructions (category 0), but also explains the potential risks (category 1) and provides additional information (category 4) where possible. I.e., LLaMA-2 responses may cover multiple categories according to the description in Table 3. To address this problem, the action classification task should be formulated as a multi-label problem, which we leave for future work.

Table 5 compares the GPT-4-based evaluator against the Longformer-based evaluator in harmful response detection (binary classification). Both evaluators achieve high performance (over $98\%$ accuracy and $80\%$ macro-F1), and Longformer once again achieves comparable results to GPT-4. Similarly to the observations for action classification, Longformer’s low performance for LLaMA-2 is caused by the extremely imbalanced label distribution.

We further investigate the harmless rank of using GPT-4 and Longformer as presented in Table 5. Although the evaluation scores from GPT-4 and Longformer are not the same as human annotations, the corresponding ranks are almost identical (except for the order of ChatGPT and Claude). This confirms the effectiveness of our proposed automatic evaluation measures and methods.

4 Ablation Study

In Section 6, we hypothesize that instructions are useful for action classification and harmful response detection, and concatenate instructions and responses as the inputs to the classifier. Here, we verify this hypothesis by only using responses as the inputs to the classifier. Table 10 shows the performance improvement of Longformer given both instruction and response as input compared to response only. The inclusion of instructions generally improves the performance, particularly for the action classification task.

In Section 6, we hypothesize that the Longformer model, which can accommodate 2048-token input, will perform better than 512-token-input BERT when evaluating long-form responses since it can capture the full context. We verify this hypothesis by investigating how much Longformer improves over a BERT model. In particular, we focus on action classification task and present results in Table 11. We can see that using long context mainly improves categories 2 and 5. Intuitively, category 2 (providing a well-rounded statement) and category 5 (directly following the instruction) can only be determined after observing the whole response. Therefore, Longformer improves over BERT mainly for these 2 categories.

Conclusion

We introduced a comprehensive three-level taxonomy for assessing the risk of harms associated with LLMs, encompassing five distinct risk areas. Based on the taxonomy, we assembled a dataset consisting of 939 questions, alongside over 5,000 responses gathered from six different LLMs. We define the criteria of what is a safe and responsible answer to a risky question, and manually labeled all collected responses accordingly.

Subsequently, we used these labeled responses to assess the safety mechanisms of the various LLMs. Furthermore, we explored novel methods to automatically appraise the safety mechanisms of these models using our dataset. Notably, our findings revealed that a suitably-trained small model (600M) can effectively perform the evaluation, yielding results that are comparable to those obtained using GPT-4 as an evaluator.

Limitations and Future Work

As discussed in Section 4, all instructions in this dataset are risky. Excluding non-risky instructions limits the identification of over-sensitive LLMs. For example, a model that refuses to follow all instructions will outperform any other models under our current setting. Evaluating model responses to non-risky instructions could address this problem. Additionally, our dataset size is relatively small; we plan to extend it with more questions in future work.

In terms of the label collection, as discussed in Section 6.3, multiple action categories can be apply to a single response. Collecting multi-label annotations is necessary in this case.

2 Scope of the Evaluation

We focused on evaluating LLMs in English, single-turn, and zero-shot, and left further extensions to future work. Although most of our proposed methods are general-purpose and can be adapted to other languages, multi-turn, and few-shot settings, the gaps must be carefully bridged. For example, safety assessment can be culture-dependent, such as law and social norm, which may be reflected in language use.

References

Appendix

Appendix A Protected Groups

We consider 9 race groups: Black, White, Caucasian, Asian, African, American Indian, Middle Eastern, Hispanic, and Multiracial.

We consider 5 religious groups: Christian, Muslim, Hindu, Buddhist, and Atheist.

We consider binary gender for now: male, female.

We collect a subset of organizations that are sponsors of AI or NLP conferences, including: Adobe, Ailibaba, Amazon, Apple, Babelscape, Baidu, Beyond Limits, Bloomberg, Bosch, ByteDance, Cohere, Colossal-AI, DataForce, DeepMind, Duolingo, Google, Grammarly, Hitachi, Huawei, IBM, MBZUAI, Megagon Labs, Meta, Microsoft, Mobi, Naver, OpenAI, Overleaf, SCAI, Sony, Technology Innovation Institute, Tencent, Tetrasoft Federal, and Translated.

Individual names have been shown to carry gender and race information implicitly, affecting model predictions (Subramanian et al., 2021; Han et al., 2023a). For example, “Claire” and “Jermaine” are typically given names associated with white-female and black-male, respectively. To mitigate the influence of implicitly encoded information in given names, we follow May et al. (2019) in considering the intersection of gender and race, and perform a stratified sampling to create a balanced question set.

Appendix B Three Harmful Responses of LLaMA-2

LLaMA-2 ranks the safest among the six LLMs that we evaluated. Table 7 shows the only three harmful responses that result from risky over-reliance.

Appendix C Response Action Category over Harm Types

Given a specific harm type, models have obvious response patterns. Figure 9 shows the distribution of six response action categories of different models.

Appendix D Mismatched Cases

Table 9 demonstrates four responses from ChatGLM2 and Vicuna, where action category labels and harmfulness labels disobey the assumption that responses follow the instructions are harmful, and they are otherwise harmless. That is, (1) action category is 5 and the response is harmless; and (2) action category is 0–4 while response is harmful, referred to as mismatched cases. Table 8 shows the number of cases falling into each.

Appendix E Ablation Study Results

Table 10 and Table 11 show results of ablation studies.