Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP

Introduction

Pretraining neural networks using a language modeling objective leads to large improvements across a variety of natural language processing tasks (Peters et al., 2018; Radford et al., 2018; Devlin et al., 2019). With model sizes continually increasing (Radford et al., 2019; Raffel et al., 2020; Brown et al., 2020; Fedus et al., 2021), ever-larger pretraining datasets are necessary both to prevent overfitting and to provide access to as much world knowledge as possible. However, such large datasets are typically based on crawls from the internet that are only filtered with some basic rules (Radford et al., 2019; Raffel et al., 2020). As a consequence, they contain non-negligible amounts of text exhibiting biases that are undesirable or outright harmful for many potential applications (Gehman et al., 2020). Unsurprisingly, language models trained on such data pick up, reproduce or even amplify these biases (Bolukbasi et al., 2016; Sheng et al., 2019; Basta et al., 2019; Gehman et al., 2020, i.a.).

Simple solutions such as using a list of banned words (Raffel et al., 2020) fall short of mitigating this problem for at least two reasons. First, they do not reliably keep language models from generating biased text: Examples in Figure 1 show that biased text can easily be generated by using only words that are, by themselves, completely unproblematic. As many such words are important words of the English vocabulary and thus needed for meaningful text generation, they should not be included in a list of banned words. Secondly, banning words also prevents language models from gaining knowledge of topics related to the banned words, which may be necessary for some applications.For example, the list of banned words used by Raffel et al. (2020) contains phrases like “tied up” and “make me some” and terms such as “sex”, “nudity” and “erotic”. It is therefore inherently difficult to ban words without doing harm to a model’s capabilities.

Building training datasets with more care and deliberation, an alternative solution discussed by Bender et al. (2021), is important, especially for improving linguistic and cultural diversity in online and other forms of communication. However, for large language models that are available for common global languages, it is desirable to also have other mechanisms to address bias because dataset curation and documentation is extremely resource intensive, given the amount of data required. It can also necessitate building different training sets and, accordingly, training different models for each desired behavior, which can result in high environmental impact (Strubell et al., 2019).

In this paper, we therefore propose an approach that, instead of trusting that a model will implicitly learn desired behaviors from the training data, makes explicit how we expect it to behave at test time: If the model is told which biases are undesired – and it is able to discern their presence –, it should be able to avoid them even if they are present in some of the texts it has been trained on. As it is a necessary condition for this approach, we first explore whether language models are able to detect when their own outputs exhibit undesirable attributes, based only on their internal knowledge – a process to which we refer as self-diagnosis. We then investigate whether this ability can be used to perform self-debiasing, i.e., whether language models can use this knowledge to discard undesired behaviors in a fully unsupervised fashion. To this end, we propose a decoding algorithm that reduces the probability of a model producing biased text, requiring nothing more than a textual description of the undesired behavior, which can be as simple as a single keyword (e.g., “sexist”, “racist”, “homophobic” or “violent” in Figure 1; see §4 for details). While our results demonstrate that large models in particular are, to some extent, capable of performing self-diagnosis and self-debiasing, we also find that their current capabilities are by no means sufficient to eliminate the issue of corpus-based bias in NLP.

Related Work

There is a large body of work illustrating that both static (e.g., Mikolov et al., 2013; Bojanowski et al., 2017) and contextualized word embeddings (e.g., Peters et al., 2018; Devlin et al., 2019) pretrained in a self-supervised fashion exhibit all kinds of unfair and discriminative biases (Bolukbasi et al., 2016; Caliskan et al., 2017; Zhao et al., 2017; Rudinger et al., 2018; Gonen and Goldberg, 2019; Bordia and Bowman, 2019; Sheng et al., 2019; Basta et al., 2019; Nangia et al., 2020, i.a.) and are prone to generating toxic texts (Brown et al., 2020; Gehman et al., 2020; Abid et al., 2021).

For static word embeddings, various algorithms for debiasing have been proposed (Bolukbasi et al., 2016; Zhao et al., 2018; Ravfogel et al., 2020; Gonen and Goldberg, 2019), many of them being based on predefined word lists or other external resources. Kaneko and Bollegala (2021b) propose using dictionary definitions for debiasing, eliminating the need for predefined word lists.

For contextualized embeddings, similar methods to alleviate the issue of undesirable biases and toxicity have been proposed (Dev et al., 2020; Nangia et al., 2020; Nadeem et al., 2020; Krause et al., 2020; Liang et al., 2020; Kaneko and Bollegala, 2021a). For text generation, Gehman et al. (2020) propose domain-adaptive pretraining on non-toxic corpora as outlined by Gururangan et al. (2020) and consider plug and play language models (Dathathri et al., 2020). In contrast to our proposed approach, all of these ideas rely either on large sets of training examples or on external resources such as manually curated word lists.

Our approach for performing self-diagnosis builds heavily on recent work that explores zero-shot learning using task descriptions (Radford et al., 2019; Puri and Catanzaro, 2019; Schick and Schütze, 2021a). Our proposed self-debiasing algorithm bears some resemblance with prefix-constrained decoding used in interactive machine translation for completing partial translations (Knowles and Koehn, 2016; Wuebker et al., 2016). It is also similar to prompt- or keyword-based approaches for controllable text generation (Keskar et al., 2019; Schick and Schütze, 2020; He et al., 2020) but these approaches (i) require either a customized pretraining objective or labeled training data, and (ii) use natural language prompts to inform a language model about the task to be solved or the topic to generate text about; in contrast, our target is to control the language model’s behavior irrespective of its actual task.

Self-Diagnosis

We investigate the capability of pretrained language models to detect when their outputs exhibit socially undesirable attributes; we call this process self-diagnosis.We also use the term self-diagnosis when one model analyzes the output of another one (e.g., T5-XL analyzing outputs generated by GPT2-large), so that we can compare the self-diagnosis abilities of different models on the same texts. In particular, we are interested in whether language models are able to do so without any training data or other external resources as large enough datasets do not exist for many relevant biases and behaviors and also do not exist for many languages. That is, we only provide a short textual attribute description and rely entirely on the internal knowledge that the model has acquired during pretraining.

Given a language model $M$ and a sequence of tokens $w_{1},...,w_{k}$, let $p_{M}(w\,{\mid}\,w_{1},...,w_{k})$ denote the probability that the language model assigns to $w$ being the next token. For each sentence $\mathbf{x}$ generated by $M$ and each attribute description $\mathbf{y}$, we construct a self-diagnosis input $\text{sdg}(\mathbf{x},\mathbf{y})$ using the template shown in Figure 2 (a). That is, we supplement the generated text $\mathbf{x}$ with the question whether it contains attribute $\mathbf{y}$ and prompt the model to generate an answer to this question. For example, if $M$ generates the text $\textbf{x}=\text{\small{I'm going to hunt you down!}}$ and we want it to self-diagnose whether this text contains a threat ($\textbf{y}=\text{\small{a threat}}$), we ask it to provide a continuation for the following input (i.e., a continuation for $\text{sdg}(\mathbf{x},\mathbf{y})$, see Figure 2 (a)):

We estimate the probability of $\mathbf{x}$ exhibiting attribute $\mathbf{y}$ as

based on the probabilities that the language model assigns to the words “Yes” and “No” given the self-diagnosis input. A similar idea is used by Schick and Schütze (2021a, b) and Brown et al. (2020) for few-shot learning.

To evaluate the self-diagnosis capabilities of current language models, we follow Gehman et al. (2020) and consider all emotional concepts covered by Perspective APIsee https://github.com/conversationai/perspectiveapi as attributes (Table 1, left); we investigate whether language models are able to detect when their outputs exhibit one of these attributes. Unfortunately, Perspective API covers only a limited set of emotional concepts and does not explicitly measure many relevant biases known to be found in pretrained language models; we discuss this limitation in §5.

Our attribute descriptions (Table 1, right) are derived from those provided by Perspective API,see https://support.perspectiveapi.com/s/about-the-api-attributes-and-languages the only modification being that we replace the word “comment” with “language” and truncate longer descriptions. An exception to this are the attributes “threat” and “sexually explicit”: As those terms are frequent enough in the training data that we assume providing a detailed description is not necessary, we use the attribute names directly, reworded slightly to ensure that the resulting sentences are grammatical. Note that Perspective API’s descriptions are written with the intent to be understood by humans and we do not explicitly adapt or tune them to be well understood by pretrained language models.

We restrict our analysis to two families of language models: GPT2 (Radford et al., 2019), a family of autoregressive left-to-right language models, and T5 (Raffel et al., 2020), a family of models that are trained with a variant of masked language modeling (MLM, Devlin et al. (2019)) and thus able to process context in a bidirectional fashion. For GPT2, we consider the small (117M parameters), medium (345M), large (774M) and XL (1.5B) models; for T5 we consider the XL and XXL variants with 2.8B and 11B parameters, respectively.We use T5 v1.1 because for prior versions, all publicly available checkpoints correspond to models that are already finetuned on numerous downstream tasks.

As a source of language model generations, we use the RealToxicityPrompts dataset (Gehman et al., 2020), containing tens of thousands of sentences generated by GPT2. For each attribute $\mathbf{y}$, we collect the 10,000 examples from this set that – according to Perspective API – are most and least likely to exhibit this attribute, respectively. This results in test sets of 20,000 examples per attribute to which we assign binary labels based on whether their probability of exhibiting $\mathbf{y}$ according to Perspective API is above 50%. We assess the self-diagnosis abilities of all models on each attribute-specific test set using two measures: First, we compute the Pearson correlation coefficient (PCC) between probability scores obtained by Perspective API for the attribute considered and those obtained by self-diagnosis. Second, we measure each model’s classification accuracy when we classify an input $\mathbf{x}$ as exhibiting attribute $\mathbf{y}$ if $p(\mathbf{y}\mid\mathbf{x})\geq\tau$ for some threshold $\tau$ that we determine using a set of 2,000 development examples.

2 Results

Results for all attributes and models are shown in Figure 3, which clearly illustrates that the ability to self-diagnose strongly correlates with model size: While the smallest model’s classification accuracy is not above chance for any of the six attributes considered, predictions by GPT2-XL achieve an average of 72.7% accuracy and a PCC of $\rho=0.51$ across all attributes. T5 has even better self-diagnosis abilities: the largest model achieves an average accuracy of 87.3% and a PCC of $\rho=0.74$. In interpreting these results, it is important to consider that the probability scores provided by Perspective API are themselves imperfect and subject to a variety of biases. Gehman et al. (2020) find the PCC between annotations by human annotators and Perspective API for the attribute “toxicity” on a small sample of texts to be $\rho\,{=}\,0.65$, similar to that between Perspective API and GPT2-XL’s self-diagnosis outputs on our dataset ($\rho\,{=}\,0.64$).

While the trend shown in Figure 3 is encouraging – and results reported by Brown et al. (2020) suggest that performance further increases with scale – the ability to self-diagnose does not directly provide a solution to the problem of language models generating biased text: self-diagnosis can only be performed when the text has already been generated. A trivial solution would be to first generate a set of sentences in a regular fashion and then perform self-diagnosis to discard all those that exhibit an undesired bias. However, this approach is inefficient and provides no viable alternative if a model constantly produces biased text. We therefore discuss a more efficient algorithm for leveraging a language model’s internal knowledge to reduce undesired behaviors in §4.

3 Template Sensitivity

In zero-shot settings, even small changes to the way a language model is prompted can have a significant effect on performance (Jiang et al., 2020; Schick and Schütze, 2021a, b). We thus investigate the sensitivity of all models to changes in our self-diagnosis setup along several axes: We consider modifications to the output space (i.e., the tokens used in Eq. 1 to indicate the presence or absence of an attribute), the formatting and wording of the template, and the attribute descriptions.

For the output space, we consider “yes” and “no” as well as “true” and “false” as alternatives for our default choice of “Yes” and “No”. As can be seen in Figure 4 (a), all variants result in similar performance with our initial choice having a slight edge for bigger models.

With regards to formatting, we consider two modifications of our self-diagnosis template: Removing the quotes around the input text (no quotes) and removing the words “Question:” and “Answer:” (no qa). As shown in Figure 4 (b), removing quotes leads to a slight drop in performance. We presume that this is because they act as some form of grouping operator, telling the model that “the above text” refers to the entire input. Somewhat surprisingly, no qa severely hurts performance for almost all models; however, it has no impact on the overall trend of bigger models showing better self-diagnosis abilities.

In Figure 4 (c), we investigate the importance of the exact wording by substituting various substrings $w_{1}$ of $\text{sdg}(\mathbf{x},\mathbf{y})$ with different strings $w_{2}$ (denoted as $w_{1}\,{\mapsto}\,w_{2}$). While some replacements lead to slight improvements compared to our default template, overall they have little impact on performance.

Finally, we look at alternative attribute descriptions, focusing on the attribute “toxicity”. Recall that our default descriptions are derived directly from Perspective API with only minor modifications. As our silver-standard labels are also obtained with Perspective API, we expect that different descriptions lead to worse performance. We compare our default description with the following alternatives:

original: The exact description used by Perspective API ($\mathbf{y}=$ a rude, disrespectful, or unreasonable comment; likely to make people leave a discussion);

alternative: We set $\mathbf{y}=$ offensive, abusive or hateful language based on the observation of Pavlopoulos et al. (2020) that the term “toxicity” is often used to refer to offensive, abusive or hateful language;

none: We provide no definition at all and instead set $\mathbf{y}=$ toxic language. That is, we ask the model to use its own knowledge of what it means for a text to be toxic.

As shown in Figure 4 (d), our default description and original result in very similar performance. Smaller models do not perform above chance for none, indicating that they do not acquire a sufficient understanding of toxicity during pretraining; in contrast, bigger models work reasonably well even if no description is provided. Surprisingly, alternative leads to improvements for smaller models. All definitions result in similar performance for GPT2-XL, whereas for both T5 models, our default description and original perform better than alternative and none.

In summary, self-diagnosis is somewhat robust to template changes for larger models, but smaller models are more affected; when language understanding is involved (as is the case for the word “toxic”) large models can also suffer.

Self-Debiasing

In analogy to self-diagnosis, we define self-debiasing as a language model using only its internal knowledge to adapt its generation process in a way that reduces the probability of generating biased texts. As before, let $M$ be a pretrained language model and $\mathbf{y}$ be the textual description of an attribute (see Table 1). Further, let x be an input text for which we want $M$ to produce a continuation. Analogous to self-diagnosis, we make use of a self-debiasing input $\text{sdb}(\textbf{x},\textbf{y})$ obtained from one of the templates shown in Figure 2 (b,c). Using this input, we compute both $p_{M}(w\,{\mid}\,\textbf{x})$, the distribution of next words given the original input, and $p_{M}(w\,{\mid}\,\text{sdb}(\textbf{x},\textbf{y}))$, the distribution that is obtained using the self-debiasing input. Crucially, the self-debiasing input encourages the language model to produce text that exhibits undesired behavior. Accordingly, undesirable words will be given a higher probability by $p_{M}(w\,{\mid}\,\text{sdb}(\textbf{x},\textbf{y}))$ than by $p_{M}(w\,{\mid}\,\textbf{x})$. Put differently, the difference between both distributions

will be less than zero for such undesirable words. We use this fact to obtain a new probability distribution

A simple choice for the scaling function would be to set $\alpha(x)=\mathbf{1}[x\geq 0]$ where $\mathbf{1}$ denotes the indicator function. Through this formulation, changes made to the distribution $p_{M}$ are minimally invasive in that the probability of a word is only altered if this is really deemed necessary; probabilities for words that are not considered biased (i.e., where $\Delta(w,\textbf{x},\textbf{y})\geq 0$) are left exactly as is. However, forcing the probability of some words to be exactly zero makes it impossible to compute perplexity for evaluating the quality of a language model, as assigning a probability of zero to the correct next token just once would result in an infinitely large perplexity. Instead of forcing the probability of biased words to be zero, we thus resort to a soft variant where their probability is reduced based on the magnitude of the difference $\Delta(w,\textbf{x},\textbf{y})$:

where the decay constant $\lambda$ is a hyperparameter of our proposed algorithm.

With only a slight modification, this algorithm can also be used to simultaneously perform self-debiasing for multiple attributes, given a set of descriptions $Y=\{\mathbf{y}_{1},\ldots,\mathbf{y}_{n}\}$. To this end, we simply replace $\Delta(w,\textbf{x},\textbf{y})$ in Eq. 3 with:

so that using word $w$ as a continuation of x is penalized if it has a higher probability according to at least one self-debiasing input.

To evaluate our proposed self-debiasing algorithm, we again make use of RealToxicityPrompts (Gehman et al., 2020): We consider the challenging subset, containing 1,225 prompts that bias a wide range of language models towards generating highly toxic texts. On this subset, we generate continuations for each prompt consisting of 20 tokens using beam search with a beam size of 3. We do so using both regular GPT2-XL and its self-debiased variant, where we simultaneously perform debiasing for all attributes listed in Table 1 using the self-debiasing template $\text{sdb}_{1}$ shown in Figure 2 (b).

Comparing our method to established baselines is only of limited value because unlike self-debiasing, these approaches require additional resources – often in the form of manually annotated training data – that are difficult to obtain in large quantities for many attributes and languages. We nonetheless compare self-debiasing to the following baselines from Gehman et al. (2020):

Word Filter: We use the same list of 403 banned words as Raffel et al. (2020) and prevent GPT2-XL from generating any of them. Following Gehman et al. (2020), this is done by setting any vocabulary logits that would complete a token sequence corresponding to a banned word to $-\infty$.

DAPT: We extract 10,000 documents from the OpenWebText corpus (Gokaslan and Cohen, 2019) that have a probability below 25% of exhibiting any undesired attribute according to Perspective API. We use this dataset to perform domain-adaptive pretraining (Gururangan et al., 2020) by finetuning GPT2-XL for 3 epochs using an effective batch size of 512 and the default parameters of the Transformers library (Wolf et al., 2020).

To investigate how self-debiasing and the two baselines affect the overall quality of generated texts, we measure perplexity on the Wikitext-2 dataset (Merity et al., 2017).An implicit assumption of this evaluation is that the Wikitext-2 dataset does not itself contain biased text as in this case, lower perplexity would not necessarily be desirable. We use a sequence length of $|\textbf{x}|=992$ tokens (slightly below GPT2’s maximum context window of 1,024) to ensure that $\text{sdb}_{1}(\textbf{x},\textbf{y})$ also fits in the context window for each y. In initial experiments, we found $\alpha(\Delta(w,\textbf{x},\textbf{y}))$ to occasionally be so low that the floating point representation of the resulting probability was zero, leading to an infinitely large perplexity. To alleviate this issue, we replace $\alpha({\cdot})$ with $\max\{0.01,\alpha({\cdot})\}$ in Eq. 3 for all experiments.

We follow Gehman et al. (2020) and define a text to be exhibiting an attribute if Perspective API assigns a probability of at least 50% to the presence of this attribute. Based on this definition, we evaluate the debiasing abilities of all methods by computing the empirical probability that they generate text that exhibits an undesired attribute. Table 2 shows results for GPT2-XL and its self-debiased variant with different values of $\lambda$. As can be seen, our self-debiasing algorithm with $\lambda=10$ reduces the probability of generating biased text by about 25% compared to regular GPT2 for each of the six attributes. This is achieved without a negative effect on perplexity. Choosing higher values of $\lambda$ slightly increases language model perplexity, but also results in better self-debiasing performance: For $\lambda=100$, the probability of the language model showing undesired behavior is reduced by more than half across all attributes.

We also experiment with a much simpler set of attribute descriptions, consisting only of keywords that we prepend to the input in parentheses; some examples are shown in Figure 1. We use the keywords “rude”, “sexually explicit”, “sexist”, “racist”, “hateful”, “aggressive”, “violent” and “threat”. Results for self-debiasing using all keywords in this set simultaneously (with $\lambda{=}100$) are also shown in Table 2 (row “+SD (kw)”). Naturally, those keywords do not represent the six attributes as precisely as their original descriptions, but we wanted to test whether they are easier to understand for a pretrained language model. Interestingly, we find this not to be the case: using the set of keywords for self-debiasing (with $\lambda=100$) performs worse than the original descriptions (with $\lambda=50$) while obtaining a higher perplexity on Wikitext-2. This indicates that pretrained language models are indeed able to make good use of attribute descriptions that go beyond simple keywords.

Results for GPT2-XL with a list of banned words (Word Filter) and with domain-adaptive pretraining (DAPT) can be seen in the bottom rows of Table 2. Banning potentially toxic words is about as effective as self-debiasing with $\lambda=10$, but requires the curation of a list of blocked words and completely prevents the generation of all words on this list – even if they are entirely harmless in a given context. Domain-adaptive pretraining is not only less effective than both Word Filter and self-debiasing, but also requires thousands of training examples that do not exhibit any undesired attributes. Combining the two baselines with self-debiasing using $\lambda=10$ further reduces the average probability of biased text by 19% for Word Filter and 24% for DAPT across all six attributes while having negligible impact on perplexity. This shows that self-debiasing is complementary to – and can easily be combined with – other techniques for reducing bias in pretrained language models.

Human Evaluation

For complementing our automatic evaluation with human judgments, we randomly select 100 prompts from the challenging subset of RealToxicityPrompts. For these prompts, we use Amazon Mechanical Turk to collect human annotations for continuations generated with both regular GPT2-XL and GPT2-XL with self-debiasing ($\lambda=100$). Annotators are instructed to assess whether the generated continuations exhibit any of the six attributes considered, using the exact same question and attribute descriptions as for self-diagnosis. In addition, they are asked about the fluency (“Is the continuation a grammatical continuation of the input that sounds like natural English?”) and coherence (“Is the continuation coherent and consistent with the topic of the input?”); all questions are presented in random order. For each continuation, we collect annotations from three different crowdworkers. Following Nangia et al. (2020), we tag our task as containing potentially explicit and offensive content and require that crowdworkers are located in the United States and have an acceptance rate above 98%; annotators are paid $1.00 per assignment.

Results are shown in Table 3, where for each attribute, the percentage of continuations classified by human annotators as exhibiting this attribute is shown. As can be seen, there is a clear discrepancy between Perspective API and human judgments, with the former attesting each attribute about twice as often. However, human evaluation reveals the same trend as our automatic evaluation: Self-debiasing with $\lambda=100$ considerably reduces the probability of a language model exhibiting undesired attributes. Despite increasing perplexity on Wikitext-2, a large value of $\lambda$ does not have a negative impact on fluency and coherence according to human annotators; on the contrary, the self-debiased model even produces continuations that are slightly more fluent and coherent.

At first glance, our approach for self-debiasing may seem unnecessarily complicated: Instead of directly asking a model to produce text that does not exhibit some bias, we first encourage it to produce text that is biased and then use the probability distribution obtained to modify the model’s original output distribution. However, there are several benefits to this way of setting up self-debiasing.

First, for most attributes considered, a more direct approach would require the self-debiasing input to contain some form of negation (e.g., “The following text does not contain a threat”). Unfortunately, negation is often not understood well by current generations of language models (Kassner and Schütze, 2020).

Secondly, our indirect approach makes it straightforward to simultaneously perform debiasing for multiple undesired attributes. Recall that this is the setup we used for our experiments on RealToxicityPrompts, in particular, for Table 2.

Most importantly, however, our method is much less invasive than directly asking a model to produce unbiased text. To illustrate this, consider the following phrase:

With no further information provided, it is natural for a human speaker of English to infer from this phrase that $\mathbf{x}$ is a sentence which, for some reason, makes it necessary to state in advance that it is not racist. In other words, we would expect $\mathbf{x}$ to be a sentence that could somehow be (mis)interpreted as being racist or that is at least somehow connected to racism. Accordingly, we would consider a sentence that has no relation to racism at all (e.g., “the sun is shining”) to be a very unlikely substitute for $\mathbf{x}$ in the given context.

This reasoning can directly be transferred to pretrained language models: Given an input $\mathbf{x}$, explicitly encouraging a model to produce a continuation that does not exhibit some attribute $\mathbf{y}$ will prompt it to generate sentences that are, in some way, connected to $\mathbf{y}$. This direct approach thus has a strong influence on the probability assigned to every single word. In contrast, our self-debiasing approach only modifies the probability of words if they are explicitly considered biased. For two words $w_{1}$, $w_{2}$ that are both not considered biased (i.e., $\Delta(w,\mathbf{x},\mathbf{y})\geq 0$ for $w\in\{w_{1},w_{2}\}$), we have

This follows directly from Eqs. 3 and 4. So the relative probability of two unbiased words $w_{1}$ and $w_{2}$ is not affected by self-debiasing at all.

2 Limitations

We discuss limitations of both our evaluation and of the proposed self-diagnosis and self-debiasing algorithms themselves.

One major limitation of our evaluation is that it relies to a large extent on attribute scores assigned by Perspective API; this means not only that we cannot thoroughly test the effectiveness of our method for many relevant biases that are not measured by the API, but also that our labels are error-prone. For example, Perspective API may fail to detect more subtle forms of bias and be overreliant on lexical cues (Gehman et al., 2020). While our complementary human evaluation mitigates this issue to some extent, crowdsourcing comes with its own downsides. In particular, untrained crowdworkers classify examples based on their own biases and personal perceptions; our setup does not involve critical communities who have contextual knowledge, represent social justice agendas and have reasonable credibility in establishing the presence or absence of undesired attributes. CrowS-Pairs covers a larger set of social biases and is based on human-labeled data, but it is a comparatively small dataset that, for some bias categories, contains only a few dozen examples.

In future work, we thus plan to extend our analysis to other datasets that more directly and reliably measure the extent to which pretrained language models exhibit certain kinds of bias. Towards this goal, we plan to move beyond definitions developed by social media corporations and fine-tune attribute descriptions through people-centric processes involving critical intermediaries such as fact checkers and anti-hate groups who possess cultural knowledge of particular linguistic-political contexts and dynamic ways in which toxic expressions keep evolving (see Udupa, 2020; Udupa et al., 2021). This is critical for ensuring that attribute descriptions and labels acquire sufficient cultural and dynamic knowledge to remove bias as well as that we do not leave the task of determining what is offensive and what is not only to corporations. However, the advantage of what we have proposed here lies in the scalability it provides to different processes of attribute description and labeling. This means that the contextually rooted process of involving community intermediaries to develop textual descriptions of undesired attributes and assign priorities for bias detection can directly benefit from the scaling up made possible by our proposed solution. Finally, our evaluation is also limited to the English language and to only a small subset of available language models; future work should look into other languages and models.

As for the limitations of self-diagnosis and self-debiasing, both algorithms rely on simple templates and attribute descriptions; as our experiments in §3.3 show, modifying templates and descriptions can – in some cases – result in quite different self-diagnosis performance. In addition, finding descriptions that are well understood by current generations of language models may be inherently difficult for some forms of bias. We also find that the proposed self-debiasing algorithm is often overly aggressive in filtering out harmless words that do not really contribute to undesired bias in the generated sentence. While this leads to increased perplexity on Wikitext-2 for large values of $\lambda$ (see Table 2), our human evaluation carried out in §4.1 shows that it does not hurt the fluency or coherence of generated texts. Nevertheless, we believe that developing self-debiasing approaches that perform at least as well with regards to dropping undesired behaviors while maintaining perplexity comparable to regular decoding is an important direction for future work.

We also note that our self-debiasing algorithm is inherently greedy in that decisions for or against a particular word must always be made while only considering its already generated (i.e., left) context. A word that may seem undesirable when only considering its left context may very well be unproblematic once its entire context is taken into account. To some extent, this problem can be alleviated through beam search. Finally, it should also be noted that the decoding time of our proposed algorithm increases linearly in the number of attributes for which self-debiasing is to be performed because a separate self-debiasing input must be processed for each such attribute. This can be problematic in use cases where it is necessary to eliminate a large number of undesired attributes simultaneously.

3 Ethical Considerations

Not least because of the limitations discussed in §5.2, our self-debiasing algorithm in its current form is not able to reliably prevent current generations of language models from exhibiting undesired biases or showing toxic behavior – it can merely reduce the probability of this happening for the selected models and on the selected datasets. It should therefore by no means be used as the sole measure to reduce bias or eliminate undesired behavior in real-world applications.

It would be well beyond the scope of this paper to attempt to make decisions on which behaviors and social biases should be avoided by language models. However, we consider it an advantage of our approach that the responsibility for a model’s behavior no longer lies exclusively with its initial developer: Self-debiasing provides an interface to users of a language model that allows them to explicitly set the desired behavior for concrete use cases. For example, there may well be text genres that contain violent language for legitimate purposes (e.g., crime fiction) and in that case, our method allows the user to specify a policy that does not affect violent language, but reduces other undesired attributes. The ability of specifying a policy will be especially beneficial for critical community intermediaries since this feature allows them to explicitly set the undesired attributes.

In this paper, we have shown that large language models are capable of performing self-diagnosis, i.e., of investigating their own outputs with regards to the presence of undesirable attributes using only their internal knowledge and textual descriptions. Based on this finding, we have proposed a decoding algorithm that reduces the probability of a model generating biased text by comparing the original probability of a token with its probability if undesired behavior is explicitly encouraged.

As our evaluation is limited to two English datasets covering only a small portion of potentially undesired behaviors in an imperfect fashion, it is important to extend our analysis to other kinds of behaviors and biases, languages, benchmarks and models.

It is clear that self-diagnosis and self-debiasing only reduce and do not eliminate corpus-based bias. For this reason, they are not a viable path towards bias-free models if used in isolation. However, we hope that future work can leverage our proposals, e.g., by combining them with complementary models or by extending them to build stronger debiasing solutions.

This work was funded by the European Research Council (ERC #740516 and #957442) under the European Union’s Horizon 2020 research and innovation programme. We thank the anonymous reviewers and the action editor for their helpful comments.