Mitigating Gender Bias in Distilled Language Models via Counterfactual Role Reversal

Introduction

The ever-increasing size of language models (LMs) have increased their energy and compute requirements, making them impractical for many real-time resource-constrained applications such as personal assistants deployed on edge devices. To address this issue, various approaches have been proposed to compress or distill these large models (e.g., Sanh et al. (2019); Jiao et al. (2020); Hinton et al. (2015)). However, distillation techniques are designed to mimic the uncompressed LM (i.e., teacher model). Thus, the societal biases encoded in the teacher models Bender et al. (2021); Bommasani et al. (2021); Sheng et al. (2021) will propagate to the distilled models. In fact, our experiments show that distilled models are adjudged to be more unfair than their teacher model counterparts. In this work, we devise techniques to train models that mitigate societal biases during knowledge distillation.

One way to demonstrate this manifestation of societal biases is by looking at text generated by LMs, as illustrated in Fig. 1. As such, the output text focuses on different characteristics of the person, solely based on which gender is mentioned in the context. To this end, we focus on reducing the disparity between groups during the language generation, considering the fairness definition for open-ended text generations as proposed in Dhamala et al. (2021) and Sheng et al. (2019). We propose an approach that uses counterfactual role-reversed sentences during knowledge distillation. In other words, our approach uses counterfactual texts that are generated by substituting mentions of one demographic group with the other. We employ an automated way to generate these counterfactuals, requiring only a paired list of words from each demographic group.

Typical knowledge distillation training loss has two components: (a) the LM training loss such as cross-entropy to learn information from the training data, and (b) a loss that enforces similarity between outcomes of teacher and student modelsThe teacher model refers to the original LM, and the student model refers to the LM being trained. The latter usually has fewer parameters.. The counterfactual knowledge is used to correct these loss components in the following ways: (a) augmenting the training set itself, which alters the training loss to learn from more equitable data; and (b) modifying the teacher’s output toward more equitability so that the student learns from a more equitable output distribution.

We first demonstrate our method using English GPT2–small Radford et al. (2019) as the teacher and a 6-layer GPT–2 (called DistilGPT–2) as the student model. We focus on binary gender disparities (male vs. female) and use the gender polarity metric for profession prompts from the BOLD dataset Dhamala et al. (2021) as the primary fairness definition. We show that our approach lowers the gender disparity in the generated text. Next, we demonstrate the applicability of our approach for finetuning English GPT2–small, i.e., using the same architecture for teacher and student models in the distillation framework. Finally, we evaluated the resultant model’s gender fairness on downstream tasks such as Contextual Embedding Association Tests (CEAT) Caliskan et al. (2017) and finetuning on Bios–Bias classification task De-Arteaga et al. (2019). We find that reduced disparity in open-ended text generation does not necessarily lead to fairness on other tasks.

Related Work

Large LMs embody societal biases that could result in harms such as misinformation, stereotype propagation, and disparate resource allocation Bender et al. (2021); Sheng et al. (2021). Multiple studies have shown that LMs are biased in producing outputs with negative connotations such as toxicity Gehman et al. (2020); Zhou et al. (2021); Xu et al. (2021) and negative regard Sheng et al. (2020, 2021) towards minority populations. Others have shown that LMs encode prevalent gender biases, such as one gender being more associated with a particular class of professions. Such biases can be revealed via contextual embedding tests Guo and Caliskan (2021), stereotype tests Sap et al. (2020); Nangia et al. (2020), and evaluation of generated texts Dhamala et al. (2021); Sheng et al. (2019). Few works have also shown that LM can be biased towards ideologies, e.g., Islam Brown et al. (2020).

Approaches to mitigate bias in LMs can be broadly summarized as: (a) training or finetuning on a balanced dataset Solaiman and Dennison (2021); Dinan et al. (2020)), (b) attaching prefix at inference or training time Sheng et al. (2020), and (c) using a bias or attribute classifier (e.g., toxicity classifier) to control fairness in text generation Dathathri et al. (2020); Liang et al. (2021); Liu et al. (2021); Krause et al. (2021). While all these debiasing approaches can be used to mitigate bias in an LM after it is distilled, no prior work aims to directly debias and distill in a single step. Furthermore, the majority of existing approaches focus on reducing toxic text generation Solaiman and Dennison (2021); Dathathri et al. (2020); Liang et al. (2021); Liu et al. (2021); Krause et al. (2021). Different from existing works, we present an approach for fair knowledge distillation that aims to mitigate gender bias in text generated from the distilled models.

Our approach is inspired by the counterfactual notion of fairness Kusner et al. (2017) and introduces two modifications to the standard distillation: (a) counterfactual data augmentation, and (b) using modified teacher probabilities. Counterfactual fairness and related notions have been previously used for bias mitigation in hate speech detection Mostafazadeh Davani et al. (2021), word embeddings Hall Maudslay et al. (2019); Lu et al. (2020); Zhao et al. (2018b), and coreference resolution Zhao et al. (2018a) tasks. Ours is the first work that uses counterfactual knowledge to achieve equitability in text generation during distillation. Our method is also applicable when the student model or architecture is the same as the teacher model, and we have demonstrated it via experiments.

Notion of Language Model Fairness

We focus on mitigating gender bias in open-ended language generation from an LM. The bias is measured by assessing the tendency of the LM to associate a specific set of professions to a specific gender, e.g., healthcare professions to female and engineering professions to male. As discussed in Sheng et al. (2021), such societal biases may cause a negative representational impact by propagating stereotypes, misrepresentations, or denigrations of social groups. We consider only binary gender in this paper as LMs often do not encode sufficient representation of non-binary gender context, restricting a meaningful analysis Dev et al. (2021). We use a related counterfactual notion of fairness, commonly studied in the NLP fairness literature, to motivate our fair distillation approach in Sec. 4. The counterfactual notion of fairness Kusner et al. (2017) adjudges a model fair if it generates similar predictions before and after swapping the sensitive features in the input.

Fair Knowledge Distillation via Counterfactual Role Reversal

In typical knowledge distillation, a smaller student model, imitating the behavior of the large teacher model, is obtained by using additional training signals from the target probabilities output by the teacher model. Let $\{x_{1}\ldots x_{m}\}$ denote sequence of text tokens in a training sample, $x_{<t}$ or $\{x_{1}\ldots x_{t-1}\}$ denotes sequence of tokens prior to $t$ and boldface denote random variables. LMs such as GPT–2 model probability distribution of next token $P(\mathbf{x}_{t}|x_{<t})$ over the vocabulary $\mathcal{V}$, i.e., $x_{t}\in\mathcal{V}$. Distillation loss is then defined as follows:

This loss consists of two terms: (a) the cross-entropy (CE) between the predicted next token probability and the observed token, and (b) the KL-divergence between the output probabilities from the teacher ($P_{teacher}$) and the student ($P_{\theta}$) models. The KL-divergence term provides a stronger training signal to the student, leading to more accurate and faster learning Hinton et al. (2015).

Knowledge distillation (Eq. (1)) will also transfer societal biases while transferring information from the teacher model. To address this problem, we propose to infuse the bias mitigation strategy with knowledge distillation to obtain a less biased and compact model. Our bias mitigating strategy is based on the intuition that given a sequence such as ‘She works as a’ and its counterfactual ‘He works as a’, a fair LM should generate similar texts. We materialize this intuition by encouraging student LM to learn similar distribution of probabilities for a sequence of tokens and its counterfactual.

To this end, we propose two modifications to the base distillation strategy: (a) Using counterfactual role reversal to modify token probabilities of the teacher model; and (b) Using counterfactual role reversed data for model distillation. We study these two modifications independently and in various combinationsOur approach may use the same student model as the teacher, as we demonstrate in Sec. 5..

Given a sequence of tokens referring to a particular demographic group, we want to generate a counterfactual sequence of tokens referring to another related demographic. For example, suppose the original text, referring to the female group was ‘She is a mother of two kids and works as a software engineer,’ we want to generate a counterfactual referring to the male group ‘He is a father of two kids and works as a software engineer.’ Inspired by existing works on counterfactual data augmentation for binary gender Lu et al. (2020); Hall Maudslay et al. (2019), we use word-swapping operations on the sequence of tokens to generate counterfactual sequences. Specifically, we use a curated dictionary of gender words with male $\rightleftharpoons$ female mapping, for instance, father $\rightarrow$ mother, she $\rightarrow$he, him$\rightarrow$her, etc. We generate a counterfactual sequence of tokens from the original sequence by substituting the gendered word in the original sequence with a matching gendered word referring to the opposite gender from this dictionaryWe found 96% of the generated data on manual analysis to be correct (See Appendix B.4 for details). . See Appendix B for the curated dictionary sources and other implementation details.

2 Modifying Teacher Probabilities

Next, we discuss how to use counterfactual sequences to modify knowledge distillation loss. In an open-ended language generation task, the LM produces a natural continuation of text given some context or a prompt ($x_{<t}$). To this end, auto-regressive LMs such as GPT–2 predict the probability distribution of the next token given the context and previously generated tokens. The next token is sampled from the predicted distribution and added to the context to generate text. This process is continued until a stopping criterion is met. Depending on the gender present in the context, the teacher model may produce different probability distributions over the vocabulary. If these predicted distributions are directly used for student model training, it could transmit gender bias in the student model.

To mitigate this unchecked transference of gender disparity, we modify the teacher probability of each token by using the next token probabilities from both the original and the counterfactual context (or both genders) during student model training. We combine them to boost the probability of more likely tokens with both genders while the probability of less likely tokens with one or both genders being suppressed or relatively unaffected (See Fig. 2 for a visual illustration). We experiment with different functions to combine these distributions. Let $z_{t}=\log P(\mathbf{x}_{t}|x_{<t})$ and $z_{s}^{\prime}=\log P(\mathbf{x}_{s}|x_{<s})$ are the log-probability distributions (or logits) for the original and the corresponding counterfactual context, respectivelyDue to sub-word tokens, the index of corresponding tokens in the original and counterfactual text may be different. We use index variable $s$ to denote the corresponding token in the counterfactual sentence, indexed at $t$ in the original sentence.. The new unnormalized logits ($z^{\prime\prime}_{t}$) are obtained with max, mean, expMean, or swap operation and illustrated in Table 1. We normalize $z^{\prime\prime}_{t}$ so that it is a valid log distribution.

Intuitively, the max operation would preserve the most likely tokens among either context. The mean is similar to taking the product of the two distributions, thereby increasing the likelihood of words that were more likely in both cases and lowering the likelihood of any other words. One may also consider any weighted combination of $z$ and $z^{\prime}$. Infact, the swap operation is an extreme case of a weighted combination with the weight of original logits (i.e., $z_{t}$) being 0. Finally, expMean is the average of two distributions. Our approach is reminiscent of post-processing approaches that modify the next step probabilities during inference. However, we adapt it here for gender fair-knowledge distillation and use this procedure during training.

3 Counterfactual Data Augmentation

Using modified probabilities to update the student model rectifies the probability for the tokens generated after the gendered word. However, it only provides a weak signal by changing the log probabilities, and the training data may contain biases, which the student model can learn via cross-entropy loss (See Eq. (1)). To this end, we also augment counterfactual data to the training set. Counterfactual data augmentation has been successfully used for gender bias mitigation in various downstream tasks such as static word embedding training Hall Maudslay et al. (2019) and co-reference resolution Lu et al. (2020). However, it has not been explored in knowledge distillation or fair LM training for open-ended language generation. Therefore, we also experiment with counterfactual data augmentation combined with the proposed next-token logit update strategy.

We refer to our approaches as Equitable Role Alteration (ERA). Primarily, the logit modification approach reduces bias in the teacher model’s predicated probabilities, thus affecting only the KL divergence component. By contrast, counterfactual data augmentation involves adding new samples to the training set, affecting both loss components.

Experiments

We use GPT2–small, a 12 layer transformer-based LM comprising of $\sim$124M parameters, as the teacher model and a six-layer version of GPT–2 as the student model. We use OpenWebText corpus, which is an open-source reproduction of WebText corpus that was used to train GPT–2 in Radford et al. (2019). Due to limitations in computational budget, we use 10% of the corpus for training. We used the knowledge distillation procedure presented in Sanh et al. (2019), but without the cosine loss between representations during knowledge transfer because adopting knowledge distillation for fair learning requires correcting the ‘biased knowledge’ from the teacher, but it is hard to amend biased contextual representations. This approach can also be used for fair finetuning of an LM by using the same teacher and the student model. In that case, one may initialize with the pre-trained teacher’s weights. For fair finetuning experiments, we use GPT2–small as both the teacher and the student. Details on training, text generation, and hyperparameters are provided in Appendix D.

2 Evaluation of Open-ended Generation

We assess gender fairness in English text generation by evaluating the bias of an LM to associate a gender with gender-neutral professions during open-ended text generation. For this, we use the profession prompts and gender polarity metrics from BOLD Dhamala et al. (2021). These prompts are 10,195 sentence beginnings extracted from the Wikipedia articles and refer to 18 different profession categories such as engineering, healthcare, arts & entertainment, etc. Some examples of BOLD profession prompts are ‘An animator is an artist who’ and ‘A flight nurse is a registered.’ Texts generated from the LMs with these prompts as contexts are evaluated for gender polarity.

The gender polarity score measures if the text is neutral, female–polar having words such as she, woman, etc., or male–polar having words such as he, boy, etc. It is computed by taking the maximum of the normalized projection of each word vector in the LM generated text onto $\vec{she}-\vec{he}$. The word vectors are computed on the debiased Word2Vec embeddings Bolukbasi et al. (2016)https://github.com/tolga-b/debiaswe. We use a threshold of $0.25$ on the polarity score to label the text as male or female polar. For each profession group, we compute the equitability ratio as $\min\{\frac{m}{f},\frac{f}{m}\}$, where $m$ and $f$ are the numbers of text generations labeled as male and female polar, respectively. The equitability ratio $\in$ with 1 indicating equitable treatment. We report average and min equitability scores across all professions to summarize the disparityWe note that this evaluation is not perfect. Gonen and Goldberg (2019) show that debiased word embedding still reserves some gender information for neutral words..

Perplexity/Fluency.

For real-world applications, an LM should demonstrate high-quality generations along with fair generations. To this end, we report the perplexity of the wikitext-2 test set Merity et al. (2017) as predicted by the trained LM. Similar to Liu et al. (2021), we evaluate the fluency of the completed prompts from BOLD. The fluency is measured as the perplexity of generated text predicted by the GPT2–large model. Lower perplexity and fluency scores are better.

3 Baselines and Other Methods

First, we test the utility of our approach in knowledge distillation compared to teacher and distilled models trained without fairness constraints. We use pre-trained GPT2–small (unfair teacher model) and DistilGPT–2 from the HuggingFace (HF) model repositoryhttps://huggingface.co/models. Since training hyperparameters and dataset used by DistilGPT–2 (HF) is different from ours, we also train a DistilGPT–2 using our setup.

Next, we compare our approach with two gender-bias mitigation approaches by applying them to the distilled version of GPT–2 and GPT2–small from the HF repository. We finetune the distilled models with the counterfactual and original sequences using only cross-entropy loss, which is similar to CDA Lu et al. (2020) and DAPT Gururangan et al. (2020). We also compare with the bias-mitigation approach of Sheng et al. (2020), which searches for adversarial prompts that increase the likelihood of specifically curated fair texts.

4 Results on Open-ended Text Generation

Table 2 summarizes results for gender disparity mitigation in open-ended generation for DistilGPT–2 and GPT2–small. We observe that compared to the teacher GPT2–small model, which has more parameters, the distilled versions (DistilGPT–2) are more biased which is indicated by lower equitability scores. Due to using only 10% sequences for training, our implementation of DistilGPT–2 has higher perplexity than the HF’s version.

Rows 4–7 in Table 2 show results of using only modified teacher logits based on counterfactuals (Sec. 4.2) with various operations. Overall, these modifications improve over the baseline DistilGPT–2 model in terms of equitability ratios with only a slight increase in perplexity. Models trained with expMean, max, and swap scored similar or higher equitability than the teacher model. The mean operation was the least effective at improving fairness. The approach that uses only counterfactual data augmentation (row 8 in Table 2) showed more than $1.5\times$ improvement in equitability while keeping perplexity almost equal to the baseline model (40.93 vs. 40.88). By contrast, the two-step process of creating a distilled model and then finetuning with counterfactual data (using only cross-entropy loss) resulted in a worse perplexity of 41.63 but better equitability. Our approach combining logit modification and data augmentation (rows 9–10, Table 2) provides better equitability among all the models. Compared to the two-step finetuning approach (i.e., distillation then bias-mitigation), it has better equitability with similar perplexity. The adversarial prompt-based approach of Sheng et al. (2020) performs much worse in terms of fairness. One of the reasons for this could be that the adversarial prompts are created to perform well on a small curated dataset which may not generalize. We omitted the perplexity values for this approach as it is not consistent with our evaluation process.

When combining logit modification and data augmentation, we experimented with modifying logits of both counterfactual and original text, and only of the original text. We found that the results with both approaches are similar and report results of modifying both texts in Table 2. The models obtained by combining the counterfactual data augmentation and logit update produce text with very little disparity and achieve the best fairness. Even though the fluency metrics are low, the perplexity for these models is higher. We noticed a high variance in fluency for some of the models. Upon further investigation, we found that the fluency can be very large for one of the profession groups, resulting in a large overall variance during macro averaging. We remark that fluency is at best a noisy measure as it uses an LM to evaluate the outputs; perplexity should be considered a more reliable measure of LM quality. For further evaluations and discussion, we use models trained with the max operation, as the results with the max operation for logit modification, with and without counterfactual augmentation, were most consistent.

Fair Finetuning with GPT–2.

We also experiment with finetuning GPT2–small to train gender-fair models. The approach is similar to finetuning with counterfactual augmented data but employs knowledge distillation loss instead. Table 2 (rows 13–16) summarizes the results for training fair GPT2–small models. Unlike results with distilled models, all the approaches are fairly competitive. We remark that finetuning and our best approach have similar fairness performance, but our approach has better perplexity owing to improved learning due to the additional KL-divergence term.

However, models trained using only data augmentation or logit modification resulted in less equitability. The student model has two loss components—cross-entropy and KL divergence loss. When employing only one of the techniques, the student model may receive training signals from unfair teacher logits in the former case and training data in the latter case, learning less equitable models. We also note that only logit modification with max operation led to worse results in terms of quality and fairness compared to the baseline GPT–2 model. This could be due to the cross-entropy loss being the dominant training signal, and original training sequences may have spurious gender correlations. The adversarial-prompt approach of Sheng et al. (2020) has lower fluency than other models. On further inspection of generated texts, we noticed that the LM sometimes generates degenerate phrases related to the adversarial prompt instead of the actual prompt about the profession, leading to poor quality generations. Additionally, we did a human evaluation to assess the quality of generated text (See Appendix A). We find the quality of texts generated from our less biased GPT2–small (ERA) to be similar to GPT2–small.

Gender Fairness on Other Tasks

It is often expected that different fairness measures designed for different but related tasks would be correlated. However, recently Goldfarb-Tarrant et al. (2021) found that fairness measures for static word embeddings and downstream tasks do not correlate. To this end, we study if our fair text generation models improve fairness on other tasks.

We evaluate if fairness in open-ended generation by LMs obtained via the proposed method also transfers to the LM’s embeddings using the CEAT metric Guo and Caliskan (2021). The WEAT metric measures the effect size of social bias in a static embedding by computing the relative associations of two sets of target words (e.g., career, office; and home, family) with two sets of attribute words (e.g., girl, woman; and boy, man). CEAT extends WEAT to contextual embedding by computing a distribution of effect sizes, each sample obtained by computing WEAT effect size on contextual embedding computed with a different context. CEAT summarizes the combined magnitude of bias by pooling effect sizes with a random-effects model. We use three CEAT tests that measure gender bias: 1) CEAT test 6 with attributes male/female names and targets career/family, 2) CEAT 7 with attributes male/female terms and target math/arts, and 3) CEAT 8 with attributes male/female terms and targets science/arts. See Appendix D for details.

According to the combined effect sizes metric (known as Cohen’s d), $d>0.5$ and $d>0.8$ are medium and large effect sizes, respectively. However, the absolute effect size is often used as the magnitude of bias Goldfarb-Tarrant et al. (2021)P-values are not reported as it does not indicate the magnitude of the bias, and all models were most certainly biased.. As shown in Table 3, baseline models have a larger effect size in tests 6 (male/female names and career/family) and 7 (math/arts and male/female terms). In test 8 (male/female terms and science/arts), there was not a strong bias in the embeddings of baseline models. Overall, we observe that the demonstrated fairness in LMs for open-ended language generation in Sec. 5 is not always reflected in the embeddings. For example, the model trained using modified logits based on max operation has a smaller absolute effect size for tests 6 and 7 but higher for test 8 compared to the baseline. Effect sizes on tests 7 and 8 have reduced when using the counterfactual data augmentation method, but it increased on test 6. Hence, the LM embedding fairness metric CEAT did not correlate with the fairness of LM in open-ended text generation tasks. This finding agrees with Goldfarb-Tarrant et al. (2021), but for contextual embeddings. They observed that downstream fairness measures and static embeddings are not correlated.

2 Fairness in Classification Task

We evaluate the hypothesis that an LM that is less biased in text generation should be less biased on downstream tasks by finetuning various baselines and fairer versions of LM obtained in Sec. 5.4 on the Bios–Bias classification task (De-Arteaga et al., 2019) and evaluating the classifier’s fairness. The objective is to predict one of the 28 profession classes from a person’s biography. We use a weighted combination of all token embeddings with a linear layer for classification. Pre-trained weights are not updated. For training details, see Appendix D. Similar to De-Arteaga et al. (2019), we take the average true positive rate difference (TPRD) between males and females across all professions as the fairness measure.

A fair model should have a similar true positive rate for both genders, i.e., TPRD $\sim$ 0. However, we observe from Table 3 that TPRD is around $0.1$ for all the models, indicating that all models lead to equally unfair outcomes. De-Arteaga et al. (2019) presented a simple debiasing technique of removing a set of predefined gendered words (such as he, she, mrs.) from the biographies before training, which resulted in an accuracy of $0.815$ and TPRD of $0.0658$ with DistilGPT–2 as the pre-trained model. Overall, this suggests that our method, even though effective in reducing disparity for open-ended text generation, is not adequate for this downstream task.

Discussion and Limitations

We conducted preliminary experiments to test if the proposed approach can be extended to different race groups. Similar to Dhamala et al. (2021), we consider race bias manifested via people’s names and race-specific tokens across four races common in the US: African, European or White, Hispanic & Latino, and Asian. We construct a many-to-many mapping that maps words referring to a given race to words referring to the other races for the counterfactual generation. The rest of the method remains the same as Sec. 4. For fairness evaluation, we use race prompts from BOLD and regard classifier from Sheng et al. (2019), which evaluates whether the person in the text is portrayed as being ‘highly thought of.’ Results show that the LMs obtained with the proposed approach were less biased in treating different races similarly, indicating that the proposed approach can be extended to other non-binary groups. However, the improvements were not as significant as gender bias mitigation, leaving plenty of scope for improvement left for future work. We describe the results and experiments in more detail in Appendix C.

Counterfactual data generation.

Dictionary-based word-swapping is a simple and effective method for counterfactual generation Lu (2020); Zhao et al. (2018a). However, blind word swapping can also result in factually and/or grammatically incorrect texts. To quantify these errors, we manually evaluated 500 randomly sampled counterfactual texts for gender category. We found that 22 $(4.4\%)$ of these sentences were incorrect (See Appendix B.4). In this paper, we demonstrate that despite counterfactual data generation not being perfect, it can effectively reduce the gender biases in the model. We expect our bias mitigation approach to benefit from further research in counterfactual data generation, especially for reducing race disparity.

Conclusion

We proposed techniques to use counterfactual information during knowledge distillation to mitigate gender bias in LMs. In experiments, we show that this approach improves fairness in text generation, but it does not simultaneously enhance fairness on LM embedding and downstream classification task. LMs have become the Swiss army knife of NLP because modeling next word probabilities can learn versatile models that are effective on many tasks. It was surprising that reducing gender disparity in text generation had little effect on other downstream tasks. This finding underscores the importance of evaluating LM fairness along multiple metrics and tasks.

Broader Impact and Ethics Statement

As language models become prominent, it is imperative to understand and mitigate various harms that they may provoke Solaiman et al. (2019); Bommasani et al. (2021). Moreover, to make language processing resource-efficient, more focus should be on achieving good performance with smaller models. Our work is a step towards mitigating such damages but not the only remedy possible. We demonstrated effective ways to incorporate counterfactual knowledge during training to avoid a two-step training process. The resulting model generates less disparate text for different groups while being equally or more accurate. However, as we have discussed in Sec. 6, this does not make the model fair with regards to other gender fairness measures. Our results essentially echo the argument made in Barocas et al. (2019) that it is meaningless to ascribe fairness to a model. Instead, fairness should be thought of, keeping the task and outputs in mind. This work in mitigating fairness is limited because we only focus on biases in English language generation. Other works, such as Zmigrod et al. (2019), have identified the difficulties in transferring these approaches to other languages. Moreover, we have considered binary gender, which does not capture all the real-world complexities. More critically, our assessment of fairness for open-ended text generation has relied on fair definitions and measures from Dhamala et al. (2021) and Sheng et al. (2019). One should interpret the results with this in perspective. Some recent works, such as Blodgett et al. (2020, 2021); Gonen and Goldberg (2019), have demonstrated critical flaws in other fairness measures. For example, Blodgett et al. (2021) found that benchmark datasets designed for measuring stereotyping behavior of LMs such as StereoSet Nadeem et al. (2021) and CrowS-Pair Nangia et al. (2020) are ambiguous and have several pitfalls which can even operationalize stereotyping. Our approach uses counterfactual data, which may inherit the flaws in original data or introduce new errors. Users should use appropriate filters/mechanisms to ensure the quality of counterfactual data used for training.

Finally, we propose approaches to create less biased LMs. However, similar to how gifts were used as weapons in Le Guin’s Gifts Le Guin (2006), our approach can be repurposed to cause even more disparate treatment. For example, one may remove the mention of a specific race or gender completely from the training set to create a dystopian LM that does not acknowledge that group or entity’s existence or the inaccuracy of counterfactual generation may cause LM to learn from fictional and non-grammatical texts. Nevertheless, we hope that our work will inspire more good than harm.

References

Appendix A Human Evaluation of Generated Text

We evaluate the quality of text generated from GPT2–small, fair-GPT2–small (ERA), and Sheng et al. (2020) (adversarial prompt method with GPT2–small). We randomly sampled 300 prompts and their corresponding text generations from all three models. We then asked annotators to annotate for two tasks. The first task was to rank the generation quality among three sentences generated with the same prompt. The labels for the ranking task were: 1 – Worst, 2 – Medium, and 3 – Best. The second task was to rate the generation quality on a scale from 1–6 — 1 being very poor, 2 being poor, 3 being fair, 4 being average, 5 being good, and 6 being excellent. Unlike the ranking task, the ratings are independent of generations from other models for the same prompt. When rating the quality, we asked the annotators to focus on the following properties of the text.

Is the text consistent and coherent? Is the generation meaningful?

Could the text have been written extracted from news, books, etc.?

Could the text have been written by a Human?

We also provided some example annotations, as shown in Table 4.

The four annotators participating in these tasks are volunteers proficient in English, originating from various countries but presently or in the past studied/worked in the US, and familiar with language models. The annotators were informed of the research problem. We followed our institution’s review process and approval guidelines for these annotation tasks. For each sentence, we collected three annotations. We only keep the ones where at least two annotators agree out of all annotations.

The mean and standard deviation of rankings for generations from GPT2–small, fair GPT2–small, and Sheng et al. (2020) were $2.55\pm 0.55$, $2.34\pm 0.64$, and $1.12\pm 0.41$, respectively. Text generated from GPT2–small is ranked highest most of the time. However, the fairer GPT2–small obtained with our method is a close second. The average ratings for generations from GPT2–small, fair GPT2–small (ERA), and Sheng et al. (2020) were respectively, $3.01\pm 1.04$, $2.707\pm 1.07$, and $1.12\pm 0.41$. Consistent with the ranking results, GPT2–small received the highest rating, followed closely by the generations from fairer GPT2–small obtained with our method. Both ranking and rating results indicate that our approach retains most of the performance while reducing gender disparity in the generated text. We find that Sheng et al. (2020) resulted in low-quality generations. As also discussed in the main paper, this could be because the adversarial prompts are designed to increase the likelihood of specially curated fair text and may not work for diverse prompt datasets like BOLD, which contains diverse sentences beginning from various Wikipedia articles. Moreover, we also noticed that the adversarial prompts could lead to generation unrelated to the actual prompt and generate text referring to phrases in the adversarial prompt instead. We provide some example text generations from these approaches in Table 5.

Appendix B Counterfactual Role-Reversal Data Generation

Counterfactual sequences were generated for $\sim 78\%$ and $\sim 65\%$ of the training sequences for gender and race domain experiments, respectively. We limit sequence lengths to 1024 for training. We generate one counterfactual sequence for every sequence in the training set that has words matching with our lists and referring to the demographic groups. The word lists are described next.

To generate counterfactual texts for gender disparity experiments, we create mappings between male-to-female words and vice versa using word lists from Zhao et al. (2018a)Specifically, we use word lists available at https://github.com/uclanlp/corefBias/blob/master/WinoBias/wino/extra_gendered_words.txt, and https://github.com/uclanlp/corefBias/blob/master/WinoBias/wino/generalized_swaps.txt. We consider some additional words to mappings derived from the above lists, shown in Table 6.

B.2 Race Word Lists

We focus on four US-specific races: Asian-American, Hispanic & Latino-American, European-American, and African-American. To create counterfactual text for mitigating racial disparity, we use word sets from different categories. Table 7 shows the word sets we have used. We process and use these word sets as follows.

For words in the country and race category, we append ‘ American’ and ‘-American’ and their equivalent lower case versions and consider these as the actual word sets. Similarly, we consider both capital and lower case variations of the country and race terms.

For words in the color category of Table 7, we use both capital/lower cases and singular/plural versions.

We use two indicators of Latin race ‘latino’ and ‘latina’ and swap them with words from Asian-, African- & European- American countries word sets but not vice versa.

We created the list of first names from Tzioumis (2018). They provide prominent first names and the percentage of times this name belonged to a particular race. We use names that are 100% of the time assigned to a particular race and that are in the top-100 names for each race. We use the capital case version of the first names.

We collected the list of common last names from Comenetz (2016) and used the capital case version. Other works have also used names as the indicator of race Mishra et al. (2020); Caliskan et al. (2017).

We replace the word from a specific row and column with words from other columns in the same row randomly to create a counterfactual text. For example, the original text, ‘With each new location, Vazquez and Maritza must maintain the quality their fans have come to associate with the brand.’ is converted to ‘With each new location, Banks and Maritza must maintain the quality their fans have come to associate with the brand.’. Similarly, in the case of gender, the text ‘Your father was a drummer in a rock band?’ is converted to ‘Your mother was a drummer in a rock band?’.

B.3 Note about Sub-word Embeddings

We use counterfactual text in two ways, as described in Sec. 4. Due to sub-word embeddings, the length of the counterfactual sequence may not be the same as the original. This is particularly problematic for modifying probability distribution as we have to know the exact location of the corresponding token in the counterfactual and original sentence. To this end, we generate ‘counterfactual token sequences’ during training instead of ‘counterfactual sentences’. We first create tokenized versions of word lists, i.e., a set of tokens representing a word (e.g., father is represented by $\{2988\}$) are mapped to another set of tokens (e.g., mother is represented by $\{2802\}$). Given a sentence such as ‘Your father was a drummer in a rock band?’, it is first tokenized as $\{7120,$ $2988,$ $373,$ $257,$ $34269,$ $287,$ $257,$ $3881,$ $4097,$ $30\}$ then converted to $\{7120,$ $2802,$ $373,$ $257,$ $34269,$ $287,$ $257,$ $3881,$ $4097,$ $30\}$ (‘Your mother was a drummer in a rock band?’).

Also, depending on where and how the word occurs, it can be tokenized differently. To illustrate, consider the word ‘he’ in the next sentence. ‘He should have arrived, but he has not arrived yet’. Clearly, the word ‘he’ appears in two different forms — capital-case and lowercase. Other forms are also possible. Also, GPT–2 tokenizer often has white space at the beginning of the token in its vocabulary. For this reason, we considered the word and some of the possible variations that can occur in the text. The next example best explains these variations. If the word were ‘he’, we use following variations — he$|$ ␣he$|$␣he,$|$␣he.$|$␣he’$|$␣he”$|$‘he␣$|$“he␣$|$He␣$|$‘He␣$|$“He␣.

B.4 On Limitations and Correctness of Counterfactual Sentences

For counterfactual data generation, we use a dictionary-based word-swapping approach. Such a naive approach has some obvious limitations as it does not guarantee the grammatical and factual correctness of the generated sentences. However, we hypothesize that while this approach can potentially generate incorrect data for some examples, overall, it is still a simple yet effective method to generate counterfactual data. In order to verify our hypothesis, we randomly sampled 500 sentences from the generated counterfactual data for gender category and analyzed these for correctness. Out of these 500 sentences, we found 22 $(4.4\%)$ incorrect sentences. Most of the errors are related to incorrect pronoun references, such as a male name being used with ‘she’ as a reference. One such example is ‘Onelki Garcia had another interesting outing as she only allowed 1 hit, but did walk three and lasted just 2.2 innings.’

We emphasize that the main focus of the paper is not to generate better counterfactual data but to show that counterfactual data can be used to mitigate bias effectively during knowledge distillation. We expect our proposed approach to further benefit from advances in counterfactual data generation.

Appendix C Mitigating Racial Disparity

While not the main focus of this study, we also conducted experiments to mitigate race bias, manifested towards the names of people from various races and certain race-related phrases/words. Since we consider more than two races and there is no one-to-one mapping between names, we cannot use the same one-to-one substitution rule for counterfactual data generation as earlier in this case. Hence, we construct a many-to-many mapping that maps multiple words in a given race to multiple words in the remaining races. For each word in the sequence of tokens referring to one race, we substitute it with a randomly chosen word from the corresponding words-set from another race. Additional details and dictionaries used for counterfactual sentence generation are in Appendix B.

Racial Fairness Measure.

We use race prompts from the BOLD Dataset to measure racial disparity and consider four races — Asian American, European American or Whites, African American or Blacks, and Hispanics & Latin Americans. We use the regard classifier to measure regard for each race. The regard classifier has three categories — positive, negative, and neutral regard. Intuitively, the regard classifier measures if sentences cause group A to be more highly thought of than group B. If this is the case, then the language model perpetuates bias towards group B Sheng et al. (2019). To this end, we measure the ratio of positive and negatively regarded sentences for each racial group. A fair LM should have the same ratio for all the races. We report the variance across groups for each model to capture this intuition, and lower variance would imply more fair treatment. We also report the fraction of generated sentences labeled as having positive, negative, and neutral regard.

Result.

Table 8 shows the result of mitigating racial disparity in text generation with our proposed approach that exploits counterfactual data. We generated counterfactual data for this purpose by replacing mentions of one racial group with the other (see Appendix B for details). The baseline pre-trained models from Hugging-Face have consistently higher regard ratios than the baseline model we trained, indicating that they generated more positive regard than our models. However, these have more variance across groups, indicating more disparate treatment in terms of regard.

We note that our counterfactual mitigation approach using both logit modification and augmentation is promising for reducing different regard to different races, but the improvement is not substantial. This could be due to our simple counterfactual generation implementation since we randomly replace race-related words. We replace first and last names independently, which could create mismatched names. There has been some work on improving counterfactual sequence generation and studying its effects, such as Maudslay et al. (2019). The authors show that techniques such as name pairing based on frequency can improve the effectiveness of counterfactual data. Another issue could be that we have focused on races in the American context, but the text sequences referring to another context (such as Indian or Asian contexts) can be mistakenly used to create counterfactuals. A better approach should identify and filter such texts. Finally, even though names have been used as indicators of race in our work and previous work, this may be a relatively poor indicator of race. Especially to identify races in the American context only compared to gendered words identifying gender roles leading to suboptimal results. We leave these explorations for future work.

Appendix D Training and Evaluation Details

We started with the knowledge distillation setup of Sanh et al. (2019)https://github.com/huggingface/transformers/tree/master/examples/research_projects/distillation and tailored it to our requirements. We did not use the cosine loss between the representation. We assigned equal weights of 0.5 to LM loss and KL divergence term with a temperature of 2.0. We only use 10% of the OpenWebText sequences. All the models are trained using HuggingFace Wolf et al. (2020) and PyTorch Paszke et al. (2019) for three epochs with a learning rate of $10^{-3}$, AdamW optimizer, and a batch size of 1600. We use DeepSpeed Rasley et al. (2020) for distributed training using 8 V100 GPUs. One epoch took between 5–8 hours.

We used DistilGPT–2, which had six layers, an embedding size of 768, and 12 attention heads as the student model. We initialize student models with weights from the even-numbered layers of the teacher model, i.e., pretrained GPT2–small. When using GPT2–small as the student, we initialize with the pretrained GPT2–small.

For finetuning with counterfactual text baseline, we use the same training hyper-parameters as above but set the weight of KL divergence term to 0, and LM loss weight is set to 1. For DistilGPT–2, we initialize with DistilGPT–2 (HF) parameters instead of GPT2–small. This is because we will first distill the model and then finetune for fairness in an actual fair-finetuning setup. However, we remark that this model is slightly advantaged compared to our approach in terms of performance (perplexity). Unlike our ERA models, which only use 10% of text sequences from OpenWebText, it was distilled using all the data. For GPT2–small experiments, we initialize with the parameters of pretrained GPT2–small.

For adversarial prompts baseline of Sheng et al. (2020) and GPT2–small, we use the adversarial prompt for man/woman condition from their paper (Appendix A, Table 5 in their paper). We use their official implementation for DistilGPT–2 experiments to find the adversarial prompt with bias mitigation setting. We set disassociation and association loss to 1 and use “The man” and “The woman” as the demographics. The adversarial prompt found was “ genomes genomes Parables Nutrition Nutrition Mathematics”.

D.2 Language Model Evaluation

We use top-$p$ sampling Holtzman et al. (2020) with $p=0.9$ and consider the top 10 sequences for all text generation experiments. We limit the max length of the generated sequence to 100.

Perplexity & Fluency.

Perplexity is measured as the exponentiated average negative log-likelihood of a sequence. Given a token sequence, $X=\{x_{0},x_{1},\ldots,x_{m}\}$, the perplexity of $X$, $ppl(X)$ is,

GPT–2 is a fixed-length model with a max length of 1024. For this reason, we compute perplexity in chunks of length 1024 and stride of 512. We define fluency as the perplexity measured by GPT2–large with stride size 0.

D.3 Bios–Bias Training and Evaluation

We finetune language models on Bios–Bias task for 20 epochs with a batch size of 256, $10^{-3}$ learning rate, and AdamW optimizer. Similar to De-Arteaga et al. (2019), we use a 65–10–25 split of the dataset for training, validation, and testing. We use the validation set to pick the best model for evaluation. We do not update the pretrained language model weights during finetuning and use a weighted combination of all the embeddings. These weights are computed using attention. More specifically, we employ a learnable vector to do a dot-product with resulting embeddings (last-layer output or output before the decoder layer). The dot product result is normalized using softmax to compute the weight vector. The weighted combination of the embeddings is passed through a linear classifier to predict the label.

D.4 CEAT Details

We use CEAT Tests 6, 7, and 8. The set of target and attribute words that were considered for each test are shown in Table 9. Each test uses four set of words — X, Y, A, and B. CEAT test works similar to WEAT Caliskan et al. (2017) and first evaluates the difference in association of word $w$ in set X and Y to set A and B by computing difference of average cosine distance as:

The cosine distances are computed between the embeddings. It then computes the difference of difference in association to measure if words in set X and Y are considered differently, i.e.,

This provides an estimate of the absolute difference between the association of embeddings. To evaluate if this difference is significant overall effect size (ES) is computed by dividing with the standard deviation the difference in the association of union of set X and Y (in-sample variance). Intuitively, we measure if the set X and Y have significantly different associations than any other shuffling of $X\cup Y$.

Since we are evaluating contextual embeddings, we will have multiple embeddings for each word based on the context of the word. Therefore, CEAT samples one of the embeddings of the word to compute ES and refers to it as $ES_{i}$. A random-effects model is used to combine results of multiple such sampling. Eventually, the combined effect size (CES) is computed as:

Where $v_{i}$ is the inverse of the sum of in-sample variance and between-sample invariance.

Different contextual embeddings for a word are derived using the random occurrence of that particular word from Reddit. We use the official implementation of CEAThttps://github.com/weiguowilliam/CEAT with N=10000, which is the default in their implementation.