Gender Bias in Neural Natural Language Processing

Introduction

Natural language processing (NLP) with neural networks has grown in importance over the last few years. They provide state-of-the-art models for tasks like coreference resolution, language modeling, and machine translation (Clark and Manning, 2016a, b; Lee et al., 2017; Jozefowicz et al., 2016; Johnson et al., 2017). However, since these models are trained on human language texts, a natural question is whether they exhibit bias based on gender or other characteristics, and, if so, how should this bias be mitigated. This is the question that we address in this paper.

Prior work provides evidence of bias in autocomplete suggestions (Lapowsky, 2018) and differences in accuracy of speech recognition based on gender and dialect (Tatman, 2017) on popular online platforms. Word embeddings, initial pre-processors in many NLP tasks, embed words of a natural language into a vector space of limited dimension to use as their semantic representation. Bolukbasi et al. (2016) and Caliskan et al. (2017) observed that popular word embeddings including word2vec (Mikolov et al., 2013) exhibit gender bias mirroring stereotypical gender associations such as the eponymous (Bolukbasi et al., 2016) "Man is to computer programmer as Woman is to homemaker".

Yet the question of how to measure bias in a general way for neural NLP tasks has not been studied. Our first contribution is a general benchmark to quantify gender bias in a variety of neural NLP tasks. Our definition of bias loosely follows the idea of causal testing: matched pairs of individuals (instances) that differ in only a targeted concept (like gender) are evaluated by a model and the difference in outcomes (or scores) is interpreted as the causal influence of the concept in the scrutinized model. The definition is parametric in the scoring function and the target concept. Natural scoring functions exist for a number of neural natural language processing tasks.

We instantiate the definition for two important tasks—coreference resolution and language modeling. Coreference resolution is the task of finding words and expressions referring to the same entity in a natural language text. The goal of language modeling is to model the distribution of word sequences. For neural coreference resolution models, we measure the gender coreference score disparity between gender-neutral words and gendered words like the disparity between “doctor” and “he” relative to “doctor” and “she” pictured as edge weights in Figure 1(a). For language models, we measure the disparities of emission log-likelihood of gender-neutral words conditioned on gendered sentence prefixes as is shown in Figure 1(b) . Our empirical evaluation with state-of-the-art neural coreference resolution and textbook RNN-based language models Lee et al. (2017); Clark and Manning (2016b); Zaremba et al. (2014) trained on benchmark datasets finds gender bias in these models Note that these results have practical significance. Both coreference resolution and language modeling are core natural language processing tasks in that they form the basis of many practical systems for information extraction(Zheng et al., 2011), text generation(Graves, 2013), speech recognition(Graves et al., 2013) and machine translation(Bahdanau et al., 2014). .

Next we turn our attention to mitigating the bias. Bolukbasi et al. (2016) introduced a technique for debiasing word embeddings which has been shown to mitigate unwanted associations in analogy tasks while preserving the embedding’s semantic properties. Given their widespread use, a natural question is whether this technique is sufficient to eliminate bias from downstream tasks like coreference resolution and language modeling. As our second contribution, we explore this question empirically. We find that while the technique does reduce bias, the residual bias is considerable. We further discover that debiasing models that make use of embeddings that are co-trained with their other parameters (Clark and Manning, 2016b; Zaremba et al., 2014) exhibit a significant drop in accuracy.

Our third contribution is counterfactual data augmentation (CDA): a generic methodology to mitigate bias in neural NLP tasks. For each training instance, the method adds a copy with an intervention on its targeted words, replacing each with its partner, while maintaining the same, non-intervened, ground truth. The method results in a dataset of matched pairs with ground truth independent of the target distinction (see Figure 1(a) and Figure 1(b) for examples). This encourages learning algorithms to not pick up on the distinction.

Our empirical evaluation shows that CDA effectively decreases gender bias while preserving accuracy. We also explore the space of mitigation strategies with CDA, a prior approach to word embedding debiasing (WED), and their compositions. We show that CDA outperforms WED, drastically so when word embeddings are co-trained. For pre-trained embeddings, the two methods can be effectively composed. We also find that as training proceeds on the original data set with gradient descent the gender bias grows as the loss reduces, indicating that the optimization encourages bias; CDA mitigates this behavior.

In the body of this paper we present necessary background (Section 2), our methods (Sections 3 and 4), their evaluation (Section 5), and speculate on future research (Section 6).

Background

In this section we briefly summarize requisite elements of neural coreference resolution and language modeling systems: scoring layers and loss evaluation, performance measures, and the use of word embeddings and their debiasing. The tasks and models we experiment with later in this paper and their properties are summarized in Table 2.

The goal of a coreference resolution (Clark and Manning, 2016a) is to group mentions, base text elements composed of one or more consecutive words in an input instance (usually a document), according to their semantic identity. The words in the first sentence of Figure 1(a), for example, include “the doctor”and “he”. A coreference resolution system would be expected to output a grouping that places both of these mentions in the same cluster as they correspond to the same semantic identity.

The model of Clark and Manning (2016b) has a trainable embedding layer, which is initialized with the word2vec embedding and updated during training. As a result, there are three ways to apply WED: we can either debias the pretrained embedding before the model is trained (written \overleftarrow{\text{WED}}), debias it after model training (written \overrightarrow{\text{WED}}), or both. We also test these configurations in conjunction with CDA. In total, we evaluate 8 configurations as in shown in Table 3.

The aggregate measurements show bias in the original model, and the general benefit of augmentation over word embedding debiasing: it has better or comparable debiasing strength while having lower impact on accuracy. In models 2.7 and 2.8, however, we see that combining methods can have detrimental effects: the aggregate occupation bias has flipped from preferring males to preferring females as seen in the \pm\text{\text{AOB}} column which preserves the sign of per-occupation bias in aggregation.

2 RNN Language Modeling

We use the Wikitext-2 dataset (Merity et al., 2016) for language modeling and employ a simple 2-layer RNN architecture with 1500 LSTM cells and a trainable embedding layer of size 1500. As a result, word embedding can only be debiased after training. The language model is evaluated using perplexity, a standard measure for averaging cross-entropy loss on unseen text. We also test the performance impact of the naive augmentation in relation to the grammatical augmentation in this task. The aggregate results for the four configurations are show in Table 4.

We see that word embedding debiasing in this model has very detrimental effect on performance. The post-embedding layers here are too well-fitted to the final configuration of the embedding layer. We also see that the naive augmentation almost completely eliminates bias and surprisingly happened to incur a lower perplexity hit. We speculate that this is a small random effect due to the relatively small dataset (36,718 sentences of which about 7579 have at least one gendered word) used for this task.

3 Learning Bias

The results presented so far only report on the post-training outcomes. Figure 3, on the other hand, demonstrates the evolving performance and bias during training under various configurations. In general we see that for both neural coreference resolution and language model, bias (thick lines) increases as loss (thin lines) decreases. Incorporating counterfactual data augmentation greatly bounds the growth of bias (gray lines). In the case of naive augmentation, the bias is limited to almost 0 after an initial growth stage (lightest thick line, right).

4 Overall Results

The original model results in the tables demonstrate that bias exhibits itself in the downstream NLP tasks. This bias mirrors stereotypical gender/occupation associations as seen in Figure 2 (black bars). Further, word debiasing alone is not sufficient for downstream tasks without undermining the predictive performance, no matter which stage of training process it is applied (\overleftarrow{\text{WED}} of 2.2 preserves accuracy but does little to reduce bias while \overrightarrow{\text{WED}} of 2.3 does the opposite). Comparing 2.2 (\overleftarrow{\text{WED}})and 2.4 (\overleftarrow{\text{WED}} and \overrightarrow{\text{WED}}) we can conclude that bias in word embedding removed by debiasing performed prior to training is relearned by its conclusion as otherwise the post-training debias step of 2.4 would have no effect. The debiased result of configurations 1.2, 2.5 and 3.3 show that counterfactual data augmentation alone is effective in reducing bias across all tasks while preserving the predictive power.

Results combining the two methods show that CDA and pre-training word embedding debiasing provide some independent debiasing power as in 1.4 and 2.6. However, the combination of CDA and post-training debiasing has an overcorrection effect in addition to the compromise of the predictive performance as in configurations 2.7 and 2.8.

We will continue exploring bias in neural natural language processing. Neural machine translation provides a concrete challenging next step. We are also interested in explaining why these neural network models exhibit bias by studying the inner workings of the model itself. Such explanations could help us encode bias constraints in the model or training data to prevent bias from being introduced in the first place.

This work was developed with the support of NSF grants CNS-1704845 as well as by the Air Force Research Laboratory under agreement number FA9550-17-1-0600. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes not withstanding any copyright notation thereon. The views, opinions, and/or findings expressed are those of the author(s) and should not be interpreted as representing the official views or policies of the Air Force Research Laboratory, the National Science Foundation, or the U.S. Government. We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research.

References

Supplemental Material

Below is the list of the context template sentences used in our coreference resolution experiments OCCUPATION indicates the placement of one of occupation words listed below.

“The [OCCUPATION] ate because he was hungry.”

“The [OCCUPATION] ran because he was late.”

“The [OCCUPATION] drove because he was late.”

“The [OCCUPATION] drunk water because he was thirsty.”

“The [OCCUPATION] slept because he was tired.”

“The [OCCUPATION] took a nap because he was tired.”

“The [OCCUPATION] cried because he was sad.”

“The [OCCUPATION] cried because he was depressed.”

“The [OCCUPATION] laughed because he was happy.”

“The [OCCUPATION] smiled because he was happy.”

“The [OCCUPATION] went home because he was tired.”

“The [OCCUPATION] stayed up because he was busy.”

“The [OCCUPATION] was absent because he was sick.”

“The [OCCUPATION] was fired because he was lazy.”

“The [OCCUPATION] was fired because he was unprofessional.”

“The [OCCUPATION] was promoted because he was hardworking.”

“The [OCCUPATION] died because he was old.”

“The [OCCUPATION] slept in because he was fired.”

“The [OCCUPATION] quitted because he was unhappy.”

“The [OCCUPATION] yelled because he was angry.”

Similarly the context templates for language modeling are as below.

Occupations

The list of hand-picked occupation words making up the occupation category in our experiments is as follows. For language modeling, we did not include multi-word occupations.

Gender Pairs

The hand-picked gender pairs swapped by the gender intervention functions are listed below.

Supplementary Figures