Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias

Introduction

The success of neural network models in various natural language processing tasks, coupled with their opaque nature, has led to much interest in interpreting and analyzing such models. Analysis methods may be categorized into structural and behavioral analyses (?). Structural analyses aim to shed light on the internal structure of a neural model, for example through probing classifiers (?, ?, ?) that predict linguistic properties using representations from trained models. This methodology has been used for analyzing sentence embeddings, machine translation models, and contextual word representation models, among other models (?). Behavioral analyses, on the other hand, aim to assess a model’s behavior by its performance on constructed examples (e.g., ?, ?, ), or by visualizing important input features via saliency methods (e.g., ?, ?, ).

Despite yielding interesting and useful insights, both types of analyses suffer from significant limitations. As pointed out by ? (?), probing classifiers only yield a correlational measure between a model’s representations and an external linguistic property, and are thus not causally connected to the model’s predictions. ? (?) further demonstrate that classifiers that aim to detect biases in learned representations focus on spurious correlations in their training data and fail to generalize to unseen data. Probing classifiers may thus fail to provide faithful interpretations. On the other hand, while behavioral analyses directly evaluate model predictions, they do not typically link them to the model’s internal structure.

To address these limitations, we introduce a methodology for interpreting neural NLP models based on causal mediation analysis (?). Causal mediation analysis is a method from causal inference, which studies the change in a response variable following an intervention, or treatment, e.g., the health outcome of a drug treatment in a clinical trial. Causal mediation analysis (Figure 1) extends this approach by considering the indirect effect of intermediaries, or mediators, on the final outcome—e.g., a drug treatment causes headaches, which cause subjects to take aspirin (mediator), which in turn impacts the health outcome. We use mediation analysis to interpret neural networks by treating internal model components, e.g., specific neurons or attention heads, as mediators between model inputs and model outputs. We propose multiple controlled interventions on the model inputs and mediators, which reveal the causal role of specific components in a model’s behavior.

We apply this framework to the analysis of gender bias in large pre-trained language models. Gender bias has surfaced as a major concern in word representations, both static word embeddings (?, ?) and contextualized word representations (?, ?, ?). We study how grammatical gender bias effects are mediated via different model components in Transformer-based language models, primarily several versions of GPT2 (?), focusing on the role of individual neurons or attention heads in mediating these effects.

Our approach is a structural-behavioral analysis. It is structural in that our results highlight internal model components that are responsible for gender bias. It is behavioral in that said components are causally implicated in how gender bias manifests in the model outputs. In an experimental evaluation using several datasets designed to gauge a model’s gender bias, we find that larger models show larger gender bias effects, potentially absorbing more bias from the underlying training data. The causal mediation analysis further yields several insights regarding the role of different model components in mediating gender bias:

Gender bias is sparse: Much of the effect is concentrated in relatively few model components.

Gender bias is synergistic: Some model components interact to produce mutual effects that amplify their individual effects. Other components operate relatively independently, capturing complementary aspects of gender bias.

Gender bias is decomposable: The total gender bias effect approximates the sum of the direct and indirect effect, a surprising result given the non-linear nature of the model.

We use GPT2 as a primary model for testing our framework, but also perform select analyses with two additional autoregressive models and three masked language models. Our experiments confirm that the insights outlined above apply to all three of the autoregressive models and, albeit to a lesser extent, to masked language models as well. This finding indicates that the causal mediation analysis framework is capable of capturing general patterns of causal structure in Transformer architectures, as opposed to only revealing model-specific characteristics.

In summary, this article makes two broad contributions. First, we cast causal mediation analysis as an approach for analyzing neural NLP models, which may be applied to a variety of models and phenomena. Second, we demonstrate this methodology in the case of analyzing gender bias in pre-trained language models, revealing the internal mechanisms by which bias effects flow from input to output through various model components.

The code for reproducing our results is available at https://github.com/sebastianGehrmann/CausalMediationAnalysis.This article expands upon our conference paper (?) in the following ways: (a) the conference paper only studied sparsity, while here we also study synergism and decomposition; (b) we extend the analysis to other models besides GPT2; (c) we consider various bias metrics; and (d) we draw broader connections to the causality literature.

Related Work

Methods for interpreting neural network models in NLP can be broadly divided into two kinds. Structural methods focus on identifying what information is contained in different model components. Probing classifiers aim to answer such questions by using models’ representations as input to classifiers that predict various properties (?, ?, ?). However, this approach is not connected to the model’s behavior (i.e., its predictions) on the task it was trained on (?, ?). The representation may thus have some information by coincidence, without it being used by the original model. In addition, it is challenging to differentiate the information learned by the probing classifier from that learned by the underlying model (?). Similarly, interactive methods can be useful to identify network components that capture specific properties by relating them to similar training examples (?).

An alternative approach is to assess how well a model captures different linguistic phenomena by evaluating its performance on curated examples (e.g., ?, ?, ?, ). This approach directly evaluates a model’s predictions but does not provide insight into the roles that the internal structure of the network played in arriving at the prediction. Another approach identifies important input features that contribute to a model’s prediction via saliency methods (?, ?, ?), which typically ignore the model’s internal structure, although they may in principle be computed with respect to internal representations (?, ?).

Our causal mediation analysis approach bridges the gap between these two lines of work, providing an analysis that is both structural and behavioral. Mediation analysis is an unexplored formulation in the context of interpreting deep NLP models. In recent work, ? (?) used mediation analysis for interpreting black-box models. However, their analysis was limited to simple datasets and models, while we focus on deep language models. Furthermore, they only considered total effects and (controlled) direct effects, while we measure (natural) direct and indirect effects, which is crucial for studying the role of internal model components.

Causal approaches for interpreting models have very recently begun to be explored in NLP. ? (?) use gradients from probing classifiers to update recurrent language model hidden states, and study the effect of such an intervention on a subject-verb agreement task. ? (?) remove linguistic information from Transformer hidden states and evaluate the effect of such removal on language modeling, effectively performing a sort of intervention at the layer level. ? (?) add auxiliary adversarial tasks to language models in order to learn counterfactual representations with respect to a given concept. Our work is distinguished from these lines of research by our focus on mediation analysis and measurement of direct and indirect effect. Our approach is complementary to the counterfactual representations of ? (?), which may be integrated in our causal mediation analysis.

This proposed strategy is rooted in theories of directed acyclic graphs (DAGs) from the causality literature, which consider DAGs as either the output of causal discovery (?, ?) or as a means to encode prior knowledge to inform the design of analysis methods, e.g., variable selection methods (?). Our approach views the neural network itself as a DAG, with a common ancestor to all nodes, the input, and a common descendent to all nodes, the output. Thus, for each input unit (e.g., a sentence), we can estimate the causal effect of an intervention (e.g., a text edit) by comparing the model output under the intervention to the output given the original input. Repeating this modification for a number of units and averaging over the obtained effects resembles the process of studying average causal effects in the population, say of a drug to treat a disease, with one major advantage. The so-called fundamental problem of causal inference (?) says that we cannot observe the counterfactual of two different interventions on the same unit, e.g., one cannot know what would have happened to a person had they been treated with drug B when in reality they were treated with drug A. Our adaption of the causal language and framework does not suffer from this problem, as we can manufacture outputs from the model under any conceivable interventions on the units.

2 Gender Bias and Other Biases

Neural networks learn to replicate historical, societal biases from training data in various tasks such as natural language inference (?), coreference resolution (?), and sentiment analysis (?). This conflicts with the principle of counterfactual fairness, which states that the model predictions should not be influenced by changes to a sensitive attribute such as gender (?); for instance, a fair and unbiased model should equally associate gendered pronouns with professions. However, biased models make this association proportionally to the distribution of gender in the training data (?). While efforts have been made to reduce bias, this remains a significant ethical challenge.

A common strategy to mitigate biases is to change the training data (e.g., ?, ?, ?, ?, ), the training process (e.g., ?, ?, ), or the model itself (e.g., ?, ?, ?, ) to ensure counterfactual fairness. The resulting biases are often measured similarly to this work by testing that mentions of occupations lead to equal probabilities across grammatical genders in referential expressions.

Others have focused on de-biasing word embeddings and contextual word representations (?, ?, ?), though recent work has questioned the efficacy of these debiasing techniques in removing both grammatical and societal biases (?, ?). Biases may also be introduced in downstream tasks and representations in models where representations depend on additional context (?, ?).

Methodology

2 Causal Mediation Analysis

Causal mediation analysis aims to measure how a treatment effect is mediated by intermediate variables (?, ?, ?). ? (?) described an example where a side effect of a drug may cause patients to take aspirin, and the latter has a separate effect on the disease the drug was originally prescribed for. Thus, the drug has a direct effect through its standard mechanism and an indirect effect operating via aspirin taking (the mediator) as illustrated in Figure 2.

We similarly frame internal model components, e.g., specific neurons, as mediators along the causal path between model inputs and outputs. We thus may consider a neuron to be analogous to aspirin in the example above: the neuron is influenced by the input and, in turn, affects the model output. By measuring the direct and indirect effects of targeted interventions on the model inputs, we can pinpoint the role of specific model components on model predictions. In this work, we focus on the use case of gender bias in language models, as past work suggests that gender is captured in specific model components, e.g., subspaces of contextual word representations (?). While we use gender bias as a case study, the approach can be applied to controlled effect or bias.

The following example illustrates the problem:

Given a prompt $u$ such as The nurse said that, a language model is asked to generate a continuation. A biased model may assign a higher likelihood to she than to he, such that $p_{\theta}(\textit{she}\mid u)>p_{\theta}(\textit{he}\mid u)$. We say that she is the stereotypical candidate, while he is the anti-stereotypical candidate, reflecting a societal bias associating nurses with women more than men.

The relative probabilities assigned to the two candidates can be thought of as a measure of grammatical gender bias in the model:

In our example, we have: ${\bm{y}}(u)=p_{\theta}(\textit{he}\mid\textit{The nurse said that})/p_{\theta}(\textit{she}\mid\textit{The nurse said that})$. If ${\bm{y}}(u)<1$, the prediction is stereotypical; if ${\bm{y}}(u)>1$, it is anti-stereotypical. A perfectly unbiased model would achieve ${\bm{y}}(u)=1$ and thus exhibit bias toward neither the stereotypical nor the anti-stereotypical case. This experimental setup is based on a binary notion of a stereotypical and an anti-stereotypical candidate. Unfortunately, the datasets investigated in this work are designed for experiments with a binary grammatical gender instead of a gender-inclusive spectrum. While they should always be used until we know an individuals preferred pronouns, it is challenging to translate this requirement into our experimental setup which aims to measure the extent to which a model is biased toward a societal stereotype. As an attempt to remedy these shortcomings, we will report experiments that treat the singular they as gender-neutral reference and person as the associated stereotypically neutral entity. These experiments measure the degree to which a model is biased against the more inclusive referential statement. For further discussion of this topic, we refer to ? (?) and ? (?).

In order to understand the role of individual model components on these biased predictions, we apply causal mediation analysis. We first perform targeted interventions on the input text and measure their effect on the gender bias measure defined above (Eq. 1), which serves as the response variable ${\bm{y}}$. Specifically, we perform the following $do$-operations: (a) set-gender: replace the ambiguous profession with an anti-stereotypical gender-specific word (that is, replace nurse with man, doctor with woman, etc.); (b) null: leave the sentence as is. The population of units for this analysis is a set of example sentences such as the above prompt. We define ${\bm{y}}_{x}(u)$ as the value that ${\bm{y}}$ attains in unit ${\bm{u}}=u$ under the intervention $do({\bm{x}}=x$).

Next, we define different kinds of effects of the intervention ${\bm{x}}$ on the response variable ${\bm{y}}$.

The unit-level total effect (TE) of ${\bm{x}}=x$ on ${\bm{y}}$ in unit ${\bm{u}}=u$ is the proportional differenceWe make the difference proportional to control for the high variance of ${\bm{y}}$ across examples. See Appendix A.1 for further evidence. While we limit the discussion to this metric for now, Section 3.5 expands to alternative metrics. between the amount of bias under a gendered reading and under an ambiguous reading:

An illustration of the totel effect is provided in Figure 3a and an example computation is given in Figure 4.

The average total effect of ${\bm{x}}=x$ on ${\bm{y}}$ is calculated by taking the expectation over the population $u$:

2.2 Direct and Indirect Effects

We now analyze the causal role of specific mediators which lie between ${\bm{x}}$ and ${\bm{y}}$. The mediator, denoted as ${\bm{z}}$, might be a particular neuron, a full layer, an attention head, or a certain attention weight. Following Pearl’s definitions, we measure the direct and indirect effects of intervening in the model relative to ${\bm{z}}$ (?).

The natural direct effect (NDE) measures how much an intervention ${\bm{x}}$ changes an outcome variable ${\bm{y}}$ directly, without passing through a hypothesized mediator ${\bm{z}}$. It its computed by applying the intervention ${\bm{x}}$ but holding ${\bm{z}}$ fixed to its original value. For the present use case, we define the NDE of ${\bm{x}}=x$ on ${\bm{y}}$ given mediator ${\bm{z}}=z$ to be the change in the amount of bias when genderizing all units $u$, e.g., changing nurse to man, while holding ${\bm{z}}$ for each unit to its original value. This measures the direct effect on gender bias that does not pass through the mediator ${\bm{z}}$ (illustrated in Figure 3b):

The natural indirect effect (NIE) measures how much the intervention ${\bm{x}}$ changes ${\bm{y}}$ indirectly, through ${\bm{z}}$. It is computed by setting ${\bm{z}}$ to its value under the intervention ${\bm{x}}$, while keeping everything else to its original value. Thus the indirect effect captures the influence of a mediator on the outcome variable. For the present use case, we define the NIE as the change in amount of bias when keeping unit $u$ as is, but setting ${\bm{z}}$ to the value it would attain under a genderized reading. This measures the indirect effect flowing from ${\bm{x}}$ to ${\bm{y}}$ through ${\bm{z}}$ (Figure 3c):

This framework allows evaluating the causal contribution of different mediators ${\bm{z}}$ to gender bias. Through the distinction between direct and indirect effect, we can measure how much of the total effect of gender edits on gender bias flows through a specific component (indirect effect) or elsewhere in the model (direct effect). We experiment with mediators at the neuron level and the attention level, which are defined next.

3 Neuron Interventions

To study the role of individual neurons in mediating gender bias, we assign ${\bm{z}}$ to each neuron ${\bm{h}}_{l,\cdot,k}$ in the LM. The dataset we use consists of a list of templates that are instantiated by profession terms, resulting in examples such as The nurse said that. For each example, we define the set-gender operation to move in the anti-stereotypical direction, changing female-stereotypical professions like nurse to man and male-stereotypical professions like doctor to woman. Section 4 provides more information on the dataset. As mentioned above, we pick person as target of the set-gender change for the gender-neutral reference and we measure the probability of the continuation they. All examples can be seen as biased against gender-neutrality since the models have had limited exposure to the singular they. Moreover, this case suffers from the additional confounder that the model could assign probability to the plural they which we are not able to disambiguate from the singular case.

Throughout the experiments, we investigate the effect of intervening on each neuron independently, as well as on multiple neurons concurrently. That is, the mediator ${\bm{z}}$ may be a set of neurons. In all cases, the mediator is in the representation corresponding to the profession word, such as nurse in the example.

4 Attention Interventions

For studying attention behavior, we focus on the attention weights, which define relationships between words. The mediators ${\bm{z}}$, in this case, are the attention heads $\alpha_{l,h}$, each of which defines a distinct attention mechanism.

We align our intervention approach with two resources for assessing gender bias in pronoun resolution: Winobias (?) and Winogender (?). Both datasets consist of Winograd-schema-style examples that aim to assess gender bias in coreference resolution systems. We reformulate the examples to study bias in LMs, as the following example from Winobias shows:

The nurse examined the farmer for injuries because she

According to the stereotypical reading, the pronoun she refers to the nurse, implying the continuation was caring. The anti-stereotypical reading links she to the farmer, this time implying the continuation was screaming. The bias measure is ${\bm{y}}(u)=p_{\theta}(\textit{was screaming}\mid u)/p_{\theta}(\textit{was caring}\mid u)$.To compute probabilities of multi-word continuations, we use the geometric mean of the token-level probabilities. In this case, we define the swap-gender operation, which changes she to he. The total effect is then

In the experiments, we study the effect of the attention from the last word (she or he) to the rest of the sentence.One may also study individual attention arcs. However, attention does not always focus on a specific word, often falling on adjacent words. See Appendix C.2 for this phenomenon. Intuitively, in the above example, if the word she attends more to nurse than to farmer, then the more likely continuation might be was caring. We compute the NDE and NIE for each head individually by intervening on the attention weights $\alpha_{l,h,\cdot,\cdot}$. We also evaluate the joint effects when intervening on multiple attention heads concurrently. The population-level TE and the NDE and NIE are defined analogously as above.

5 Alternate Metrics and Algorithmic Fairness

In this section, we consider alternate metrics to the ones discussed above and their connections to algorithmic fairness. A natural starting point would be to redefine bias as a simple difference between probabilities for the grammatical genders.

To address the issue of high variance in the raw predicted probabilities for the stereotypical and anti-stereotypical candidates across different input sentences (Appendix A.1), they can be normalized to form a probability distribution such that $p_{\theta}(\text{anti-stereotypical}\mid\textit{u})+p_{\theta}(\text{stereotypical}\mid\textit{u})=1$. Similarly, effects can then be computed as the difference between bias before and after intervention. This intuitive approach closely mirrors the original approach in Pearl’s work (?).

where Effect $\in\{\text{TE},\text{NDE},\text{NIE}\}$ and ${\bm{y}}_{\texttt{intervention}}$ would correspond to the intervention for computing the respective effect.

Since the causal effect is the primary result of interest, we can alternatively directly construct metrics that quantify the difference between prediction outcomes before and after the intervention. The disparate impact affected by gender bias can be considered a representational harm rather than an allocative harm (?). Most existing work on fair classification focuses on allocative harms (?, ?). On the other hand, bias in NLP is centered more on representational harms, particularly in word representations (?, ?). We are nonetheless considering gender bias in a supervised prediction task, so the alternate measures of causal effect will be based on fairness in classifiers rather than fairness in word representations.

A recent line of work in algorithmic fairness is known as individual fairness, and is motivated by the concept that similar individuals should be treated similarly (?). Unlike group fairness, which provides broad aggregate statements about fairness, individual fairness operates at the level of each individual input. In this case, each input sentence can be thought of as an individual, and the unit $u$ before and after an intervention or gendered reading can be treated as similar individuals. In the neuron interventions, the examples The nurse said that and The man said that can be considered “similar” individuals, with the latter being the outcome of applying the set-gender operation on the former. The same concept applies in the attention intervention case, only with the swap-gender operation instead of the set-gender operation. The outcomes generated for each “individual” will be the predicted next word, and the difference between the outcome distributions for two similar input sentences can will be the unfairness. Statistical distance measures are a suggested choice for comparing these outcomes (?). Two such examples are:

Statistical distance, or total variation norm, between two probability measures $P$ and $Q$ on a domain $A$ is defined as:

The relative $l_{\infty}$ metric is similarly defined between two probability measures $P$ and $Q$ on a domain $A$ by the following expression:

Here, we choose the domain $A$ to be the set of predicted outcomes of interest, specifically the stereotypical and anti-stereotypical candidates. When working with neuron interventions, the outcomes of interest in response to a prompt like The nurse said that will be the set of possible genders predicted by the model, specifically $A=\{\text{he},\text{she}\}$ in our case. Since these metrics quantify distributional differences, $P$ and $Q$ will be derived by normalizing the raw probabilities predicted by the neural network for each gender in $A$ such that they sum to one and form a probability distribution. $P$ and $Q$ would respectively refer to the outcome probability distribution under the original reading and under the alternate reading with different interventions depending on the causal effect (TE, NDE, or NIE) being computed.

The probability distributions used in the metrics discussed above can reasonably be extended to the full range of possible output words by the neural network. The inclusion of only pronouns in our construction, though, is due to the focus on the specific case of gender bias. Considering the larger subset, or even the full set, of possible output words and how their predicted probabilities change with various interventions would make it more difficult to concretely identify the effect on gender bias. However, it may also yield interesting insights on more subtle consequences of the interventions, such as how other non-pronoun words with associated stereotypes may be affected by these interventions.

We find that our results are robust to the metric used to quantify the effects and the work in this article focuses on measures of bias and effects originally described in the previous subsections. For reference, Appendix F contains the results of the same analyses but using the alternate metrics described here.

Experimental Details

As an example large pre-trained LM, we use GPT2 (?), a Transformer-based (English) LM trained on massive amounts of data. We use several model sizes: small, medium, large, extra-large (xl), and a very small distilled model (?). To test the universality of our findings across Transformer-based architectures, we perform a subset of experiments with two additional autoregressive models – Tranformer-XL (?) and XLNet (?) – as well as three masked LMs – BERT (?), DistilBERT (?), and RoBERTa (?). We use the Transformers library (?) for access to all of the listed models.

2 Data

For neuron intervention experiments, we augment the list of templates from ? (?) with several other templates, instantiated with professions from ? (?). The professions are accompanied by crowdsourced ratings between $-1$ and $1$ for definitionality and stereotypicality. Actress is definitionally female, while nurse is stereotypically female. None of the professions are stereotypically or definitionally gender-neutral in the sense that people working in the profession are referred to in singular they. To simplify processing by GPT2 and focus on common professions, we only take examples that are not split into sub-word units, resulting in 17 templates and 169 professions, or 2,873 examples in total. The full lists of templates and professions are given in Appendix A.1. We refer to these examples as the Professions dataset.

For attention intervention experiments, we use examples from Winobias Dev/Test (?) and Winogender (?), totaling 160/130 and 44 examples that fit our formulation, respectively. We experiment with the full datasets and with filtering by total effect. Both datasets include statistics from the U.S. Bureau of Labor Statistics to assess the gender stereotypicality of the referenced occupations. Appendix A.2 provides additional details about the datasets and preprocessing methods.

Results

Before describing the results from the mediation analysis, we summarize some insights from measurements of the total effect. Table 1 shows the total effects of gender bias in the different GPT2 models, on three datasets, as well as the effects with a randomly initialized GPT2-small model. Random model effects are much smaller, indicating that it is the training that causes gender bias.

In the Winograd-style datasets, the total effect mostly increases with model size, saturating at the large and xl models. In the professions dataset, model size is not well correlated with total effect, but GPT2-xl has a much larger effect. Since larger models can more accurately emulate the training corpus, it makes sense that they would more strongly integrate its biases.

It is difficult to compare effect magnitudes in the three datasets because of their different nature. The professions dataset yields much stronger effects than the Winograd-style datasets. This may be attributed to the more explicit source of bias, the word representations, as compared to intricate coreference relations in the Winograd-style datasets.

In the professions dataset, we found moderate positive correlations between the external gender biasFor this analysis, we add the stereotypicality and definitionality of each profession to capture the overall bias value. and the log-total effect, ranging from $0.35$ to $0.45$ over the different models, indicating that the model captures the expected biases. It further shows that the effect is amplified by the model for words that are perceived as more biased. In the Winograd-style datasets, we found relatively low correlations between the log-total effect and the log-ratio of the two occupations’ stereotypicality, ranging from $0.17$ to $0.26$. This low correlation may be due to either a smaller size compared to the professions dataset or the more complex relations in these datasets.

Throughout the templates in the professions experiments, the baseline probability $p(they|u)$ is much more consistent, but low, across all professions. Consider the template “The X said that” — in this case, under GPT2-distil “they” varies in probability from 0.2% to 4.2% while “he” has a much wider range from 1.1% to 31.8%. Consequently, the total effect for neutral interventions is also much more consistent across models and templates. For the professions dataset, the GPT2 variants from distil to large have respective total effects of $8.3$, $7.5$, $9.6$, and $12.5$, all with standard deviations $<10$. We hypothesize that this can mostly be attributed to very low probability for the singular “they” and a consistent baseline probability where “they” is part of a referential statement toward a group of individuals, for example in “The accountant said that they [the people] need to pay taxes”.

2 Sparsity

Where in the model are gender bias effects captured? Are the effects mediated by only a few model components or distributed across the model? Here we answer these questions by measuring the indirect effect flowing through different mediators.

Figure 5(a) shows the indirect effects for each head in GPT2-small on Winobias. The heatmap shows interventions on each head individually. A small number of heads, concentrated in the middle layers of the model, have much higher indirect effects than others. The bar chart shows indirect effects when intervening on all heads in a single layer concurrently. Consistent with the head-level heatmap, the effects are concentrated in the middle layers. We did not find similar behavior in a randomly initialized model, indicating that these patterns do not occur by chance. We found this sparsity consistent in all model variants and datasets we examined. See Appendix C.1 for additional visualizations of indirect effects as well as direct effects.

To determine how many heads are required to achieve the full effect of intervening on all heads, we also intervene on groups of heads. We do so by selecting a subset of heads, using either a Greedy approach, which iteratively selects the head with the maximal marginal contribution to the indirect effect, or a Top-k approach, which selects the $k$ elements with the strongest individual effects. Appendix D provides more information on these algorithms. Only 10 heads are required to match the effect of intervening on all 144 heads at the same time (Figure 5(b)). The first 6 selected ones are from layers 4 and 5, further demonstrating the concentration of the effect in the middle layers.

Figure 6(a) shows the indirect effects from the top 5% of neurons from each layer in different models. The word embeddings (layer 0) and the first hidden layer have the strongest effects. This stands in contrast to the attention intervention results, where middle layers had much larger effects. However, we still observe a small increase in effect within the intermediate layers across all models except for the randomized one. Figure 6(b) shows the effect from the top5% neurons for the neutral intervention with GPT2-medium. We can observe that, while the variance of the embedding importance is much higher, the effect size is in line with the rest of the model. Additionally, the effect is much more evenly distributed across all layers. These two observations are further indications that the model has not learned a distinct and sparse representation of gender-neutral references.

Figure 7 shows the indirect effects when selecting neurons by the Top-k algorithm.For computational reasons, we select sets of 96 neurons. Similar to the attention result, a tiny fraction of neurons is sufficient for obtaining an effect equal to that of intervening on all neurons concurrently. Most of the top selected neurons are concentrated in the embedding layer and first hidden layer.

3 Synergism

How do different model components interact in capturing gender bias? Do different components work independently or jointly? Are gender bias effects amplified by different components or constrained?

Recent work found that attention heads in GPT2 and other Transformers play highly differentiated roles. For instance, some heads focus on adjacent tokens while others align with syntactic properties (?, ?, ?, ?). We use mediation analysis to study the interdependence of attention heads.

Figure 8 compares indirect effects of concurrent intervention on all heads (NIE-all) to summing the effects of independent interventions (NIE-sum). The differences are fairly small (maximum relative distance from NIE-all between 0.7% and 11.3%), indicating that heads operate primarily in an independent and complementary manner, capturing different aspects of gender bias. As Figure 5(b) shows, most heads do not contribute much to the indirect effect, and many reduce it. This trend is consistent across models and datasets (Appendix D).

Figure 9 shows the attention of the three heads with the highest indirect effects on Winobias. The figure demonstrates that they capture different coreference aspects: one head aligns with the stereotypical coreference candidate, another head attends to the tokens following that candidate, while a third attends to the anti-stereotypical candidate. ? (?) previously identified the same head (layer 5, head 10; noted as 5-10) as relating to coreference resolution based on visual inspection. ? (?) found an attention head in BERT (?) that was highly predictive of coreference, also in layer 5 out of 12.

Similar to the case of attention, Figure 7 shows that after a few neurons (4%) match the model-wise concurrent effect, most neurons do not contribute much, and many even diminish the effect. This result suggests that neurons may be as specialized as the individual attention heads. However, an analogous qualitative analysis is challenging due to the large number of neurons.

By definition, concurrent intervention on all neurons entails TE = NIE-all, since then ${\bm{y}}_{\texttt{set-gender}}(u)$ $=$ ${\bm{y}}_{\texttt{null},{\bm{z}}_{\texttt{set-gender}}(u)}(u)$. Notably, the sum of independent indirect effects (NIE-sum) is much smaller than the concurrent intervention (NIE-all), as shown in the following table. Thus, neurons combine synergistically to compound independent effects.

Visualizing the indirect effect of each neuron in a heatmap (Figure 10 top) reveals vertical stripes when a neuron at the same index, but different layers, has a similar effect. While sparse, this effect sometimes continues over multiple layers. Two possible explanations for this are random alignments of two effective cells or the residual connections between the layers. To analyze this, we computed the number of stripes between layer pairs across the professions dataset, with and without randomizing neuron indices. As Figure 10 (bottom) shows, the stripes are less random in higher layers. This implies that, as the information gets transformed, the model converges on a representation. This is akin to gated recurrent networks, except that those transform across time steps instead of layers. This result may partially explain the higher neuron importance in earlier layers since those neurons have not yet converged to a representation and thus have a higher variance and contribution to the representation in other neurons.

4 Decomposition of the Total Effect

Figure 8 also shows the concurrent direct and indirect effects, when intervening on all heads. In all but the smallest model (distil), the concurrent indirect effect is larger than the direct effect, indicating that most of the effect is mediated through the attention heads. Other model components (e.g., word representations) are nonetheless responsible for a portion of the total effect. This might be due to biased word embeddings predisposing the model towards certain continuations. For instance, the representation of he might lead the model to predict a lower probability for was caring compared to she, irrespective of any previous occupation mention.

In linear models, it is known that the linear total effect decomposes to direct and indirect effects (?). Intuitively, intervention effects either flow through a mediator or directly. In our case, we have a highly non-linear model and this decomposition is not guaranteed.? (?) discuss which decompositions can be guaranteed. While causal effect definitions are model-free, some decompositions are possible even for non-linear models and effects, in the presence of, for example, interaction between the intervention and the mediator. However, the TE = NDE + NIE decomposition is not guaranteed without further assumptions (e.g., under linear models). In our case, an additional no-interaction condition was needed for the decomposition to hold, as shown and discussed in Appendix E. Nevertheless, Figure 11 shows such approximate decomposition for the top heads in GPT2-small.In the neuron intervention case, by definition TE $=$ NIE-all and NDE-all $=$ , so the decomposition trivially holds. The same holds for concurrent interventions (Figure 8), where TE $\approx$ NDE-all + NIE-all. To understand this phenomenon, observe that under our formulation of the effects using a proportional difference, a decomposition of the form TE = NDE + NIE is expected if the following equality holds for all $u$:

See Appendix E for intuition, a proof, and evidence that Eq. 11 approximately holds in our results.

5 Experiments with other models

How specific are the findings outlined above to GPT2? In this section, we examine whether causal mediation analysis has yielded any general insights about gender bias effects in Transformer-based LMs.

Figure 12 shows the indirect effects for each head and layer in Transformer-XL and XLNet-base on Winobias (top) along with bar charts of top heads by indirect effect and their respective direct effects for both models (bottom). These results support our main findings with GPT2: the heat maps show the sparsity of indirect effects in Transformer-XL and XLNet while the bar charts demonstrate decomposability. The agreement between all autoregressive models we have tested indicates that our framework has revealed general patterns in gender bias effects manifesting across different architectures rather than features of a particular model.

The results of attention intervention experiments with masked language models – BERT, DistilBERT, and RoBERTa – show mixed levels of agreement with GPT2 results. For example, Figure 13 illustrates the lack (left) and apparent presence (right) of sparsity of indirect effects in BERT-large-uncased when taking two different approaches for scoring candidates. In contrast to the autoregressive case, there does not seem to be a single obvious way to score multi-token continuations with masked LMs since they do not directly output probability distributions for the next word given a prefix. This led us to try out several different scoring schemes in our experiments (see Appendix G for details). We suspect that this difference in how candidates are scored in autoregressive models on the one hand and masked LMs on the other may be the underlying cause for the discrepancy in results for the two kinds of models.

One general trend that seems to manifest across all of the different models we have tested is that of larger variants of the same models having larger total effects. Appendix G contains a table illustrating this, as well a more detailed exposition of the results with masked LMs.

Figure 14 shows the indirect effects from the top 5% of neurons from each layer in all of our five additional models along with the same plot for GPT2-small. None of the non-GPT2 models seem to share the exact same pattern that we have observed with GPT2 variants, where the strongest effects in the word embedding layer used to be followed by a rapid drop in the successive layers. Here, a general trend seems to be one of roughly gradual decrease to 0 in the strength of effects, albeit with more and less significant fluctuations, the only exception being Transformer-XL whose effects are steadily increasing in the first third of the layers. As can be seen from Figure 14, we also observe considerably higher variances with some of the new models than in the case of GPT2. We do not have a compelling theory to explain these differences in results.

Discussion and Conclusion

This article introduced a structural-behavioral framework for interpreting neural models based on causal mediation analysis. An application of this framework to gender bias in NLP yielded several insights regarding the mechanisms by which gender bias is mediated in large Transformer LMs, revealing that gender bias effects are sparse, synergistic, and decomposable to direct and indirect effects.

This work can be extended in multiple ways. While the framework may apply to any property expressed as a function of model predictions, our experimental design focuses on gender bias in a binary setup. Preliminary experiments with gender-neutral references showed that models have not learned the concept of these references, at least for our chosen templates. Applying the methodology to more gender-inclusive setups and other kinds of biases is thus especially important since it can uncover these shortcomings in models.

Our results can also guide model selection for limiting the amount of bias. In related work, ? (?) demonstrated that changing hidden representations in an LSTM based on the output of diagnostic classifiers can decrease the error on a subject-verb agreement classification task. Inspired by these results, mediation analysis could be used to determine where and how to intervene in similar ways and thus not only assess the success of debiasing techniques, but also motivate new debiasing methods that establish counterfactual fairness for protected groups. Another extension of this work for training a fair model would be to use the individual fairness framework (?) that inspired some of the alternate metrics discussed in this article. This approach would involve maximizing a traditional objective function through the typical training process subject to an additional Lipschitz constraint.

This work is a first attempt to adopt mediation analysis for interpreting NLP models. The causality literature often focuses on assumptions needed for identification of mediation effects from observed data (?, ?, ?). The challenge with inferring causality from observational data is that for each unit, the outcome is observed under a single intervention. However, in this work we use the language of causal mediation to study the structure of NLP models, utilizing the fact that the outcome of the same unit (e.g., a sentence) can be observed under any intervention given a trained model. As a result, causal effects can be computed in a relatively simple manner. While we observed consistent results under multiple metrics, our definitions of causal effects to quantify bias could be refined, and alternative definitions might be advantageous for NLP research.

The causality literature offers many avenues for continuing this line of work, including mediation analysis with non-linear models, and alternative effect decompositions (?, ?, ?). A promising direction is to focus on path-specific effects (?), to identify the exact mechanisms through which biases arise. Characterizing specific paths from model input to output might also be useful during training by disincentivizing the creation of paths leading to bias. We believe the present work sets the ground for employing this literature in the ongoing effort to analyze neural NLP models.

Acknowledgments

S. G. was supported by a Siebel Fellowship. Y.B. was supported by the Harvard Mind, Brain, and Behavior Initiative. Work conducted while J.V. was at Palo Alto Research Center.

A Data Preparation

Figure 15 shows the 17 base templates used for the neuron interventions. To validate that each template would capture gender bias, we instantiate each with an occupation of woman and man and verify that the conditional probabilities of she and he align with gender. Given woman as the occupation word, the probability ratio $p(\text{she})/p(\text{he})$ ranges from $2.5$ to $45.1$ across templates ($\mu=17.2,\sigma=13.1$). Given man, the value $p(\text{he})/p(\text{she})$ ranges from $3.0$ to $55.4$ ($\mu=21.9,\sigma=16.2$). Thus the relative probabilities align with gender, though they vary greatly in magnitude.

For each of the templates, we used the following professions. Words in italics are definitional and were thus excluded from the total effect calculation:

female: actress, advocate, aide, artist, baker, clerk, counselor, dancer, educator, instructor, maid, nun, nurse, observer, performer, photographer, planner, poet, protester, psychiatrist, secretary, singer, substitute, teacher, teenager, therapist, treasurer, tutor, waitress neutral: acquaintance, character, citizen, correspondent, employee, musician, novelist, psychologist, student, writer male: accountant, actor, administrator, adventurer, ambassador, analyst, architect, assassin, astronaut, astronomer, athlete, attorney, author, banker, bartender, biologist, bishop, boss, boxer, broadcaster, broker, businessman, butcher, campaigner, captain, chancellor, chef, chemist, cleric, coach, collector, colonel, columnist, comedian, comic, commander, commentator, commissioner, composer, conductor, congressman, consultant, cop, critic, curator, dad, dean, dentist, deputy, detective, diplomat, director, doctor, drummer, economist, editor, entrepreneur, envoy, farmer, filmmaker, firefighter, fisherman, footballer, goalkeeper, guitarist, historian, inspector, inventor, investigator, journalist, judge, landlord, lawmaker, lawyer, lecturer, legislator, lieutenant, magician, magistrate, manager, mathematician, mechanic, medic, midfielder, minister, missionary, monk, narrator, negotiator, officer, painter, pastor, philosopher, physician, physicist, policeman, politician, preacher, president, priest, principal, prisoner, professor, programmer, promoter, prosecutor, protagonist, rabbi, ranger, researcher, sailor, saint, salesman, scholar, scientist, senator, sergeant, servant, soldier, solicitor, strategist, superintendent, surgeon, technician, trader, trooper, waiter, warrior, worker, wrestler

A.2 Winobias and Winogender

For both Winobias and Winogender datasets, we exclude templates in which the shared prompt does not end in a pronoun.An example of a removed template is: “The receptionist welcomed the lawyer because this is part of her job.” / “The receptionist welcomed the lawyer because it is his first day to work.” For Winobias, we only consider Type 1 examples, which follow the format of a shared prompt and two alternate continuations. We also experiment with filtering by total effect, removing examples with a negative total effect as well as examples in the bottom quartile of those with a positive total effect. The sizes of all dataset variations may be found in Table 2. Results are reported for filtered versions of both datasets and the Dev set of Winobias unless otherwise noted.

Both datasets include statistics from the U.S. Bureau of Labor Statistics (BLS) to assess the gender stereotypicality of the referenced occupations. Winogender additionally includes gender estimates from text (?), which we also include in our analysis. Whereas each Winobias example includes two occupations of opposite stereotypicality, each Winogender example includes one occupation and a participant, for which no gender statistics are provided. For consistency with the Winobias analysis, we make the simplifying assumption that the gender stereotypicality of the participant is the opposite of that of the occupation.

B Additional Total Effects

Table 3 provides the total effects across all variations of the Winograd-style datasets. The relationship between model and effect size is relatively consistent across dataset variations (Winobias/Winogender, filtered/unfiltered, Dev/Test, BLS/Bergsma gender statistics), though the magnitudes of the effects may vary between dataset variations.

Table 4 provides the total effects on the professions dataset when separated to stereotypically female and male professions, where stereotypicality is defined by the profession statistics provided by ? (?). Notably, the effects are much larger in the female case. This may be explained by stereotypicaly-female professions being of higher stereotypicality than stereotypically-male professions, reflecting a societal bias viewing women’s professions as more narrowed.

C Additional Attention Results

Figure 16 complements Figure 5(a) by visualizing the indirect effects for additional GPT2 models. As with Figure 5(a), the attention heads with the largest indirect effects lie in the middle layers of each model. Figure 17 shows the indirect effects for a model with randomized weights. Figures 18 and 19 visualize the indirect effects for other dataset variations for the GPT2-small model from Figure 5(a). The attention heads with largest indirect effect have significant overlap across the dataset variations.

Figure 20 visualizes direct effects on Winobias for GPT2-small and GPT2-large. As discussed in Section 5.4, the sum of direct and indirect effects approximate the total effect.

C.2 Examples

Figure 21 visualizes attention for the Winobias examples with the greatest total effect in GPT2-small, complementing the example shown in Figure 9. Figure 22 visualizes attention for additional models for the same example shown in Figure 9.

D Additional subset selection results

We wish to select a subset of attention heads or neurons that perform well together to better understand the sparsity of attention heads and neurons and their impact on gender bias in Transformer models.

The problem of subset selection (selecting $k$ elements from $n$) is an NP-hard combinatorial optimization problem. To construct a meaningful solution set, we employ several algorithms for subset selection from submodular maximization. We note that while our objective functions are not strictly submodular as they do not satisfy the diminishing returns property, our objectives exhibit submodular-like properties and numerous algorithms have been proposed to efficiently maximize submodular and variants of submodular functions.

For monotone submodular functions, it is known that a greedy algorithm that iteratively selects the element with the maximal marginal contribution to its current solution obtains a $1-1/e$ approximation for maximization under a cardinality constraint (?) and that this bound is optimal. For non-monotone submodular functions, there is the randomized greedy algorithm which emits a $1/e$ approximation to the optimal solution (?).

To select subsets of attention heads, we compare Top-k (selecting $k$ elements with the largest individual values) and Greedy. Even though randomized greedy has stronger theoretical guarantees because our objective is clearly non-monotonic, we favor the deterministic algorithm for increased interpretability. Figure 23 shows results for head selection across different models on Winogender and Winobias. Sparsity is consistent across all experiments where only a small proportion of heads are sufficient to achieve the full model effect of intervening at all heads. On Winogender, only 4/4/5/4% of heads are needed to saturate, while on Winobias, only 6/7/8/6% of heads are needed in GPT2-distil/small/medium/large.

To select subsets of neurons, we use Top-k to compute NIE of sets of neurons because sequential greedy is too computationally intensive to run. Alternative methods using adaptive sampling techniques have been proposed to speed-up Greedy for submodular functions under cardinality constraints (?, ?, ?, ?). For non-monotone or non-submodular functions, there are parallelized algorithms that use similar techniques to select sets (?, ?, ?). These methods provide an alternative approach to Top-k for selecting subsets of neurons and can be explored in future work.

E Proof that no-interaction in the difference NIE implies decomposition of the TE

Since by definition ${\bm{y}}_{\texttt{set-gender}}(u)={\bm{y}}_{\texttt{set-gender},{\bm{z}}_{\texttt{set-gender}}(u)}(u)$ and ${\bm{y}}_{\texttt{null}}(u)={\bm{y}}_{\texttt{null},{\bm{z}}_{\texttt{null}}(u)}(u)$, it can be seen that both sides of Eq. 11 describe a form of NIE (defined on the difference scale), where each contrasts ${\bm{y}}$ under two different interventions on ${\bm{z}}$ while keeping the sentence $u$ the same (left side, under set-gender; right side, under null). This equation parallels a previously-described assumption in the causal mediation analysis literature that ascertains that the NIE is the same regardless of the fixed value at which the intervention (the analogue of set-gender/null) is held, known as a no-interaction assumption (?). We show that no-interaction in the indirect effect on the difference scale implies that the TE = NDE + NIE under our scale. Eq. 11 can be rewritten as

Now, dividing both sides of the equation by ${\bm{y}}_{\texttt{null}}(u)$ and taking expectations over $u$ yields

It should be noted that even if Eq. 11 does not hold, but the equation approximately holds upon dividing both sides by $y_{\texttt{null}}$, we would expect the decomposition TE $\approx$ NDE + NIE to hold. Indeed, further inspection of the trained model revealed that the left side and right side of Eq. 11 were very close in the case of the attention intervention. Figure 24 shows a plot of the values attained by the two sides of the equation (normalized by $y_{\texttt{null}}$ to make consistent with the earlier analyses) for all attention heads across all examples in the Winobias dataset. Fitting the data to a linear model yields a coefficient of 1.04 and an intercept of 0.00 ($R^{2}=0.78$).

F Alternate Metrics

Here are the results of select analyses from throughout the article, replicated using the different alternate metrics described in Section 3.5. Specifically, we investigate a total of four metrics, consisting of the primary metric and the three alternate metrics, referred to and defined by the following:

Original metric: The primary measurement of bias and effects as described in the setup in Sections 3.2, 3.3, and 3.4 and as used throughout the article.

Normalized difference: Effects based on the distance measure described in Equation 8.

Total Variation distance (or TV-distance): Effects based on the distance measure described in Equation 9.

Figures 25, 26, and 27 evaluate all the metrics on the GPT2-small model using the filtered Winobias Dev, filtered Winogender Bergsma, and Professions data sets respectively.

Tables 5 and 6 reporting total effects are also included. Overall, the main takeaways are that the results of our analyses are quite robust to the metric that is being used to compute the effects. There are minor visible differences, but the relative behavior overall is consistent, which is not necessarily an intuitive result.

For instance, one notable difference is that there appears to be less absolute sparsity, but relative sparsity holds, which suggests that these alternate metrics are more sensitive to the effects of causal mediation analysis while demonstrating that the conclusions drawn by our article continue to hold in all of these situations. Similarly, larger models appear to exhibit a more pronounced effect than smaller models even under alternate metrics, which is another consistent result.

It is worth noting that because of the way that TV-distance is constructed in terms of difference between magnitudes rather than ratios, the effect values for each model and each data set are on different levels. However, when normalizing relative to a constant (i.e., the minimum effect across layers for the neuron line plots or across heads for the attention heatmaps), then the emerging behavior is typically consistent with the broader patterns that have been identified.

G Attention intervention experiments with masked language models

Scoring a multi-token continuation, say $[x_{3},x_{4}]$, given a prefix, say $[x_{1},x_{2}]$, is straightforward with autoregressive models: we simply need to combine the individual token-level probabilities $p_{\theta}(x_{3}\mid x_{1},x_{2})$ and $p_{\theta}(x_{4}\mid x_{1},x_{2},x_{3})$ (which we do by taking a geometric mean). The problem becomes less trivial when we are faced with masked LMs, as there is not a single obvious way to compute token-level probabilities anymore. In this work, we try out three different ways of doing so, which in our running example would correspond to defining token-level probabilities as:

$p_{\theta}(x_{3}\mid x_{1},x_{2},\texttt{mask})$ and $p_{\theta}(x_{4}\mid x_{1},x_{2},x_{3},\texttt{mask})$;

$p_{\theta}(x_{3}\mid x_{1},x_{2},\texttt{mask},x_{4})$ and $p_{\theta}(x_{4}\mid x_{1},x_{2},x_{3},\texttt{mask})$;

$p_{\theta}(x_{3}\mid x_{1},x_{2},\texttt{mask},\texttt{mask})$ and $p_{\theta}(x_{4}\mid x_{1},x_{2},x_{3},\texttt{mask})$.

We propose scheme 1 because it seems to give the closest formulation to the autoregressive setting, while schemes 2 and 3 are derived from existing literature (?, ?, ?). We also have a choice of whether or not to include the special cls and sep tokens (used during the pre-training of the models; e.g., [CLS], and [SEP] in the case of BERT) when feeding examples to the masked LMs. Consequently, we define three additional scoring schemes which are exact copies of the original ones but with the special tokens included:

$p_{\theta}(x_{3}\mid\texttt{cls},x_{1},x_{2},\texttt{mask},\texttt{sep})$ and $p_{\theta}(x_{4}\mid\texttt{cls},x_{1},x_{2},x_{3},\texttt{mask},\texttt{sep})$;

$p_{\theta}(x_{3}\mid\texttt{cls},x_{1},x_{2},\texttt{mask},x_{4},\texttt{sep})$ and $p_{\theta}(x_{4}\mid\texttt{cls},x_{1},x_{2},x_{3},\texttt{mask},\texttt{sep})$;

$p_{\theta}(x_{3}\mid\texttt{cls},x_{1},x_{2},\texttt{mask},\texttt{mask},\texttt{sep})$ and $p_{\theta}(x_{4}\mid\texttt{cls},x_{1},x_{2},x_{3},\texttt{mask},\texttt{sep})$.

As mentioned in Section 5.5, we find considerable variation in the results with different masked LMs and scoring schemes. Figures 31, 32, 33, 34, and 35 show the indirect effects for each head and layer in DistilBERT, BERT-base-uncased, BERT-large-uncased, RoBERTa-base, and RoBERTa-large, respectively, using all six of the scoring schemes, on the filtered Winobias Dev dataset. We do observe the general trend of total effects being larger for larger variants of the same model, however. This is illustrated in tables 7 and 8, which list the total effects for different models and scoring schemes on the filtered Winobias Dev, Winogender Bergsma, and Professions datasets.

References