Causal Abstractions of Neural Networks

Introduction

Explainability and interpretability have long been central issues for neural networks, and they have taken on renewed importance as such models are now ubiquitous in research and technology. Recent structural evaluation methods seek to reveal the internal structure of these “black box” models. Structural methods include probes, attributions (feature importance methods), and interventions (manipulations of model-internal states). These methods can complement standard behavioral techniques (e.g., performance on gold evaluation sets), and they can yield insights into how and why models make the predictions they do. However, these tools have their limitations, and it has often been assumed that more ambitious and systematic causal analysis of such models is beyond reach.

Although there is a sense in which neural networks are “black boxes”, they have the virtue of being completely closed and controlled systems. This means that standard empirical challenges of causal inference due to lack of observability simply do not arise. The challenge is rather to identify high-level causal regularities that abstract away from irrelevant (but arbitrarily observable and manipulable) low-level details. Our contribution in this paper is to show that this challenge can be met. Drawing on recent innovations in the formal theory of causal abstraction [Beckers and Halpern, 2019, Beckers et al., 2020, Chalupka et al., 2016, Rubenstein et al., 2017], we offer a methodology for meaningful causal explanations of neural network behavior.

Our methodology causal abstraction analysisWe provide tools for causal abstraction analysis at http://github.com/hansonhl/antra and the code base for this paper at http://github.com/atticusg/Interchange consists of three stages. (1) Formulate a hypothesis by defining a causal model that might explain network behavior. Candidate causal models can be naturally adapted from theoretical and empirical modeling work in linguistics and cognitive sciences. (2) Search for an alignment between neural representations in the network and variables in the high-level causal model. (3) Verify experimentally that the neural representations have the same causal properties as their aligned high-level variables using the interchange intervention method of Geiger et al. .

As a case study, we apply this methodology to LSTM-based and BERT-based natural language inference (NLI) models trained on the logically complex Multiply Quantified NLI (MQNLI) dataset of Geiger et al. . This challenging dataset was constructed with a tree-structured natural logic causal model [MacCartney and Manning, 2007, van Benthem, 2008, Icard and Moss, 2013]. Our BERT-based model has the structure of a standard NLI classifier, and yet it is able to perform well on MQNLI (88%), a result Geiger et al. achieved only with highly customized task-specific models. By contrast, our LSTM-based model is much less successful (46%).

The obvious scientific question in this case study is what drives the success of the BERT-based model on this challenging task. To answer this we employ our methodology. (1) We formulate hypotheses by defining simplified variants of the natural logic causal model. (2) We search over potential alignments between neural representations in BERT and variables in our high-level causal models. (3) We perform interchange interventions on the BERT model for each alignment. We find that our BERT model partially realizes the causal structure of the natural logic causal model; crucially, the LSTM model does not. High-level causal explanation for system behavior is often considered a gold standard for interpretability, one that may be thought quixotic for complex neural models [Lillicrap and Kording, 2019]. The point of our case study is to show that this high standard can be achieved.

We conclude by comparing our methodology to probing and the attribution method of integrated gradients [Sundararajan et al., 2017]. We argue probing is unable to provide a causal characterization of models. We show formally that attribution methods do measure causal properties, and in that way they are similar to the tool of interchange interventions. However, our methodology of causal abstraction analysis provides a framework for systematically measuring and aggregating such causal properties in order to evaluate a precise hypothesis about abstract causal structure.

Related Work

C_{+} is a causal abstraction of a neural network $N_{+}$ under a particular alignment (top). We can experimentally confirm this hypothesis by conducting an interchange intervention on both the network and the computation with every pair of inputs and evaluating whether the intervened network and intervened computation have the same counterfactual output behavior. We schematically depict an interchange intervention on the network $N_{+}$ (bottom left) and the computation $C_{+}$ (bottom right) with the base input $(1,2,3)$ and the source input $(4,5,6)$. Observe that the output of the intervened neural network matches the output of the intervened symbolic computation, so we have success for this pair of inputs. Probes Probes are generally supervised models trained on the internal representations of networks with the goal of determining what those internal representations encode [Clark et al., 2019, Hupkes et al., 2018, Peters et al., 2018, Tenney et al., 2019]. Probes are fundamentally unable to directly measure causal properties of neural representations, and Ravichander et al. , Elazar et al. , and Geiger et al. have argued that probes are limited in their ability to provide even indirect evidence of causal properties.

The output for an input sequence $\mathbf{x}=(i,j,k)$ is given by $\left(\mathbf{x}W_{1};\mathbf{x}W_{2};\mathbf{x}W_{3}\right)\mathbf{w}$.

In this network, $\mathbf{x}W_{1}$ perfectly encodes $i+j$, and $\mathbf{x}W_{3}$ perfectly encodes $k$. Thus, the identity model probe will be perfect in probing those representations for this information. However, neither representation plays a causal role in the network behavior; only $\mathbf{x}W_{2}$ contributes to the output.

Attribution Methods

Attribution methods aim to quantify the degree to which a network representation contributes to the output prediction of the model, for a specific example or set of examples [Binder et al., 2016, Shrikumar et al., 2016, Springenberg et al., 2014, Sundararajan et al., 2017, Zeiler and Fergus, 2014]. In contrast to probing, the well known integrated gradients method (IG) can be given an unambiguous causal interpretation. Following Sundararajan et al. we define the vector $\emph{IG}(\mathbf{x})$, for an input $\mathbf{x}$ relative to a baseline $\mathbf{b}$, to have $i$th component $\emph{IG}_{i}(\mathbf{x})$ given by the expression on the left:

Abbreviating the weighted average $\alpha\mathbf{x}+(1-\alpha)\mathbf{b}$ by $\mathbf{x}^{\alpha}$, letting $\mathbf{x}^{\alpha,\epsilon}$ be the vector that differs from $\mathbf{x}^{\alpha}$ in that the $i$th coordinate is increased by $\epsilon$, and then expanding the definition of partial derivative, this can be written in the form given on the right. The difference $F(\mathbf{x}^{\alpha,\epsilon})-F(\mathbf{x}^{\alpha})$ is known in the causal literature as the (individual) causal effect on the output (e.g., Imbens and Rubin ) of increasing neuron $i$ by $\epsilon$ relative to the fixed input $\mathbf{x}^{\alpha}$. So, essentially, $\emph{IG}_{i}(\mathbf{x})$ is measuring the average “limiting” causal effect of increasing neuron $i$ along the straight line from the baseline vector to the input vector $\mathbf{x}$, weighted by the difference at $i$ between input and baseline. More recently, Chattopadhyay et al. develop an attribution method that explicitly treats neural models as structured causal models and directly computes the individual causal effect of a feature to determine its attribution.

Attribution methods can measure causal properties, and, in that way, they are similar to the tool of interchange interventions. However, our methodology of causal abstraction analysis provides a framework for systematically measuring and aggregating such causal properties in order to evaluate a precise hypothesis about abstract causal structure.

Causal Abstraction

Our goal is to evaluate whether the internal structure of a neural network realizes an abstract causal process. To concretize this, we turn to formal, broadly interventionist theories of causality [Spirtes et al., 2001, Pearl, 2001], in which causal processes are characterized by effects of interventions, and theories of abstraction [Beckers and Halpern, 2019, Beckers et al., 2020, Chalupka et al., 2016, Rubenstein et al., 2017] where relationships between two causal processes are determined by the presence of systematic correspondences between the effects of interventions.

The notion of abstraction that we employ here is a relatively simple one called constructive abstraction [Beckers and Halpern, 2019]. Informally, a high-level model is a constructive abstraction of a low-level model if there is a way to partition the variables in the low-level model where each high-level variable can be assigned to a low-level partition cell, such that there is a systematic correspondence between interventions on the low-level partition cells and interventions on the high-level variables.

There are two properties of constructive abstraction that make it ideal for neural network analysis. First, the information content of partition cells of low-level variables can be determined by the high-level variables that they correspond to. For neural networks, the partition cells of low-level variables are sets of neurons, and our method supports reasoning at the level of vector representations (sets of neurons). Second, the causal dependencies between partitions of low-level variables are not necessarily preserved as causal dependencies between the high-level variables corresponding to these partitions. For example, the low-level model might be a fully connected neural network, whereas the high-level model might have much sparser connections. For neural network analysis, this means we can find causal abstractions that have far simpler causal structures than the underlying neural networks. We provide an example in the next section.

Causal Abstraction Analysis of Neural Networks

We now describe our methodology in more detail, illustrating the relevant concepts with an example of a neural network performing basic arithmetic. Specifically, suppose that we have a neural network $N_{+}$ that takes in three vector representations $D_{x},D_{y},D_{z}$ representing the integers $x$, $y$, and $z$, and outputs the sum of the three inputs: $N_{+}(D_{x},D_{y},D_{z})=x+y+z$. We seek an informative causal explanation of this network’s behavior.

A human performing this task might follow an algorithm in which they add together the first two numbers and then add that sum to the third number. We can hypothesize that the behavior of $N_{+}$ is explained by this symbolic computation. Specifically, the network combines $D_{x}$ and $D_{y}$ to create an internal representation at some location $L_{1}$ encoding $x+y$; it encodes $z$ at some location $L_{2}$; and $L_{1}$ and $L_{2}$ are composed to encode $a+z$ at the location of the output representation. This hypothesis is given schematically in Figure 1(a).

Following our methodology, we first define the causal model $C_{+}$ in Figure 1(a). Our informal hypothesis that a neural network’s behavior is explained by a simple algorithm can then be restated more formally: $C_{+}$ is a constructive abstraction of the neural network $N_{+}$.

Alignment Search

Now that we have hypothesized that the causal model $C_{+}$ is a causal abstraction of the network $N_{+}$, the next step is to align the neural representations in $N_{+}$ with the variables in $C_{+}$. The input embeddings $D_{x}$, $D_{y}$, and $D_{z}$ must be aligned with the input variables $X$, $Y$, and $Z$ and the output neuron $O$ must be aligned with the output variable $S_{2}$. That leaves the intermediate variables $S_{1}$ and $W$ to be aligned with neural representations at some undetermined locations $L_{1}$ and $L_{2}$. If this were an actual experiment (see below), we would perform an alignment search to consider many possible values for $L_{1}$ and $L_{2}$. Each alignment is a hypothesis about where the network $N_{+}$ stores and uses the values of $S_{1}$ and $W$. For the example, we assume the alignment in Figure 1(a).

Interchange Interventions

Finally, for a given alignment, we experimentally determine whether the neural representations at $L_{1}$ and $L_{2}$ have the same causal properties as $S_{1}$ and $W$. The basic experimental technique is an interchange intervention, in which a neural representation created during prediction on a “base” input is interchanged with the representation created for a “source” input [Geiger et al., 2020]. We now show informally that this method can be used to prove that the causal model $C_{+}$ is a constructive abstraction of the neural network $N_{+}$ (Appendix G has formal details).

to be the output provided by $C_{+}$ when $S_{1}$, the variable representing the intermediate sum, is intervened on and set to the value $x^{\prime}+y^{\prime}$. Thus, for example, if the base input is $C_{+}(1,2,3)=6$, and the source input is $\mathbf{a}^{\prime}=(4,5,6)$, then $C_{+}^{S_{1}\leftarrow\mathbf{a}^{\prime}}(1,2,3)=4+5+3=12$. This process is depicted in Figure 1(c).

to be the output provided by $N_{+}$ processing the input $D$ when the representation at location $L_{1}$ is replaced with the representation at location $L_{1}$ created when $N_{+}$ is processing the input $D^{\prime}$. This process is depicted in Figure 1(b).

With these two definitions, we can define what it means to test the hypothesis that $N_{+}$ computes $x+y$ at position $L_{1}$. Where $D_{\mathbf{a}}$ is an embedding for $\mathbf{a}$ and $D_{\mathbf{a}^{\prime}}$ is an embedding for $\mathbf{a}^{\prime}$, we test:

If this equality holds for all source and base inputs $\mathbf{a}$ and $\mathbf{a}^{\prime}$, then we can conclude that, for every intervention on $S_{1}$, there is an equivalent intervention on $L_{1}$. If we can establish a corresponding claim for $W$ and $L_{2}$, then we have shown that $C_{+}$ is a constructive abstraction of $N_{+}$, since the inputs’ relationships are established by our embedding and there are no other interventions on $C_{+}$ to test.

Analysis

Suppose that all of our intervention experiments verify our hypothesis that $C_{+}$ is a constructive abstraction of $N_{+}$ with variables $S_{1}$ and $W$ aligned to neural representations at $L_{1}$ and $L_{2}$. This explains network behavior by resolving two crucial questions.

First, we learn what information is encoded in the representations $L_{1}$ and $L_{2}$. Neural representations encode the values of the high-level variables they are aligned with. The location $L_{1}$ encodes the variable $S_{1}$ and the location $L_{2}$ encodes the variable $W$. This is similar to what probing achieves. However, our method is crucially different from probing. In probing, information content is established through purely correlational properties, meaning a neural representation with no causal role in network behavior can be successfully probed, as we showed in Section 2. In causal abstraction analysis, information content is established through purely causal properties, ensuring that the neural representation is actually implicated in model behavior.

Second, we learn what causal role $L_{1}$ and $L_{2}$ play in network behavior. Neural representations play a parallel causal role to their aligned high-level variables. At the location $L_{1}$, $D_{x}$ and $D_{y}$ are composed to form a neural representation with content $x+y$ that is then composed with $L_{2}$ to create an output. The fact that $S_{1}$ doesn’t depend on $z$ tells us that while $L_{1}$ depends on $D_{z}$ and representations at $L_{1}$ may even correlate with $z$, the information about $z$ is not causally represented at $L_{1}$. At the location $L_{2}$, the value of $z$ is simply repeated and then composed with $L_{1}$ to create a final output.

Our method assigns causally impactful information content, but also identifies the abstract causal structure along which representations are composed. It thus encompasses and improves on both correlational (probing) and attribution methods.

The Natural Language Inference Task and Models

The Multiply Quantified NLI (MQNLI) dataset of Geiger et al. contains templatically generated English-language NLI examples that involve very complex interactions between quantifiers, negation, and modifiers. We provide a few examples in Figure 2(b); the empty-string symbol $\varepsilon$ ensures perfect alignments at the token level both between premises and hypotheses and across all examples.

The MQNLI examples are labeled using an algorithmic implementation of the natural logic of MacCartney and Manning over tree structures, and MQNLI has train/dev/test splits that vary in their difficulty. In the hardest setting, the train set is provably the minimal set of examples required to ensure that the dev and test sets can be perfectly solved by a simple symbolic model; in the easier settings, the train set redundantly encodes necessary information, which might allow a model to perform perfectly in assessment by memorization despite not having found a truly general solution. For a fuller review of the dataset, see Appendix A.

MQNLI is a fitting benchmark given our goals for a few reasons. First, we can focus on the hardest splits that can be generated, which will stress-test our NLI architectures in a standard behavioral way. Second, the MQNLI labeling algorithm itself suggests an appropriate causal model of the data-generating process. Figure 2(a) summarizes this model in tree form, and it is presented in full detail in Geiger et al. . This allows us to rigorously assess whether a neural network has learned to implement variants of this causal model. The complexity of the MQNLI examples creates many opportunities to do this in linguistically interesting ways.

Models

We evaluated two models on MQNLI: a randomly initialized multilayered Bidirectional LSTM (BiLSTM; Schuster and Paliwal ) and a BERT-based classifier model in which the English bert-base parameters [Devlin et al., 2019] are fine-tuned on the MQNLI train set. Output predictions are computed using the final representation above the [CLS] token. Models are trained to predict the relation of every pair of aligned phrases in Figure 2(a). Additional model and training details are given in Appendix B.

Results

Figure 2(c) summarizes the results of our BERT and BiLSTM models on the hardest fair generalization task Geiger et al. creates with MQNLI. We find that our BiLSTM model is not able to learn this task, and that our BERT model is able to achieve high accuracy. The only models in Geiger et al. able to achieve above 50% accuracy were task-specific tree-structured models with the structure of the tree in Figure 2(a). Thus, our BERT-based model is the first general-purpose model able to achieve good performance on this hard generalization task. Without pretraining, the BERT-based model achieves $\approx$49.1%, confirming that pretraining is essential, as expected.

A natural hypothesis is that the BERT-based model achieves this high performance because it has in effect induced some approximation to the tree-like structure of the data-generating process in its own internal layers. With causal abstraction analysis, we are actually in a position to test this hypothesis.

A Case Study in Structural Neural Network Analysis

We proceed just as we did for the simple motivating example in Section 3, except that we are now seeking to assess the extent to which the natural logic algebra in Figure 2(a) is a causal abstraction of the trained neural models in the above section.

The hallmark of Figure 2(a) is that it defines an alignment between premise and hypothesis at both lexical and phrasal levels. This permits us to run interchange interventions in a naturally compositional way. For a given non-leaf node ${\mathit{N}}$ in Figure 2(a), let $C_{\emph{NatLog}}^{\mathit{N}}$ be a submodel of $C_{\emph{NatLog}}$ that computes the relation between the aligned phrases under ${\mathit{N}}$ and uses them to compute the final output relation between premise and hypothesis. For example, let $C_{\emph{NatLog}}^{\text{NP}_{\text{Obj}}}$ be the submodel of $C_{\emph{NatLog}}$ that computes the relation between the two aligned object noun phrases and then uses that relation in computing the final output relation between premise and hypothesis (see Figure 3 right). We would like to ask whether our trained neural models also compute this relation between object noun phrases and use it to make a final prediction. We can pose this same question for other nodes which correspond to a pair of aligned subphrases.

Alignment Search

For each ${\mathit{N}}$, we search for an alignment between a neural representation in $N_{\emph{NLI}}$ and the variable ${\mathit{N}}$ in $C_{\emph{NatLog}}^{\mathit{N}}$. In principle, any location in the network could be the right one for any causal model. Testing every hypothesis in this space would be intractable. Thus, for each $C_{\emph{NatLog}}^{\mathit{N}}$, we consider a restricted set of hidden representations based on the identity of ${\mathit{N}}$. The BERT model we use has 12 Transformer layers [Vaswani et al., 2017], meaning that there are 12 hidden representations for each input token. Each alignment search considers aligning the intermediate high-level variable with dozens of possible locations in the grid of BERT representations. Specifically, the following locations were considered for each ${\mathit{N}}$:

$\text{Q}_{\text{Subj}}$, $\text{Adj}_{\text{Subj}}$, $\text{N}_{\text{Subj}}$, Neg, Adv, V, $\text{Q}_{\text{Obj}}$, $\text{Adj}_{\text{Obj}}$, $\text{N}_{\text{Obj}}$: hidden representations above the two descendant leaf tokens.

$\text{NP}_{\text{Subj}},\text{VP}$, and $\text{NP}_{\text{Obj}}$: same but above the four descendant leaf tokens.

$\text{QP}_{\text{Obj}}$: hidden representations above $\text{Q}_{\text{Obj}}^{P}$ and $\text{Q}_{\text{Obj}}^{H}$.

NegP: same but above $\text{Neg}^{P}$ and $\text{Neg}^{H}$.

All nodes (for BERT): same but above [CLS] and [SEP].

For each alignment considered, we performed a full causal abstraction analysis. We report the results from the best alignments in Table 1, and we summarize the results from all alignments in Appendix D.

Interchange Interventions

We first focus on our high-level causal models. Consider a non-leaf node ${\mathit{N}}$ from Figure 2(a) and two input token sequences $e$ and $e^{\prime}$ from MQNLI. Define

to be the output provided by the causal model $C_{\emph{NatLog}}^{\mathit{N}}$ when processing input $e$ where the relation between the aligned subphrases under the node ${\mathit{N}}$ is changed to the relation between those subphrases in $e^{\prime}$. For example, simplifying for the sake of exposition, suppose $e$ is (some happy baker, no $\epsilon$ baker), which has output label contradiction, and suppose $e^{\prime}$ is (every happy person, some happy baker), which has output label entailment. We wish to intervene on the noun phrase, so ${\mathit{N}}=\text{NP}$. In $e$, the noun phrase relation is entailment; in $e^{\prime}$, it is reverse entailment. Thus, $C_{\emph{NatLog}}^{{\text{NP}}\leftarrow e^{\prime}}(e)$ changes the object noun phrase relation in $e$ to entailment while holding everything else about $e$ constant. This results in the output label for the example (some happy person, no $\epsilon$ baker), which is neutral.

Next, we consider interventions in a neural model $N_{\emph{NLI}}$. Define

to be the output provided by $N_{\emph{NLI}}$ processing the input $e$ when the representation at location $L$ is replaced with the representation at location $L$ created when $N_{\emph{NLI}}$ is processing $e^{\prime}$. This is exactly the process depicted in Figure 1, except now the networks are the complex trained networks of Section 4.

Our hypothesis linking Figure 2(a) with a model $N_{\emph{NLI}}$ takes the same form as (3). The causal model $C_{\emph{NatLog}}^{\mathit{N}}$ is a constructive abstraction of $N_{\emph{NLI}}$ when, for some representation location $L$, it is the case that, for all MQNLI examples $e$ and $e^{\prime}$, we have

This asserts a correspondence between interventions on the representations at $L$ in network $N_{\emph{NLI}}$ and interventions on the variable ${\mathit{N}}$ in the causal model $C_{\emph{NatLog}}^{\mathit{N}}$. If it holds, then $N_{\emph{NLI}}$ computes the relation between the aligned phrases under the node ${\mathit{N}}$ and uses this information to compute the relation between the premise and hypothesis.

We call a pair of examples $(e,e^{\prime})$ successful if it satisfies equation (6), i.e., interventions in both the target causal model and neural model produce equal results. In addition, to isolate the causal impact of our interventions, we specifically focus on pairs $(e,e^{\prime})$ for which performing the intervention produces a different output value than without the intervention. We call a pair $(e,e^{\prime})$ impactful if:

Quantifying Partial Success

Equation (6) universally quantifies over all examples. We do not expect this kind of perfect correspondence to emerge in practice for real problems: neural network training is often approximate and variable in nature, and even our best model does not achieve perfect performance. However, we can still ask how widely (6) holds for a given model. To do this, we seek to find the largest subset of MQNLI on which $C_{\emph{NatLog}}^{\mathit{N}}$ is an abstraction of our neural models, for each non-leaf node ${\mathit{N}}$ in $C_{\emph{NatLog}}$.

More specifically, considering each example in MQNLI as a vertex in a graph, we add an undirected edge between two examples $e_{i}$ and $e_{j}$ if and only if both the ordered pairs $(e_{i},e_{j})$ and $(e_{j},e_{i})$ satisfy (6). In other words, $C_{\emph{NatLog}}^{\mathit{N}}$ is an abstraction of a neural model on a subset of examples $S$ of MQNLI if and only if all examples in $S$ form a clique.

The number of interventions we need to run scales quadratically with the number of inputs we consider, so we sample $1000$ MQNLI examples, producing a total of $1000^{2}=$ 1M ordered pairs. We only consider examples for which the neural network outputs a correct label. For each node ${\mathit{N}}$ and each of its corresponding neural network locations $L$, we perform interventions on all of these pairs.

We choose to measure the largest clique with at least one impactful edge, because (1) the causal abstraction relation holds with full force on that clique, but other measures such as the total number of connections lack this theoretical grounding, and (2) if a clique has at least one impactful edge, that guarantees the high-level variable is being used.

Results and Analysis

For each target causal model node ${\mathit{N}}$ and neural network representation location $L$, we construct a graph as described above with 1000 examples as vertices and add an edge between two examples $e_{i}$ and $e_{j}$ if and only if both $(e_{i},e_{j})$ and $(e_{j},e_{i})$ are successful. We then find the largest clique in this graph with at least one impactful edge and record its size.

Table 1(a) shows, for each causal model node ${\mathit{N}}$, the maximum size of cliques found among all neural locations. With this stricter impactful criterion (as opposed to simply using intervention success), our results show that, for almost all nodes ${\mathit{N}}$, our target causal model $C_{\emph{NatLog}}^{\mathit{N}}$ is indeed a causal abstraction of BERT on a significant number of examples in our dataset. These subsets are much smaller for the BiLSTM model.

We also investigated alternative high-level causal structures that are not variants of $C_{\emph{NatLog}}$ from Figure 2(a). Specifically, we consider alternative models in a “neighborhood” around the model $C_{\emph{NatLog}}^{\text{NP}_{\text{Obj}}}$ that can be obtained by adding one leaf, or by removing one or two leaves to the composition. These results are in Table 1(b). Remarkably, all of these alternative models result in smaller clique sizes, significantly so for many of them. This further supports the significance of our results.

This analysis is similar to the analysis of our hypothetical addition example in Section 3, except for two crucial differences. First, for each variable ${\mathit{N}}$, we are hypothesizing that the causal model $C_{\emph{NatLog}}^{\mathit{N}}$ is an abstraction of $N_{\emph{NLI}}$, whereas in the addition example there was only one model. To investigate this difference, we take ${\mathit{N}}=\text{NP}_{\text{Obj}}$ as a paradigm case, as it is the model with the strongest results. (The results for other nodes are in Appendix D.) Second, we only achieved partial experimental success, whereas in the addition example we assumed complete success. Crucially, this means that the following analysis will be valid only on subsets of the input space on which the abstraction relation holds between $N_{\emph{NLI}}$ and $C_{\emph{NatLog}}^{\text{NP}_{\text{Obj}}}$.

We visualize the results of our intervention experiments for the node $\text{NP}_{\text{Obj}}$ in Figure 4. The alignment with the largest subset of inputs aligns the $\text{NP}_{\text{Obj}}$ variable in $C_{\emph{NatLog}}^{\text{NP}_{\text{Obj}}}$ with the neural representation on the fourth layer of BERT above the $\text{Adj}_{\text{Obj}}^{P}$ token (see Figure 3). Because neural representations encode the value of their aligned variables and play a parallel causal role to their high-level variables, we know that, on this subset of input examples, at the fourth neural representation above the $\text{Adj}_{\text{Obj}}^{P}$ token, the four input embeddings for the object nouns and adjectives in the premise and hypothesis are composed to form a neural representation with information content of the relation between the object noun phrases in the premise and hypothesis. Then this representation is composed with the other input-embeddings to create an output representing the relation between the premise and hypothesis.

2 Comparison with Other Structural Analysis Methods

We probed neural representation locations for the relation between aligned subexpressions on a subset of 12,800 randomly selected MQNLI examples. For a pair of aligned subexpressions below a node ${\mathit{N}}$ in Figure 2(a), we probe the columns above the same set of restricted class of tokens as described in Section 5.1.

To evaluate these probes, we report accuracy as well as selectivity as defined by Hewitt and Liang : probe accuracy minus control accuracy, where control accuracy is the train set accuracy of a probe with the same architecture but trained on a control task to factor out probe success that can be attributed to the probe model itself. Our control task is to learn a random mapping from node types to semantic relations; see Appendix C for full details on how this task was constructed.

Figure 4 summarizes our probing results for ${\mathit{N}}=\text{NP}_{\text{Obj}}$, along with corresponding interchange intervention results for comparison. Probes tell us that information about the relation between the aligned noun phrases is encoded in nearly all of the locations we considered, and using the selectivity metric does not result in any qualitative change. In contrast, our intervention heatmaps indicate only a small number of locations store this information in a causally relevant way. Clearly, our intervention experiments are far more discriminating than probes. Appendix D provides examples involving other variables along with the intervention experiments, where the general trend of interchange interventions being more discriminating holds.

Integrated Gradients

Attribution methods that estimate feature importance can measure causal properties of neural representations, but a single feature importance method is an impoverished characterization of a representation’s role in network behavior. Whereas our interchange interventions gave us high-level information about how a neural representation is composed and what it is composed into, attribution methods simply tell us “how much” a representation contributes to the network output on a give input. Moreover, intervention interchanges provide a rich, high-level characterization of causal structure on a space of inputs.

We use integrated gradients on our models to verify the intuitive hypothesis that if a premise and hypothesis differ by a single token, then the neural representations above that token should be more causally responsible for the network output than other representations. For example, given premise ‘Every sleepy cat meows’ and hypothesis ‘Some hungry cat meows’, the attributive modifier position is different and the rest are matched. The neural representations above the adjective tokens sleepy and hungry should be more important for the network output than others, because if those adjectives were the same, the example label would change from neutral to entailment. We summarize the results of our integrated gradient experiments in Appendix E, where we confirm our intuitive hypothesis.

Conclusion

We have introduced a methodology for deriving interpretable causal explanations of neural network behaviors, grounded in a formal theory of causal abstraction. The methodology involves first formulating a hypothesis in the form of a high-level, interpretable causal model, then searching for an alignment between the neural network and the causal model, and finally verifying experimentally that the neural representations encode the same causal properties and information content as the corresponding components of the high-level causal model. As a case study demonstrating the feasibility of the approach, we analyzed neural models trained on the semantically formidable MQNLI dataset. Guided by the intuition that success on this challenging task may call for a way of recapitulating the causal structure of the natural logic model that generates the MQNLI data, we were able to verify the hypothesis that a state-of-the-art BERT-based model partially realizes this structure, whereas baseline models that do not perform as well fail to do so. This suggestive case study demonstrates that our theoretically grounded methodology can work in practice.

Acknowledgments and Disclosure of Funding

Our thanks to Amir Feder, Noah Goodman, Elisa Kreiss, Josh Rozner, Zhengxuan Wu, and our anonymous reviewers. This research is supported in part by grants from Facebook and Google.

References

Appendix A Additional Details on MQNLI

The MQNLI dataset contains sentences of the form

$\text{Q}_{\text{S}}$ $\text{Adj}_{\text{S}}$ $\text{N}_{\text{S}}$ $\text{Neg}_{\text{}}$ $\text{Adv}_{\text{}}$ $\text{V}_{\text{}}$ $\text{Q}_{\text{O}}$ $\text{Adj}_{\text{O}}$ $\text{N}_{\text{O}}$

where $\text{N}_{\text{S}}$ and $\text{N}_{\text{O}}$ are nouns, $\text{V}_{\text{}}$ is a verb, $\text{Adj}_{\text{S}}$ and $\text{Adj}_{\text{O}}$ are adjectives, and $\text{Adv}_{\text{}}$ is an adverb. These categories all have 100 words. $\text{Neg}_{\text{}}$ is does not, and $\text{Q}_{\text{S}}$ and $\text{Q}_{\text{O}}$ can be every, not every, some, or no. Additionally, $\text{Adj}_{\text{S}}$, $\text{Adj}_{\text{O}}$, $\text{Adv}_{\text{}}$, and $\text{Neg}_{\text{}}$ can be the empty string $\varepsilon$.

NLI examples are constructed so that non-identical non-empty nouns, adjectives, verbs, and adverbs with identical positions in $s_{p}$ and $s_{h}$ are semantically unrelated. This means that the learning task is trivial for these lexical items, as the correct relation is equivalence when they are identical and independence when they are not identical.

For our experiments, we used a train set with 500K examples, a dev set with 60k examples, and a test set with 10K examples – the most difficult generalization scheme of Geiger et al. .

A.2 A Natural Logic Causal Model

Geiger et al. construct a natural logic model that solves MQNLI using a formalization they call composition trees, which is easily translated into the causal model we call $C_{\emph{NatLog}}$. Natural logic is a flexible approach to doing logical inference directly on natural language expressions [Icard and Moss, 2013, MacCartney and Manning, 2007, van Benthem, 2008] where the semantic relations between phrases are compositionally computed from the semantic relations between aligned subphrases and projectivity signatures, which encode how semantic operators interact compositionally with their arguments (which are semantic relations). The causal model $C_{\emph{NatLog}}$ performs inference on aligned semantic parse trees that represent both the premise and hypothesis as a single structure and calculates semantic relations between all subphrases compositionally.

Appendix B Model Training and Interchange Experiment Details

We evaluated two models on MQNLI: a multi-layered bidirectional LSTM baseline and a Transformer-based model trained to do masked language modeling and next-sentence prediction [Devlin et al., 2019]. We rely on the uncased BERT-base initial parameters from Hugging Face transformers [Wolf et al., 2019]. For both models, we concatenate the premise $s_{p}$ and hypothesis $s_{h}$ into one string with special separator tokens: [CLS] $s_{p}$ [SEP] $s_{h}$ [SEP].

For the BiLSTM, we concatenate the hidden state above the last [SEP] and the [CLS] in the last layer for the forward and backward directions respectively to obtain a representation for the whole input, and then apply three linear transformations on top of that. The final transformation outputs a logit score for each class in the label space.

For the BERT model, we apply one linear transformation to the final layer’s hidden representation above the [CLS] token to obtain a logit score for each label class.

In the original setting of MQNLI, some positions in the premise and hypothesis consist of two words such as not every in $\text{Q}_{\text{S}}$ and $\text{Q}_{\text{O}}$ and does not in the leaf nodes Neg${}^{\text{P}}$ and Neg${}^{\text{H}}$ (as shown in the beginning of Section A.1). We treat them as two separate tokens in order to utilize BERT’s knowledge of these function words. To ensure all sentences have identical length, we introduce one extra empty string tokens $\varepsilon$ to single-word quantifiers and two such tokens in the place of NegP and NegH for sentences without negation.

For consistency, we use the same tokenization method for both models.

B.2 Dataset Augmentation with Labeled Subphrases

The hard but fair MQNLI generalization task requires the dataset to explicitly expose the model to labels for each intermediate node that is a relation in $C_{\emph{NatLog}}$. For each training example $(s_{p},s_{h},y)\in\mathcal{S}$, we create an additional example $(s_{p}^{N},s_{h}^{N},y^{N})$ for each node $N$. $(s_{p}^{N},s_{h}^{N})$ is a subphrase pair made up of all the leaf tokens under node $N$ in the original input $(s_{p},s_{h})$, and $y^{N}$ is the relation computed by $C_{\emph{NatLog}}$ for that subphrase pair. The set of labels we use for these subphrase examples is disjoint from that of the full-sentence examples. During training, the augmented examples are coupled with original examples in each batch. For BERT, the subphrase pairs occupy their original positions in the sentence, while we pad and apply an attention mask over all other positions. For the BiLSTM, we align them to the left, with [SEP] in between the two parts of the pair.

We performed an ablation experiment to test whether removing the augmented examples would affect BERT’s performance. Using the same grid-search setting, we see that BERT’s dev set accuracy decreased from 88.25% to 55.42%, and test set accuracy decreased from 88.50% to 54.51%. This indeed shows that the above data augmentation method is important for BERT to learn the type of generalization required for the hard MQNLI task.

B.3 Training Procedure

For the BiLSTM, we use 256 dimensions for token embeddings and 128 dimensions for the hidden states in each LSTM direction. We grid search for $\{2,4,6\}$ layers. We randomly initialize each element in the token embeddings from the distribution $\mathcal{N}(0,1)$ scaled down by a factor of 0.1. We use a batch size of $768=64\times 12$, with 64 original examples per batch and 11 augmented examples for each one. We apply a dropout of 0.1, and grid search for learning rates in $\{0.001,0.0001\}$. We train for a maximum of 400 epochs and perform early stopping when the dev set accuracy does not increase for 20 epochs. We train each grid search setting 3 times with different random seeds.

For BERT, we use the same model architecture for the uncased base variant. We use a batch size of $192=16\times 12$, and grid search for learning rates in $\{2.0\times 10^{-5},5.0\times 10^{-5}\}$. We train for a maximum of {3, 4} epochs. We warm up the learning rate linearly from 0 to the specified value in the first 25% of steps of the first epoch, and linearly decrease the learning rate to following that until the end of training.

All models were trained with 1 GPU core on a cluster with models including GeForce RTX 2080 Ti, GeForce GTX Titan X, Titan XP and Titan V, each with 11-12GB memory. Each instance of the grid search took on average 5.5 hours to train. We repeated each grid search setting with 4 different random seeds and took the instance with the highest dev set accuracy.

B.4 Interchange experiment details

There are 14 intermediate nodes in the high-level causal model (NegP, $\text{QP}_{\text{Obj}}$, $\text{Q}_{\text{Subj}}$, $\text{NP}_{\text{Subj}}$, $\text{Adj}_{\text{Subj}}$, $\text{N}_{\text{Subj}}$, Neg, VP, $\text{Adv},$ V, $\text{Q}_{\text{Obj}}$, $\text{NP}_{\text{Obj}}$, $\text{Adj}_{\text{Obj}}$, $\text{N}_{\text{Obj}}$). For each high-level node, we conducted a set of interchange experiments on each one of 11 BERT layers (excluding the final layer, since only the [CLS] token causally impacts the output). Each high-level node has its own fixed set of hand-specified intervention locations in the time-step/sentence length dimension, and we use the same intervention locations on each layer. For each of the $14\times 11=154$ interchange experiments, it took on average 1.15 hours to run using the same computation resources mentioned above.

Appendix C Probing Details

We train the probes on hidden representations of a set of 12,800 examples that are randomly selected from the model’s original training set. We additionally take 2,000 examples to form a development set for early stopping. We filter out examples for which the model outputs a wrong prediction.

Using the same computation resources described above, each grid search setting took approximately 5 hours to run. For each grid search setting we trained a separate probe for every possible $\langle\text{causal model node, BERT representation}\rangle$ combination, where for the latter we use the intervention locations outlined in the “Alignment Search" part of Section 5.1 on each BERT layer.

C.2 Control Task

C.3 Extended Probe Analysis

In Figures 5–7 we report some more representative selectivity and accuracy results for our probing experiments on BERT trained on the hard variant, juxtaposed against intervention experiments on the same model. For open-class words and full phrases, probing and intervention show similar trends. For aligned closed-class words, we find near-zero selectivity because the domain of the control function is so small.

In general, probing and intervention experiments for relations between aligned single open-class words (i.e., $\text{N}_{\text{Subj}}$, $\text{Adj}_{\text{Subj}}$, $\text{N}_{\text{Obj}}$, $\text{Adj}_{\text{Obj}}$, Adv, V) show similar trends, which can be seen in Figures 4(c)–4(b). Every location except those above the [CLS] and [SEP] tokens has a near-100% accuracy, while selectivity is only high in the last few layers. Lower layers of BERT contains more information about word identity and hence may allow the probe to memorize each input pair, resulting in higher control task accuracy and lower selectivity for lower layers.

Probing experiments for relations between aligned multi-word subphrases (i.e., $\text{NP}_{\text{Subj}}$, VP, $\text{NP}_{\text{Obj}}$, $\text{QP}_{\text{Obj}}$ and NegP) show similar trends as shown in the row of figures 6(m) to 6(h). As described in Section C.2, all control probes for these achieve near-random performance, so selectivity and accuracy differ by the random baseline accuracy, which is evident by comparing figures 6(m) and 6(n).

On the other hand, probing experiments for aligned closed-class words (quantifiers and negation) have near-zero selectivity, as shown in Figure 6(a). This is because the domain of the control function is the small set of closed-class word pairs, so memorizing the identity of these words becomes trivial for the probe.

Appendix D Probing and Intervention Heatmaps

Appendix E Integrated Gradients

We report attributions for the first BERT layer; later layers tend to concentrate importance onto the [CLS] token, since it is the direct basis for the classifier head in our model. To simplify the analysis, we restrict attention to examples in which exactly one position is different across the premise and hypothesis, and ‘Matched’ is a randomly selected position from elsewhere in the example. We see that the ‘Matched’ are positive in general, which aligns with our expectation that they are the most important positions in these examples (Figure 8).

Appendix F Background on Causal Models and Causal Abstraction

In this appendix we provide relevant background on causal models and causal abstraction, sufficient to define the notion of constructive abstraction.

(Signatures) A signature $S$ is a pair $(\mathcal{V},\mathcal{R})$, where $\mathcal{V}$ is a set of variables and $\mathcal{R}$ is a function that associates with every variable $X\in\mathcal{V}$ a nonempty set $\mathcal{R}(X)$ of possible values. If $\mathbf{X}=(X_{1},\dots,X_{n})$, $\mathcal{R}(\mathbf{X})$ denotes the cross product $\mathcal{R}(X_{1})\times\dots\times\mathcal{R}(X_{n})$.

(Causal models) A causal model $M$ is a pair $(\mathcal{S},\mathcal{F})$, where $\mathcal{S}$ is a signature and $\mathcal{F}$ defines a function that associates with each variable $X$ a structural equation $\mathcal{F}^{X}$ giving the value of $X$ in terms of the values of other variables. Formally, the equation $\mathcal{F}^{X}$ maps $\mathcal{R}(\mathcal{V}-\{X\})$ to $\mathcal{R}(X)$, so $\mathcal{F}^{X}$ determines the value of $X$, given the values of all the other variables in $\mathcal{V}$.

(Dependence) $X$ causes $Y$ according to $M$, denoted $X\rightsquigarrow Y$, if there is some setting of the variables other than $X$ and $Y$ such that varying the value of $X$ results in a variation in the value of $Y$; that is, there is a setting $\mathbf{z}$ of the variables $\mathbf{Z}=\mathcal{V}-\{X,Y\}$ and values $x$ and $x^{\prime}$ of $X$ $\mathcal{F}^{Y}(x,\mathbf{z})\neq\mathcal{F}^{Y}(x^{\prime},\mathbf{z})$.

(Intervention) An intervention $i$ has the form $\mathbf{X}\leftarrow\mathbf{x}$, where $\mathbf{X}$ is a vector of variables. Intuitively, this means that the values of the variables in $\mathbf{X}$ are set to $\mathbf{x}$. Setting the value of some variables $\mathbf{X}\leftarrow\mathbf{x}$ in a causal model $M=(\mathcal{S},\mathcal{F})$ results in a new causal model, denoted $i(M)$, which is identical to $M$, except that $\mathcal{F}$ is replaced by $i(\mathcal{F})$: for each variable $Y\not\in\mathbf{X}$, $i(\mathcal{F}^{Y})=\mathcal{F}^{Y}$ (i.e., the equation for $Y$ is unchanged), while for each $X^{\prime}\in\mathbf{X}$ , $i(F^{X^{\prime}})$ is the constant function sending all arguments to $x^{\prime}$ (where $x^{\prime}$ is the value in $\mathbf{x}$ corresponding to $X_{i}$).

When we write out the structured equations for a variable $X$, for simplicity’s sake, we treat $\mathcal{F}^{X}$ as a map from $\mathcal{R}(\{Y\in\mathcal{V}:Y\rightsquigarrow X\})$ to $\mathcal{R}(X)$.

Note that interventions $\mathbf{X}\leftarrow\mathbf{x}$ correspond 1–1 with variable settings $\mathbf{x}$. We make use of this in what follows.

F.2 Constructive Abstraction

The following definitions are in agreement with the definitions from Beckers and Halpern , but differ somewhat in presentation. We additionally omit exogenous variables, as they play no role in our deterministic setting. In this section we take causal models to be pairs $(M,\mathcal{I})$, with a set $\mathcal{I}$ of admissible interventions made explicit.

(Projection and Inverse Projection) Given some $\mathbf{v}\in\mathcal{R}(\mathcal{V})$ and $\mathbf{X}\subseteq\mathcal{V}$, define $\mathsf{Proj}(\mathbf{v},\mathbf{X})$ to be the restriction of $\mathbf{v}$ to the variables in $\mathbf{X}$. Given some $\mathbf{x}\subseteq\mathcal{V}(\mathbf{X})$, the inverse $\mathsf{Proj}^{-1}(\mathbf{x})$ is defined as usual:

We are interested in (possibly partial) functions $\tau:\mathcal{R}_{L}(\mathcal{V}_{L})\rightarrow\mathcal{R}_{H}(\mathcal{V}_{H})$ mapping settings of low-level variables to settings of high-level variables. Such a function $\tau$ naturally induces a function $\omega_{\tau}$ between sets of interventions, where $\omega_{\tau}(\mathbf{x})=\mathbf{y}$ just in case

We are now in a position to define $\tau$-abstraction:

($\tau$-abstraction) Fix a function $\tau:\mathcal{R}_{L}(\mathcal{V}_{L})\rightarrow\mathcal{R}_{H}(\mathcal{V}_{H})$, which in turn fixes $\omega_{\tau}:\mathcal{I}_{L}\rightarrow\mathcal{I}_{H}$. We say $(M_{H},\mathcal{I}_{H})$ is a $\tau$-abstraction of $(M_{L},\mathcal{I}_{L})$ if the following hold:

for all $i_{L}\in\mathcal{I}_{L}$ we have $\tau(i_{L}(M_{L}))=\omega_{\tau}(i_{L})(M_{H})$.

One way to think of this is: $\tau$ is a map from $\mathcal{R}(\mathcal{V}_{L})$ to $\mathcal{R}(\mathcal{V}_{H})$, which in turn induces a map $\omega_{\tau}$ from the space of projections on $\mathcal{R}(\mathcal{V}_{L})$ to projections on $\mathcal{R}(\mathcal{V}_{H})$. The conditions on $\tau$-abstraction below then simply become that $\tau$ and $\omega_{\tau}$ are both total and surjective on their respective (co)domains, and a second condition that can be easily encoded in terms of potential outcomes. For any setting/projection $\mathbf{x}$ at the low-level, we require that $M_{L}\models\mathbf{v}_{\mathbf{x}}$ iff $M_{H}\models\tau(\mathbf{v})_{\omega_{\tau}(\mathbf{x})}$.

Finally, to be a constructive $\tau$-abstraction we simply require that $\tau$ decompose into a family of “component” functions, as below.

$(M_{H},\mathcal{I}_{H})$ is a constructive $\tau$-abstraction of $(M_{L},\mathcal{I}_{L})$ if, in addition to being a $\tau$-abstraction, we can associate with each $X_{H}$ a subset $P_{X_{H}}$ of $\mathcal{V}_{L}$, such that the mapping $\tau:\mathcal{R}(\mathcal{V}_{L})\rightarrow\mathcal{R}(\mathcal{V}_{H})$ decomposes into a family of functions $\tau_{X_{H}}:\mathcal{R}(P_{X_{H}})\rightarrow\mathcal{R}(X_{H})$. We say $M_{H}$ is a constructive abstraction of $M_{L}$ if it is a constructive $\tau$-abstraction for some $\tau$.

In other words, for a constructive abstraction it suffices to define the component functions $\tau_{X_{H}}$, as these completely determine $\tau$. In fact, the maps $\tau_{X_{H}}$ more generally induce a (partial) function from projections of $\mathcal{R}(\mathcal{V}_{L})$ to (in fact, onto) projections of $\mathcal{R}(\mathcal{V}_{H})$ in the following sense. For any setting $\mathbf{h}=[h_{1}\dots h_{k}]$ of high-level variables $H_{1},\dots,H_{k}$ we can find low-level setting $\mathbf{y}$ such that projections of $\mathbf{y}$ map via $\tau_{H_{i}}$ to $h_{i}$. Slightly abusing notation, denote this (partial) low-level setting $\mathbf{y}$ as $\tau^{-1}(\mathbf{h})$. So, in particular when $\mathbf{h}$ corresponds to an intervention in $\mathcal{I}_{H}$, the setting $\tau^{-1}(\mathbf{h})$ should specify a corresponding intervention in $\mathcal{I}_{L}$. Indeed, point (2) of Def. F.6 tells us that (the intervention corresponding to) $\tau^{-1}(\mathbf{h})$ should be mapped via $\omega_{\tau}$ to (the intervention corresponding to) $\mathbf{h}$.

N_{+} In the main text, we did not provide a specific identity for $N_{+}$. Here, we define $N_{+}$ to be a feed forward network, which we represent directly as a causal model $C_{N_{+}}=(\mathcal{V}_{N_{+}},\mathcal{R}_{N_{+}},\mathcal{F}_{N_{+}})$. The location $L_{1}$ from Figure 1 is the hidden unit $H_{3}$, the location $L_{2}$ is the hidden unit $H_{1}$.

C_{+} is an abstraction of $N_{+}$ We now prove that $C_{+}$ is an abstraction $C_{N_{+}}$

Let $\mathcal{I}_{+}$ be the set of all interventions on $C_{+}$ that determine values for (at least) $X$, $Y$, and $Z$. Let $\mathcal{I}_{N_{+}}=\emph{dom}(\omega_{\tau})$. That is, $\mathcal{I}_{N_{+}}$ includes exactly the (interventions corresponding to) projections of $\mathcal{R}_{N_{+}}(\mathcal{V}_{N_{+}})$ that map via $\omega_{\tau}$ to some admissible intervention on $C_{+}$. Because elements of $\mathcal{I}_{+}$ always determine values for $X,Y,Z$, every intervention in $\mathcal{I}_{N_{+}}$ determines a value for each of $D_{x},D_{y},D_{z}$. In fact, these values are guaranteed to be in the domains of $\tau_{X},\tau_{Y},\tau_{Z}$, respectively.

We now prove the three conditions guaranteeing $(C_{+},\mathcal{I}_{+})$ is a $\tau$-abstraction of $(C_{N_{+}},\mathcal{I}_{N_{+}})$.

(1) The first point is that the map $\tau$ is surjective. Take an arbitrary $(x,y,z,w,s_{1},s_{2})\in\mathcal{R}_{+}(\mathcal{V}_{+})$. We determine an element of $\mathcal{R}_{N_{+}}(\mathcal{V}_{N_{+}})$ as follows: $[d_{x}d_{y}d_{z}]=B^{-1}([x,y,z])$, $\left[h_{1}h_{2}h_{3}\right]=\left[s_{1}d_{2}s_{1}\right]$, and $o=s_{2}$. It’s then clear that $\tau(d_{x},d_{y},d_{z},h_{1},h_{2},h_{3},o)=(x,y,z,w,s_{1},s_{2})$. As $(x,y,z,w,s_{1},s_{2})$ was chosen arbitrarily, $\tau$ is surjective.

(2) The second point is that $\omega_{\tau}$ must also surject onto the set $\mathcal{I}_{+}$ of all interventions on $C_{+}$. Any intervention $i_{+}\in\mathcal{I}_{+}$ can be identified with a vector $\mathbf{i}^{+}$ of values of variables in $\mathcal{V}_{+}$. By definition of $\mathcal{I}_{+}$, $i_{+}$ fixes at least the values of $X,Y,Z$. Consider the intervention $i_{N_{+}}$ that sets $D_{x}$, $D_{y}$, and $D_{z}$ to the one-hot representations of $X$, $Y$, and $Z$ for the values they were set. Furthermore, if $i_{+}$ sets $W$ to $w$ then $i_{N_{+}}$ sets $H_{1}$ to $w$ and if $i_{+}$ sets $S_{2}$ to $s_{2}$, then $i_{N_{+}}$ sets $H_{3}$ to $s_{2}$. It suffices to show that $\omega_{\tau}(i_{N_{+}})=i_{+}$. In other words, we need to show that $\tau(\mathsf{Proj}^{-1}(\mathbf{i}^{N_{+}}))=\mathsf{Proj}^{-1}(\mathbf{i}^{+})$.

First, we show for all $\mathbf{v}_{L}\in\mathsf{Proj}^{-1}(\mathbf{i}^{N_{+}})$ that $\tau(\mathbf{v}_{L})\in\mathsf{Proj}^{-1}(\mathbf{i}^{+})$. By construction of $i_{+}$, any variables fixed by $i_{N_{+}}$ will correspond (via $\tau$ component functions) to values of variables fixed by $i_{+}$, except for the variable $H_{3}$, which has no corresponding high level variable. We merely need to observe that for any values of variables not set by $i_{N_{+}}$, there exist corresponding values for the variables that are not set by $i_{+}$, such that the appropriate $\tau$ component functions map the former to the latter (with the exception of $H_{3}$, which has no corresponding high level variable). This is obvious from the definition of the components of $\tau$.

Second, we show for all $\mathbf{v}_{H}\in\mathsf{Proj}^{-1}(\mathbf{i}^{+})$ there is $\mathbf{v}_{L}\in\mathsf{Proj}^{-1}(\mathbf{i}^{N_{+}})$ such that $\tau(\mathbf{v}_{L})=\mathbf{v}_{H}$. Again, by construction of $i_{+}$, any variables fixed by $i_{+}$ will correspond (via $\tau$ component functions) to values of variables fixed by $i_{N_{+}}$. We merely need to observe that for any values of variables not set by $i_{+}$, there exist corresponding values for the variables not set by $i_{N_{+}}$, such that the appropriate $\tau$ component functions map the former to the latter, with $H_{3}$ taking on any value. This is obvious from the definition of the components of $\tau$. This concludes the argument that $\omega_{\tau}(i_{N_{+}})=i_{+}$.

(3) Finally, we need to show for each $i_{N_{+}}\in\emph{dom}(\omega_{\tau})$ that $\tau(i_{N_{+}}(C_{N_{+}}))=\omega_{\tau}(i_{N_{+}})(C_{+})$. The point here is that the two causal processes unfold in the same way, under any intervention.

Indeed, pick any $i_{N_{+}}$ and suppose that $i_{+}=\omega_{\tau}(i_{N_{+}})$. We know that $i_{+}$ fixes values $x,y,z$ of $X,Y,Z$, and likewise that $i_{N_{+}}$ fixes values $d_{x},d_{y},d_{z}$ of $D_{x},D_{y},D_{z}$ such that $\tau_{D_{j}}(x_{j})=d_{j}$ for $j\in\{1,2,3\}$. Any other variables fixed by $i_{+}$ from among $W,S_{1},S_{2}$ will likewise correspond (via $\tau_{W},\tau_{S_{1}},\tau_{S_{2}}$) to values of $H_{2},H_{1},O$ fixed by $i_{N_{+}}$. We merely need to observe that any variables that are not set by $i_{+}$ and $i_{N_{+}}$ will still correspond via the appropriate $\tau$-component, given their settings in $i_{+}(C_{+})$ and $i_{N_{+}}(C_{N_{+}})$. The mechanisms in $C_{N_{+}}$ were devised precisely to guarantee this.

Thus we have fulfilled the three requirements and we have shown that $C_{+}$ is an abstraction $C_{N_{+}}$.

The proof that $C_{\emph{NatLog}}$ is a constructive abstraction of $N_{\emph{NLI}}$ follows this same pattern.

We formally define the model $C_{\emph{NatLog}}=(\mathcal{V}_{\emph{NatLog}},\mathcal{R}_{\emph{NatLog}},\mathcal{F}_{\emph{NatLog}})$ as follows:

The set $\{\#,\equiv,\sqsubset,\sqsupset,|,\ \hat{}\ ,\smile\}$ contains the seven relations used in the natural logic of MacCartney and Manning . The set $\bf{N}_{\emph{Subj}}$ contains the subject nouns used to create MQNLI, $\bf{N}_{\emph{Obj}}$ the set of object nouns, $\bf{Adj}_{\emph{Subj}}$ the subject adjectives, $\bf{Adj}_{\emph{Obj}}$ the object adjectives, $\bf{V}$ the verbs, and Adv the adverbs. Additionally, $\mathcal{Q}$ is the set of joint projectivity signatures between every, some, not every, and no, $\mathcal{N}$ is the set of joint projectivity signatures between not and $\epsilon$, $\mathcal{A}$ is the set of joint projectivity signatures between intersective adjectives and adverbs and $\epsilon$. REL$(x,y)$ outputs the lexical relation between $x$ and $y$. Finally, COMP$(f,x_{1},x_{2},\dots,x_{n})=f(x_{1},x_{2},\dots,x_{n})$ and PROJ$(f,g)=P_{f/g}$ where $P_{f/g}$ is the joint projectivity signature between $f$ and $g$. See Geiger et al. for details about these sets and functions.

For some non-leaf node $N$ of the tree in Figure 2(a), we define $C_{\emph{NatLog}}^{\mathit{N}}$ to be the marginalization of $C_{\emph{NatLog}}$ where all variables are removed other than the input variables

along with the output variable $\text{QP}_{\emph{Subj}}$ and the intermediate variable $N$. For a definition of marginalization, see Bongers et al. .

In the main text, $N_{\emph{NLI}}$ could represent either our BERT model or our LSTM model. We will maintain this ambiguity, because while these two models are drastically different at the highest level of detail, for the sake of our analysis we can view them both as creating a grid of neural representations where each representation in the grid is caused by all representations in the previous row and causes all representations in the following row. We will now formally define the causal model $C_{N_{\emph{NLI}}}$.

For the LSTM model $n=2$ and for the BERT model $n=12$. $m$ is the number of tokens in a tokenized version of an MQNLI example.

For all $j$ and $k$ and where $d$ is the dimension of the vector representations.

where $\mathbf{NN}_{jk}$ is either the LSTM function or the BERT function that creates the neural representation at the $j$th row and $k$th column.

where $\mathbf{NN}_{O}$ is the neural network that makes a three class prediction using the final representation of the [CLS] token.

See Appendix B for details about these functions.

We will now formally prove that that $C_{\emph{NatLog}}^{N}$ is a constructive abstraction of $N_{\emph{NLI}}$ if the following holds for all $e,e^{\prime}\in\text{MQNLI}$, where the representation location $L$ is equivalent to the variable $R_{jk}$ for some $j$ and $k$. This would mean that every single one of our intervention experiments at this location are successful.

We define the mapping $\tau:\mathcal{R}_{N_{\emph{NLI}}}(\mathcal{V}_{N_{\emph{NLI}}})\to\mathcal{R}^{N}_{\emph{NatLog}}(\mathcal{V}_{\emph{NatLog}})$ as follows. We first partition the “low level” variables of $N_{\emph{NLI}}$ into partition cells:

where $R_{1j},R_{1(j+1)},\dots,R_{1(j+k)}$ are the token vectors associated with the input variable $X$. Some of our causal model’s input variables are tokenized into several tokens (see Appendix B for details).

For all $\forall X\in\mathcal{V}_{\emph{NatLog}}^{\emph{Input}}$, we set $\tau_{X}$ to be $T$. We additionally set $\tau_{N}$ to be $Q_{L}$ and $\tau_{\text{QP}_{\emph{Subj}}}$ to be $P$.

Let $\mathcal{I}_{\emph{NatLog}}$ be the set of all interventions on $C_{\emph{NatLog}}$ that intervene on (i.e., determine the values for) at least the elements of $\mathcal{V}_{\emph{NatLog}}^{\emph{Input}}$. Let $\mathcal{I}_{N_{\emph{NLI}}}$ be the set of interventions that is the domain of the partial function $\omega_{\tau}$. In other words, $\mathcal{I}_{N_{\emph{NLI}}}$ includes exactly the projections of $\mathcal{R}_{N_{\emph{NLI}}}(\mathcal{V}_{N_{\emph{NLI}}})$ that map via $\omega_{\tau}$ to some intervention on $C_{+}$. The fact that $P$, $Q_{L}$, an $T$ are all proper partial functions prevent $\mathcal{I}_{N_{\emph{NLI}}}$ from including all possible interventions on $C_{N_{\emph{NLI}}}$.

We now prove the three conditions that must hold for $(C_{\emph{NatLog}},\mathcal{I}_{\emph{NatLog}})$ to be a $\tau$-abstraction of $(C_{N_{\emph{NLI}}},\mathcal{I}_{N_{\emph{NLI}}})$.

(1) The first point is to show the map $\tau$ is surjective. So take an arbitrary element $(\vec{v}^{\emph{input}},n,q)\in\mathcal{R}_{\emph{NatLog}}(\mathcal{V}_{\emph{NatLog}})$. We specify an element of $\mathcal{R}_{N_{\emph{NLI}}}(\mathcal{V}_{N_{\emph{NLI}}})$ as follows:

where $r_{1j},r_{1(j+1)},\dots,r_{1(j+k)}$ are the token vectors corresponding to the input variable $v^{\emph{input}}$.

It’s then patent that $\tau(r_{11},\dots,r_{n1},r_{12},\dots r_{nm},o)=(\vec{v}^{\emph{input}},n,q)$. As $(\vec{v}^{\emph{input}},n,q)$ was chosen arbitrarily, we have shown $\tau$ is surjective.

(2) The second point is that $\omega_{\tau}$ must also be surjective onto the set $\mathcal{I}_{\emph{NatLog}}$ of interventions on $C_{\emph{NatLog}}$. Any intervention $i_{\emph{NatLog}}\in\mathcal{I}_{\emph{NatLog}}$ can be identified with with a vector $\mathbf{i}^{\emph{NatLog}}$ of values of variables in $\mathcal{V}_{\emph{NatLog}}$. By the definition of $\mathcal{I}_{\emph{NatLog}}$, $i_{\emph{NatLog}}$ fixes the values of the variables in $\text{V}^{input}$ and may also determine $N$ and/or $\text{QP}_{\emph{\emph{Subj}}}$. Consider the intervention $i_{N_{\emph{NLI}}}$ corresponding to $\mathbf{i^{N_{\emph{NLI}}}}=\tau^{-1}(\mathbf{i}^{\emph{NatLog}})$ as described in Section F.2. It suffices to show that $\omega_{\tau}(i_{N_{\emph{NLI}}})=i_{\emph{NatLog}}$. In other words, we need to show parts 1, 2, and 3 from the definition above.

Part 1 is clear, since by the definition of $\mathcal{I}_{\emph{NatLog}}$ we are guaranteed that $\mathbf{i}^{\emph{NatLog}}$ determines values for $\mathbf{V}^{\emph{input}}$, and hence $\mathbf{i}^{N_{\emph{NLI}}}$ fixes values for $R_{11},\dots,R_{1m}$ in the domains of $\tau_{\text{V}^{\emph{input}}}$ for $V\in\mathbf{V}^{\emph{input}}$. Then any intervention that intervenes only on the values of

Part 2 requires that for every $\mathbf{v}_{N_{\emph{NLI}}}\in\mathsf{Proj}^{-1}(\mathbf{i}^{N_{\emph{NLI}}})$, we have $\tau(\mathbf{v}_{N_{\emph{NLI}}})\in\mathsf{Proj}^{-1}(\mathbf{i^{\emph{NatLog}}})$. Because of how we defined $i_{\emph{NatLog}}$, any variables fixed by $i_{N_{\emph{NLI}}}$ will correspond (via $\tau$ component functions) to values of variables fixed by $i_{\emph{NatLog}}$, except for the variables $R_{jk}\not\in\mathbf{V}^{\emph{input}}\cup\{L\}$, which have no corresponding high level variables. We merely need to observe that, for any values for the variables that are not set by $i_{N_{\emph{NLI}}}$, there exists corresponding values for the variables that are not set by $i_{\emph{NatLog}}$ such that the appropriate $\tau$ component functions map the former to the latter, except for the variables $R_{jk}\not\in\mathbf{V}^{input}\cup\{L\}$, which, again, have no corresponding high level variables. This is plainly obvious from the definition of the components of $\tau$.

Part 3 requires that for any $\mathbf{v}_{\emph{NatLog}}\in\mathsf{Proj}^{-1}(\mathbf{i}^{\emph{NatLog}})$, there exists a $\mathbf{v}_{N_{\emph{NLI}}}\in\mathsf{Proj}^{-1}(\mathbf{i}^{N_{\emph{NLI}}})$ such that $\tau(\mathbf{v}_{N_{\emph{NLI}}})=\mathbf{v}_{\emph{NatLog}}$. Again, because of how we defined $i_{\emph{NatLog}}$, any variables fixed by $i_{\emph{NatLog}}$ will correspond (via $\tau$ component functions) to values of variables fixed by $i_{N_{\emph{NLI}}}$. We merely need to observe that for any values for the variables that are not set by $i_{\emph{NatLog}}$, there exists corresponding values for the variables that are not set by $i_{N_{\emph{NLI}}}$, such that the appropriate $\tau$ component functions map the former to the latter, with $R_{jk}\not\in\mathbf{V}^{\emph{input}}\cup\{L\}$ taking on any value. This is plainly obvious from the definition of the components of $\tau$.

Thus, we have shown that $\omega_{\tau}(i_{N_{\emph{NLI}}})=i_{\emph{NatLog}}$.

(3) Finally, we need to show for each $i_{N_{\emph{NLI}}}\in\emph{dom}(\omega_{\tau})$ that $\tau(i_{N_{\emph{NLI}}}(C_{N_{\emph{NLI}}}))=\omega_{\tau}(i_{N_{\emph{NLI}}})(C_{\emph{NatLog}})$. The point here is that the two causal processes unfold in the same way, under any intervention. Indeed, pick any $i_{N_{\emph{NLI}}}$ and suppose that $i_{\emph{NatLog}}=\omega_{\tau}(i_{N_{\emph{NLI}}})$. We know that $i_{\emph{NatLog}}$ fixes values for the variables in $\mathbf{V}^{input}$, and likewise that $i_{N_{\emph{NLI}}}$ fixes values for the variables $R_{11},\dots,R_{1m}$. Any other variables fixed by $i_{\emph{NatLog}}$ from among $N,\text{QP}_{\emph{Subj}}$ will likewise correspond (via the component functions of $\tau$) to values of $L$ and $O$. We merely need to observe that any variables that are not set by $i_{\emph{NatLog}}$ and $i_{N_{\emph{NLI}}}$ will still correspond via the appropriate $\tau$-component, given their settings in $i_{\emph{NatLog}}(C_{\emph{NatLog}})$ and $i_{N_{\emph{NLI}}}(C_{N_{\emph{NLI}}})$. The intervention experiments on $N_{\emph{NLI}}$ that we are assuming were successful were devised precisely to guarantee this.

We have thus fulfilled the three requirements and shown that $C_{\emph{NatLog}}$ is an abstraction of $C_{N_{\emph{NLI}}}$.