Linking artificial and human neural representations of language

Related work

Several papers have begun to explore the brain–machine link in language understanding, asking whether human brain activations can be matched with the activations of computational language models. Mitchell et al. (2008) first demonstrated that distributional word representations could be used to predict human brain activations, when subjects were presented with individual words in isolation. Huth et al. (2016) replicated and extended these results using distributed word representations, and Pereira et al. (2018) extended these results to sentence stimuli. Wehbe et al. (2014), Qian et al. (2016), Jain and Huth (2018), and Abnar et al. (2019) next introduced more complex word and sentence meaning representations, demonstrating that neural network language models could better account for brain activation by incorporating representations of longer-term linguistic context. Gauthier and Ivanova (2018) and Sun et al. (2019) further demonstrated that optimizing model representations for different objectives yielded substantial differences in brain decoding performance. This paper extends the neural network brain decoding paradigm both in breadth, studying a wide class of different task-optimal models, and in depth, exploring the particular representational contents of each model responsible for its brain decoding performance.

Methods

Figure 1 describes the high-level design of our experiments, which attempt to match human neuroimaging data with different candidate model representations of sentence inputs. Using a dataset of human brain activations recorded in response to complete sentences, we learn linear regression models which map from human brain activity to representations of the same sentences produced by different natural language understanding models.

To the extent that this linear decoding is successful, it can reveal parallel structure between brain and model representations. Consider a softmax neural network classifier model optimized for some task $T$, mapping input sentences $\mathbf{x}$ to class outputs $y$. We can factor this neural network classifier into the composition of two operations, a representational function $r(\mathbf{x})$, and an affine operator $A$:

Under this factorization, $r(\mathbf{x})$ must compute representations which are linearly separable with respect to the classes of the task $T$.

Research in cognitive neuroscience has shown that surprisingly many features of perceptual and cognitive states are likewise linearly separable from images of human brain activity, even at the coarse spatial and temporal resolution afforded by functional magnetic resonance imaging (fMRI; see e.g. Haxby et al., 2001; Kriegeskorte et al., 2006). However, the full power of linear decoding with fMRI remains unknown within language neuroscience and elsewhere. One possibility (1) is that the representational distinctions intrinsically required to describe language understanding behavior are linearly decodable from fMRI data. If this were the case, we could use performance in brain decoding to gauge the similarity between the mental representations underlying human language understanding and those deployed within artificial neural network models. Conversely (2), if the representations supporting language understanding in the brain are not linearly decodable from fMRI, we should be able to demonstrate this fact by showing specific ablations of sentence representation models do not degrade in brain decoding performance. Thus, the brain decoding framework offers possibilities both for (1) discriminating among NLU tasks as faithful characterizations of human language understanding, and for (2) understanding potential limitations of fMRI imaging and linear decoding methods. We explore both of these possibilities in this paper.

Section 2.1 describes the human neuroimaging data used as the source of this learned mapping. Section 2.2 next describes how we derive the target representations of sentence inputs from different natural language understanding models. Finally, Section 2.3 describes our method for deriving and evaluating mappings between the two representational spaces.

2 Sentence representation models

We will match the human brain activation data described above with a suite of different sentence representations. Our primary concern in this evaluation is to compare alternative tasks and the representational contents they demand, rather than comparing neural network architectures. For this reason, we draw sentence representations from a unified neural network architecture — the bidirectional Transformer model BERT (Devlin et al., 2018) — as we optimize it to perform different tasks.

The BERT model uses a series of multi-head attention operations to compute context-sensitive representations for each token in an input sentence. The model is pre-trained on two tasks: (1) a cloze language modeling task, where the model is given a complete sentence containing several masked words and asked to predict the identity of a particular masked word; and (2) a next-sentence prediction task, where the model is given two sentences and asked to predict whether the sentences are immediately adjacent in the original language modeling data.See Devlin et al. (2018); Vaswani et al. (2017) for further details on model architecture and training. For our purposes, this pre-training process produces a set of BERT parameters ${\Theta_{\text{LM}}}$ jointly optimized for these two objectives, consisting of word embeddings and attention mechanism parameters.

For an input token sequence $w_{1},\ldots,w_{T}$, the output of the BERT model is a corresponding sequence of contextualized representations of each token. We derive a single sentence representation vector by prepending a constant token $w_{0}=\texttt{[CLS]}$, and extracting its corresponding output vector at the final layer, following Devlin et al. (2018).We also repeated the experiments of this paper using sentence representations computed by uniformly averaging BERT’s contextualized token representations of each input sentence, and found similar qualitative results as will be presented below.

We use the code and pre-trained weights released by Devlin et al. (2018) to fine-tune the BERT model on a suite of natural language processing classification tasks, shown in Table 2.Details on the fine-tuning procedure are available in Devlin et al. (2018). This fine-tuning process jointly optimizes the pre-trained word embeddings and attention weights drawn from ${\Theta_{\text{LM}}}$, along with a task-specific classification model which accepts the sentence representations produced by the BERT model (the vector corresponding to the prepended [CLS] token) as input.

We fine-tune the pretrained BERT model on a set of popular shared NLU tasks, shown in Table 2, with fixed hyperparameters across tasks (available in Appendix B). Each fine-tuning operation is run for 250 iterations, before which all models show substantial improvements on the fine-tuning task. Figure 2 shows the learning curves for each of the models fine-tuned by this procedure.

In order to better understand why models might fail or succeed at brain decoding, we also produced several custom fine-tuning tasks. Each task was a modified form of the standard BERT cloze language modeling task, manipulated to strongly select for or against some particular aspect of linguistic representation.The training and test data for these tasks were generated from the Toronto Books Corpus (Zhu et al., 2015), a subset of the data used to train the original BERT model. Further details and examples for these tasks are given in Appendix C.

We first design two language modeling tasks to select against fine-grained syntactic representation of inputs. We randomly shuffle words from the corpus samples used for language modeling, to remove all first-order cues to syntactic structure. Our first custom task, LM-scrambled, deals with sentence inputs where words are shuffled within sentences; our second task, LM-scrambled-para, uses inputs where words are shuffled within their containing paragraphs in the corpus. This shuffling method removes first-order cues to constituency: for example, the fact that the table appears directly to the right of on suggests that they are members of a single constituent. It is still possible that the model can exploit second-order cues to structure. For example, if two nouns ranch and rancher appear in the same sentence as dressing, we can still guess that ranch is more likely to modify dressing.

By shuffling inputs in this way, we effectively turn the cloze task into a bag-of-words language modeling task: given a set of words from a sentence or a random draw of words from a paragraph, the model must predict a missing word. After optimizing models on these scrambled tasks, we design a probe to validate the effects of the task on the model’s syntactic representations. This probe is detailed in Section 3.2.2.

We next design a task, LM-pos, to select against fine-grained semantic representation of inputs. We do this by requiring a model to predict only the part of speech of a masked word, rather than the word itself. This manipulation removes pressure for the model to distinguish predictions between target words in the same syntactic class.

2.2 Word vector baseline

As a baseline comparison, we also include sentence representations computed from GloVe word vectors (Pennington et al., 2014). Unlike BERT’s word representations, these word vectors are insensitive to their surrounding sentential context. These word vectors have nevertheless successfully served as sentence meaning representations in prior studies (Pereira et al., 2018; Gauthier and Ivanova, 2018). We let $C_{\text{GloVe}}(w_{1},\ldots,w_{T})=\frac{1}{T}e(w_{t})$, where $e(w_{i})$ retrieves the GloVe embedding for word $w_{i}$.We use publicly available GloVe vectors computed on Common Crawl, available in the spaCy toolkit as en_vectors_web_lg.

3 Brain decoding

where $\beta$ is a regularization hyperparameter. For each subject’s collection of brain images and each target model representation, we train and evaluate the above regression model with nested 8-fold cross-validation (Cawley and Talbot, 2010). The regression models are evaluated under two metrics: mean squared error (MSE) in prediction of model activations, and average rank (AR):

These two metrics serve complementary roles: the MSE metric strictly evaluates the ability of human brain activations to exactly match the representational geometry of model activations, while the AR metric simply requires that the brain activations be able to support the relevant meaning contrasts between the 384 sentences tested.

Results

We first present the performance of all of the BERT models tested and the GloVe baseline in Figure 3. This figure makes apparent a number of surprising findings, which we validate by paired $t$-tests.Each sample in our statistical tests compares the brain decoding performance matching a subject’s brain image with model representations before and after fine-tuning on a particular task. See Figures 8(b) and 9(b) of Appendix A for further visualizations. Results are reported throughout with a significance level $\alpha=0.01$.

On average, fine-tuning on the standard NLU tasks yields increased error in brain decoding under both metrics relative to the BERT baseline (MSE, $t\approx 14.8,p<3\times 10^{-36}$; AR, $t\approx 17,p<3\times 10^{-42}$). This trend is significant for each model individually, except QQP (MSE, $t\approx-2.2,p>0.03$; AR, $t\approx 0.79,p>0.4$).

Fine-tuning on the LM-scrambled-para custom task yields decreased error in brain decoding under both metrics relative to the BERT baseline (MSE, $t\approx-33.6,p<3\times 10^{-45}$; AR, $t\approx-26.9,p<5\times 10^{-39}$) and GloVe vectors (MSE, $t\approx-23.4,p<4\times 10^{-35}$; AR, $t\approx-12.1,p<6\times 10^{-19})$.

Fine-tuning on both the control language-modeling task and the LM-pos custom task yields ambiguous results: decreased MSE and no significant change in AR relative to the BERT baseline (LM-pos: MSE, $t\approx-2.90,p<0.007$; AR, $t\approx 2.10,p>0.04$; LM: MSE, $t\approx-5.68,p<4\times 10^{-7}$; AR, $t\approx-2.36,p>0.02$).

When during training do these models diverge in brain decoding performance? We repeat our brain decoding evaluation on model snapshots taken every 5 steps during fine-tuning, and chart brain decoding performance over time for each model in Figure 4. We find that models rapidly diverge in brain decoding performance, but remain mostly stable after about 100 fine-tuning steps. This phase of rapid change in brain decoding performance is generally matched with a phase of rapid change in task performance (compare each line in Figure 4 with the learning curves in Figure 2).

2 Representational analysis

We next investigate the structure of the model representations, and find that differing fidelity of syntactic representation can explain some major qualitative differences between the models in brain decoding performance.

The language modeling fine-tuning runs (especially the two LM-scrambled tasks) are the only models which have reliable high correlations between one another.

Language modeling tasks yield representations which make similar sentence-sentence distance predictions between different runs on the same task, while the rest of the models are less coherent across runs (see matrix diagonal).

The scrambled LM tasks produce sentence representations which are reliably coherent across runs (Figure 5), and produce reliable improvements in brain decoding performance (Figure 3). What is it about this task which yields such reliable results? We attempt to answer this question in the following section.

2.2 Syntactic probe

Because the scrambled LM tasks were designed to remove all first-order cues to syntactic constituency from the input, we hypothesized that the models trained on this task were succeeding due to their resulting coarse syntactic representations. We tested this idea using the structural probe method of Hewitt and Manning (2019). This method measures the degree to which word representations can reproduce syntactic analyses of sentences. We used dependency-parsed sentences from the Universal Dependencies (UD) English Web Treebank corpus (Silveira et al., 2014) to evaluate the performance of each fine-tuned BERT model with a structural probe.

where $|w_{i}\leftrightarrow w_{j}|$ denotes the number of edges separating $w_{i}$ and $w_{j}$ in a dependency parse of the sentence.

We learn this parameter matrix $B$ for a set of training sentences randomly sampled from the UD corpus, and then apply the distance measure above to model representations for a set of held-out test sentences. For any sentence $w_{1},\ldots,w_{T}$, the measure induces a $T\times T$ pairwise distance matrix, where each entry $(i,j)$ predicts the distance (in grammatical dependencies) between words $i$ and $j$. By applying a minimum spanning tree algorithm to this matrix, we derive an (undirected) parse tree for the sentence which best matches the predictions of the distance measure. We measure the accuracy of the reconstructed tree by calculating its unlabeled attachment score (UAS) relative to ground-truth parses from the UD corpus.

We apply the probe described above to every fine-tuning run of each model, and to baseline GloVe representations. We expected the GloVe representations to perform worst, since they cannot encode any context-sensitive features of input words. The probe results are graphed over fine-tuning time in Figure 6, relative to a probe induced from the GloVe representations (dashed blue line). This analysis shows that the models optimized for LM-scrambled and LM-scrambled-para — the models which improve in brain decoding performance — progressively worsen under this syntactic probe measure during fine-tuning. Their probe performance remains well above the performance of the GloVe baseline, however.

Figure 7 shows a representative sample sentence with parses induced from the syntactic probes of LM-scrambled (after 250 fine-tuning steps) and the GloVe baseline. While both parses make many mistaken attachments (dashed arcs), the parse induced from LM-scrambled (blue arcs) makes better guesses about local attachment decisions than the parse from GloVe (red arcs), which seems to simply link identical and thematically related words. This is the case even though LM-scrambled is never able to exploit information about the relative positions of words during its training. Overall, this suggests that much (but not all) of the syntactic information initially represented in the baseline BERT model is discarded during training on the scrambled language modeling tasks. Surprisingly, this loss of syntactic information seems to yield improved performance in brain decoding.

Discussion

The brain decoding paradigm presented in this paper has led us to a set of scrambled language modeling tasks which best match the structure of brain activations among the models tested. Optimizing for these scrambled LM tasks produces a rapid but stable divergence in representational contents, yielding improvements in brain decoding performance (Figures 3 and 4) and reliably coherent predictions in pairwise sentence similarity (Figure 5). These changes are matched with a clear loss of syntactic information (Figure 6), though some minimal information about local grammatical dependencies is decodable from the model’s context-sensitive word representations (Figure 7).

We do not take these results to indicate that human neural representations of language do not encode syntactic information. Rather, we see several possible explanations for these results:

Functional magnetic resonance imaging — the brain imaging method used to collect the dataset studied in this paper — may be too temporally coarse to detect traces of the syntactic computations powering language understanding in the human brain.

This idea may conflict with several findings in the neuroscience of language. For example, Brennan et al. (2016) compared how language models with different granularities of syntactic representation map onto human brain activations during passive listening of English fiction. They derived word-level surprisal estimates from n-gram models (which have no explicit syntactic representation) and PCFG models (which explicitly represent syntactic constituency). In a stepwise regression analysis, they demonstrated that the surprisal estimates drawn from the PCFG model explain variance in fMRI measures of brain activation not already explained by estimates drawn from the n-gram model.

Pallier et al. (2011) examined a different hypothesis linking mental and neural representations of language. They presented subjects with strings of words which contain valid syntactic constituents of different lengths. They assumed that, since subjects will attempt to construct syntactic analyses of the word strings, the length of the possible syntactic constituents in any stimulus should have some correlate in subjects’ neural activations. They found a reliable relationship between the size of the available constituents in the input and region-specific brain activations as measured by fMRI.

Our results are compatible with the idea that specific syntactic features like those discussed above are still represented in the brain at the time scale of fMRI. Figures 6 and 7 demonstrate, in fact, that the LM-scrambled models still retain some syntactic information (or correlates thereof), in that they clearly outperform a baseline model in predicting the syntactic parses of sentences.

While these brain mapping studies have detected particular summary features of syntactic computation in the brain, these summary features do not constitute complete proposals of syntactic processing. In contrast, each of the models trained in this paper constitutes an independent candidate algorithmic description of sentence representation. These candidate descriptions can be probed (as in Section 3.2.2) to reveal exactly why brain decoding fails or succeeds in any case.

Our paradigm thus enables us to next ask: what specific syntactic features are responsible for the improved performance of the LM-scrambled models? By further probing the models and designing ablated datasets, we plan to narrow down the particular phenomena responsible for the findings presented here. These results should act as a source of finer-grained hypotheses about what sort of syntactic information is preserved at coarse temporal resolutions, and allow us to resolve the conflict between our results those of Pallier et al. (2011) and Brennan et al. (2016), among others.

Our linear linking hypothesis (presented in Section 2.3) that representations of syntactic structure are encoded entirely in the linear geometry of both neural networks and human brains. It is likely that some syntactic information — among other features of the input — are conserved in the fMRI signal, but not readable by a linear decoder. Future work should investigate how more complex transforms linking brain and machine can reveal parallel structure between the two systems.

The fMRI data used in this study (presented in Section 2.1) was collected as subjects read sentences and were asked to simply think about their meaning. This vague task specification may have led subjects to engage only superficially with the sentences. If this were the case, these shallow mental representations might present us with correspondingly shallow neural representations — just the sort of representations which might be optimal for the simple tasks such as LM-scrambled and LM-scrambled-para. Future work should integrate brain images derived from different behavioral tasks, and study which model–brain relationships are conserved across these behaviors. Such studies could illuminate the degree to which there are genuinely task-general language representations in the mind.

Our broader framework of analysis promises to reveal further insights about the parallel contents between artificial and human neural representations of language. In this spirit, we have released our complete analysis framework as open source code for the research community, available at http://bit.ly/nn-decoding.

Acknowledgments

JG gratefully acknowledges support from the Open Philanthropy Project, and RPL gratefully acknowledges support from a Newton Brain Science Research Seed Award and from the MIT–SenseTime Alliance on Artificial Intelligence.

References

Appendix A Supplemental figures

Figure 8(b) shows the change in brain decoding performance after fine-tuning grouped by subject, under both the mean squared error and rank metrics.

Figure 9(b) shows the change in brain decoding performance after fine-tuning grouped by model, under both the mean squared error and rank metrics.

Appendix B Hyperparameters

Appendix C Custom task information

For each language modeling task, we randomly sampled and concatenated documents from the Toronto Books Corpus (Kiros et al., 2015). Each language modeling dataset contained 1,000,000 training sentences and 100,000 development and test sentences. (We generated over-sized datasets in order to ensure that multiple runs of the same model would be highly unlikely to see similar samples of training data.)

For the part-of-speech task, we tagged each sentence using spaCy (Honnibal and Montani, 2017) and followed the same random masking procedure as in the typical cloze language modeling task. spaCy assigned 49 unique part-of-speech tags to the sentences, yielding a 49-way classification task.

For all tasks, we retained the secondary BERT objective requiring the model to predict whether two sentences are adjacent or not in a source document. (This objective did not differ from the standard setup for the part-of-speech task; for the scrambling task, the input sentences were independently randomly shuffled.)

Table 5(d) shows training examples from each of these custom tasks. Figure 10 shows learning curves and validation accuracy curves for the models trained on each task.