A Systematic Assessment of Syntactic Generalization in Neural Language Models

Introduction

A growing body of work advocates that assessment of neural language models should include both information-theoretic metrics, such as perplexity, as well as targeted linguistic evaluation. Benchmarks such as GLUE (Wang et al., 2019a, b) have demonstrated that neural language models trained on naturalistic corpora for next-word prediction learn representations that can yield remarkable performance on many semantic tasks. Targeted syntactic evaluations have shown that these models also implicitly capture many syntactic generalizations, ranging from subject–verb agreement to long-distance filler–gap dependencies (Linzen et al., 2016; Marvin and Linzen, 2018; Futrell et al., 2018; Wilcox et al., 2019b). This paper aims to bring targeted evaluations of syntactic performance to scale, complementing similar developments in semantic evaluation (McCoy et al., 2019).

Because the most widespread currency of evaluation for language models is perplexity—how well, on average, a model predicts a word in its context—a primary focus of this paper is the relationship between a model’s perplexity and its performance on targeted syntactic evaluations. As perplexity improves, can we expect more human-like syntactic generalization? How do training dataset size and model architecture jointly affect syntactic generalization? And what picture of models’ syntactic generalization emerges when evaluation is brought to scale, across dozens of controlled syntactic tests?

In this paper we offer initial answers to these questions, systematically assessing the syntactic generalization abilities of neural language models on 34 targeted test suites (33 adapted from previously published work, and 1 novel) covering a wide range of syntactic phenomena. Test suites are written using a standard format that allows for flexible predictions which more closely resemble those used in psycholinguistic studies, specifically allowing for predictions about interactions among multiple testing conditions. Performance on each test suite is reported as a Syntactic Generalization (SG) score. We group test suites into six syntactic circuits based on the linguistic representations needed to achieve high performance on each suite.

We train four classes of neural models and one baseline $n$-gram model on four datasets derived from a newswire corpus, consisting of 1, 5, 14, and 42 million tokens. While previous work has compared model architectures for a fixed dataset size (e.g. Wilcox et al., 2019b) and network sizes for a fixed architecture (e.g. van Schijndel et al., 2019), our controlled regime allows us to make an apples-to-apples comparison across model architectures on a range of sizes. In addition, we evaluate several off-the-shelf models which were trained on datasets ranging up to 2 billion tokens.

Our results address the three questions posed above: First, for the range of model architectures and dataset sizes tested, we find a substantial dissociation between perplexity and SG score. Second, we find a larger effect of model inductive bias than training data size on SG score, a result that accords with van Schijndel et al. (2019). Models afforded explicit structural supervision during training outperform other models: One structurally supervised model is able to achieve the same SG scores as a purely sequence-based model trained on $\sim$100 times the number of tokens. Furthermore, several Transformer models achieve the same SG score as a Transformer trained on $\sim$200 times the amount of data. Third, we find that architectures have different relative advantages across types of syntactic tests, suggesting that the tested syntactic phenomena tap into different underlying processing capacities in the models.

Background

Standard language models are trained to predict the next token given a context of previous tokens. Language models are typically assessed by their perplexity, the inverse geometric mean of the joint probability of words $w_{1},\dots,w_{N}$ in a held-out test corpus $C$:

Models with improved perplexity have also been shown to better match various human behavioral measures, such as gaze duration during reading (Frank and Bod, 2011; Fossum and Levy, 2012; Goodkind and Bicknell, 2018; Wilcox et al., 2020). However, a broad-coverage metric such as perplexity may not be ideal for assessing human-like syntactic knowledge for a variety of reasons. In principle, a sentence can appear with vanishingly low probability but still be grammatically well-formed, such as Colorless green ideas sleep furiously (Chomsky, 1957). While perplexity remains an integral part of language model evaluation, fine-grained linguistic assessment can provide both more challenging and more interpretable tests to evaluate neural models.

2 Targeted tests for syntactic generalization

Alternatively, a language model can be evaluated on its ability to make human-like generalizations for specific syntactic phenomena (Linzen et al., 2016; Lau et al., 2017; Gulordava et al., 2018). The targeted syntactic evaluation paradigm (Marvin and Linzen, 2018; Futrell et al., 2019) incorporates methods from psycholinguistic experiments, designing sentences which hold most lexical and syntactic features of each sentence constant while minimally varying features that determine grammaticality or surprise characteristics of the sentence. For example, given the two strings The keys to the cabinet are on the table and *The keys to the cabinet is on the table, a model that has learned the proper subject–verb number agreement rules for English should assign a higher probability to the grammatical plural verb in the first sentence than to the ungrammatical singular verb in the second (Linzen et al., 2016).

Although some targeted syntactic evaluations, such as the example discussed above, involve simple comparisons of conditional probabilities of a word in its context, other evaluations are more complex. We can demonstrate this with an evaluation of models’ “garden-pathing” behavior (Futrell et al., 2019). For example, the sentence The child kicked in the chaos found her way back home yields processing disruption for humans at the word found. This is because, up to right before that word, the part-of-speech ambiguous kicked is preferentially interpreted as the main verb of the sentence, whereas it turns out to be a passive participle in a reduced relative clause modifying child. This garden-path disambiguation effect is ameliorated by replacing kicked with forgotten, which is not part-of-speech ambiguous (B below; Trueswell et al., 1994) or by using an unreduced relative clause (C below; Ferreira and Clifton, 1986). In probabilistic language models, these garden-path disambiguation effects are well captured by word negative log probabilities, or surprisals (Hale, 2001): $S(w|C)=-\log_{2}p(w|C)$, which are independently well-established to predict human incremental processing difficulty over several orders of magnitude in word probability (Smith and Levy, 2013). A targeted syntactic evaluation for garden-pathing is provided by comparing surprisals at the disambiguating word found in the set of four examples below (Futrell et al., 2019):

The child who was kicked in the chaos found …

The child who was forgotten in the chaos found …

Successful human-like generalization involves three criteria: (i) found should be less surprising (i.e., more probable) in B than A; (ii) found should be more probable in C than A; (iii) the C–D surprisal difference should be smaller than the A–B surprisal difference—a $2\times 2$ interaction effect on surprisal—because the syntactic disambiguation effect of not reducing the relative clause was achieved by using a part-of-speech unambiguous verb.

We will use these controlled tests to help us describe and test for human-like syntactic knowledge in language models.

3 Related work

The testing paradigm presented here differs in several crucial ways from recent, related syntactic assessments and provides complementary insights. Unlike Warstadt et al. (2019a), our approach does not involve fine-tuning, but rather assesses what syntactic knowledge is induced from the language modeling objective alone. The most closely related work is the Benchmark of Linguistic Minimal Pairs (Warstadt et al., 2020), which is a challenge set of automatically-generated sentence pairs also designed to test language models on a large set of syntactic phenomena. Our approach differs in important ways: we compare critical sentence regions instead of full-sentence probabilities, and employ a $2\times 2$ paradigm with a strict, multi-fold success criterion inspired by psycholinguistics methodology. This allows us to factor out as many confounds as possible, such as the lexical frequency of individual tokens and low-level $n$-gram statistics.

Methods

We designed a controlled paradigm for systematically testing the relationship between two design choices — model class and dataset size — and two performance metrics — perplexity and syntactic generalization capacity. Section 3.1 describes the test suites collected for our evaluation, and Sections 3.2 and 3.3 describe the datasets and model classes investigated.

We assemble a large number of test suites inspired by the methodology of experimental sentence-processing and psycholinguistic research. Each test suite contains a number of items (typically between 20 and 30), and each item appears in several conditions: across conditions, a given item will differ only according to a controlled manipulation designed to target a particular feature of grammatical knowledge. Each test suite contains at least one prediction, which specifies inequalities between surprisal values at pairs of regions/conditions that should hold if a model has learned the appropriate syntactic generalization.

We expect language models which have learned the appropriate syntactic generalizations from their input to satisfy these inequalities without further fine-tuning. We compute accuracy on a test suite as the proportion of items for which the model’s behavior conforms to the prediction. Most of our test suites involve 2$\times$2 designs and a success criterion consisting of a conjunction of inequalities across conditions, as in the garden-pathing example described in Section 2.2.The exception is Center Embedding, which features a 2-condition design with a single-inequality criterion. Random baseline accuracy varies by test suite and is $\sim$25% overall. Most of these test suites and criteria are designed so that $n$-gram models cannot perform above chance for $n=5$ (sometimes greater).

In order to assess the coverage of our test suites, we manually inspected the phenomena covered in Carnie (2012), a standard introductory syntax textbook. Of the 47 empirical phenomena reviewed in the summary sections at the end of each chapter, our tests target 16 ($\sim$34%). These are evenly distributed across the whole range of subject matter, with tests targeting phenomena in 11 of the 15 chapters ($\sim$73%).For more details on this analysis, see Appendix 0.A.

Five test suites include paired modifier versions, where extra syntactically irrelevant (but semantically plausible) content, such as a prepositional phrase or relative clause, is inserted before the critical region being measured. We use these paired test suites to evaluate models’ stability to intervening content within individual syntactic tests.

The test suites are divided into 6 syntactic circuits, based on the type of algorithm required to successfully process each construction. We give a brief overview of each circuit below.A full overview of our test suites is given in Appendix 0.B.

Agreement is a constraint on the feature values of two co-varying tokens. For example, the number feature of a verb must agree with the number feature of its upstream subject. We include 3 Subject-Verb Number Agreement suites from Marvin and Linzen (2018).

Licensing occurs when a particular token must exist within the scope of an upstream licensor token. Scope is determined by the tree-structural properties of the sentence. Test suites include Negative Polarity Item Licensing (NPI) (4 suites) and Reflexive Pronoun Licensing (6 suites), both from Marvin and Linzen (2018).

Garden-Path Effects are well-studied syntactic phenomena that result from tree-structural ambiguities that give rise to locally-coherent but globally implausible syntactic parses. Garden-path test suites include Main Verb / Reduced Relative Clause (MVRR) (2 suites) and NP/Z Garden-paths (NPZ) (4 suites), both from Futrell et al. (2018).

Gross Syntactic Expectation is a processor’s expectation for large syntactic chunks such as verb phrases or sentences, and are often set up by subordinating conjunctions such as while, although and despite. Our tests for gross syntactic expectation include Subordination (4 suites) from Futrell et al. (2018).

Center Embedding sentences are sentences recursively nested within each other. Subject and verbs must match in a first-in-last-out order, meaning models must approximate a stack-like data-structure in order to successfully process them. Our 2 suites of Center Embedding sentences come from the items presented in Wilcox et al. (2019a).

Long-Distance Dependencies are co-variations between two tokens that span long distances in tree depth. Test suites include Filler-Gap Dependencies (FGD) (6 suites) from Wilcox et al. (2018) and Wilcox et al. (2019b), and 2 novel Cleft suites, described in detail below.

We introduce one novel test suite that assesses models’ ability to process pseudo-cleft constructions, which are used to put a particular syntactic constituent into focus via passive transformation. Consider Example (1):

What he did after coming in from the rain was eat a hot meal. [DO/VP]

*What he devoured after coming in from the rain was eat a hot meal. [LEX/VP]

*What he did after coming in from the rain was a hot meal. [DO/NP]

What he devoured after coming in from the rain was a hot meal. [LEX/NP]

When this constituent is a verb, it must be replaced in the wh-clause that heads the sentence with the DO verb, as in (1a), below. However, when it is a noun, the lexical verb for which it serves as an object must be preserved, as in (1d). If models have properly learned the pseudo-cleft construction, then DO verbs should set up expectations for VPs (the region in bold should have a lower surprisal in (1a) than in (1b)) and lexicalized verbs should set up expectations for NPs (the region in bold should have a lower surprisal in (1d) than in (1c)).

2 Model training data

We train and evaluate models on English newswire corpora of four different sizes, obtained by randomly sampling sections from the Brown Laboratory for Linguistic Information Processing 1987-89 Corpus Release 1 (BLLIP; Charniak et al., 2000). The corpora are sampled such that the training set of each corpus is a proper subset of each larger corpus. We call these four corpora BLLIP-xs (40K sentences, 1M tokens); BLLIP-sm (200K sentences, 5M tokens); BLLIP-md (600K sentences, 14M tokens); and BLLIP-lg (2M sentences, 42M tokens). Table 1 summarizes statistics of the training set for each corpus.

To ensure consistency in perplexity evaluation across datasets, we report perplexity scores achieved by the models on a shared held-out test set. We additionally use a shared held-out validation for tuning and early stopping.

We use the NLTK implementation of the Penn Treebank tokenizer to process all datasets (Bird and Loper, 2004; Marcus et al., 1993).

For each corpus, we designate a token as OOV if the token appears fewer than two times in the training set. Our larger training datasets thus contain larger vocabularies than our smaller training datasets. This allows larger-training-set models to learn richer word-specific information, but may also harm perplexity evaluation because they have vocabulary items that are guaranteed to not appear in the BLLIP-xs test set. This means that perplexity scores across training dataset sizes will not be strictly comparable: if a larger-training-set model does better than a smaller-training-set model, we can be confident that it has meaningfully lower perplexity, but the reverse is not necessarily the case. The exception to the above is GPT-2, which uses sub-words from byte-pair encoding and has no OOVs (see also Footnote 6).

We follow the convention used by the Berkeley parser (Petrov and Klein, 2007), which maps OOVs to UNK classes which preserve fine-grained information such as orthographic case distinctions and morphological suffixes (e.g. UNK-ed, UNK-ly). Before training, we verified that the UNK classes in the test and validation sets were all present in the training set.

3 Model classes

In order to study the effects of model inductive bias and dataset size, we trained a fleet of models with varying inductive biases on each corpus. Because many of our test suites exploit ambiguities that arise from incremental processing, we restrict evaluation to left-to-right language models; future work could involve evaluation of bidirectional models (Devlin et al., 2018; Yang et al., 2019) on an appropriate subset of our test suites, and/or adaptation of our suites for use with bidirectional models (Goldberg, 2019). Training ran until convergence of perplexity on a held-out validation set. Wherever possible, we trained multiple seeds of each model class and corpus size. We use the model sizes and training hyperparameters reported in the papers introducing each model (Table 2).Due to computational constraints, we performed only minimal tuning past these recommended hyperparameters. The full parameter counts and perplexity scores for each model $\times$ corpus combination are given in Tables 3 and 4, respectively.

Our baseline neural model is a vanilla long short-term memory network (LSTM; Hochreiter and Schmidhuber, 1997) based on the boilerplate PyTorch implementation (Paszke et al., 2017).

We consider the Ordered-Neurons LSTM architecture (ON-LSTM; Shen et al., 2019), which encodes an explicit bias towards modeling hierarchical structure.

Recurrent neural network grammars (RNNG; Dyer et al., 2016) model the joint probability of a sequence of words and its syntactic structure. RNNG requires labeled trees that contain complete constituency parses, which we produce for BLLIP sentences with an off-the-shelf constituency parser (Kitaev and Klein, 2018).While the BLLIP corpus already contains Treebank-style parses, we strip the terminals and re-parse in order to obtain more accurate, up-to-date syntactic parses. To compute surprisals from RNNG, we use word-synchronous beam search (Stern et al., 2017) to approximate the conditional probability of the current word given the context.

Transformer models (Vaswani et al., 2017) have recently gained popularity in language processing tasks. We use GPT-2 (Radford et al., 2019) as a representative Transformer model and train it from scratch on our BLLIP corpora.Our GPT-2 code is based on nshepperd/gpt-2. The model vocabulary consists of byte-pair encoded sub-words extracted from the GPT-2 pre-trained model, not from the BLLIP training corpora. To calculate GPT-2 perplexities, we divide the sum of all sub-word conditional log-probabilities by the total number of words in the corpus.

As a baseline, we consider a 5-gram model with modified Kneser-Ney smoothing.

4 Off-the-shelf models

We also test five off-the-shelf models: GRNN, trained on 90M tokens from Wikipedia (Gulordava et al., 2018); JRNN, trained on 800M tokens from the 1 Billion Word Benchmark (Jozefowicz et al., 2016); Transformer-XL, trained on 103M tokens from WikiText-103 (Dai et al., 2019); and the pre-trained GPT-2 and GPT-2-XL, trained on 40GB of web text (Radford et al., 2019). These models are orders of magnitude larger than our controlled ones in parameter count and/or training set size.

Results

Figure 1 shows the average accuracy of all models on the complete set of SG test suites. Asterisks denote off-the-shelf models. All neural models achieve a SG score significantly greater than a random baseline (dashed line). However, the range within neural models is notable, with the best-performing model (GPT-2-XL) scoring over twice as high as the worst-performing model (LSTM). Also notable are the controlled GPT-2 and RNNG models, which achieve comparable performance to Transformer-XL and JRNN, despite being trained on significantly smaller data sizes.

We now return to the three major issues presented in Section 1. In 4.1 we present evidence that SG score is dissociated from perplexity. In 4.2 we argue that model architecture accounts for larger gains in SG score than amount of training data. And in 4.3 we show that this cross-architecture difference is due largely to variance on a handful of key test suites.

Figure 2 shows the relationship between SG score and perplexity on the BLLIP test set across models and training set sizes. As expected, $n$-gram models never rise appreciably above chance in SG score. Among neural models, GPT-2 achieves both the worst (BLLIP-xs and BLLIP-sm) and best (BLLIP-md and BLLIP-lg) performance; the impressive performance of these latter models comes with the caveat that the sub-words come from the pre-trained GPT-2 model, tacitly importing information from a larger training dataset (see further discussion in Section 4.5). For the remaining neural models, there is no simple relationship between perplexity and SG score, especially once training dataset size is controlled for (comparing points in Figure 2 of the same color). For example, there is a remarkable amount of variance in the SG score of models trained on BLLIP-lg not explained by perplexity. This suggests that targeted syntactic evaluation can reveal information that may be orthogonal to perplexity.

2 Inductive bias and data scale

In order to decouple the effects of model class and data scale from test suite difficulty, we represent a particular trained model’s performance on each test suite as a delta relative to the average performance of all models on this test suite. Unless noted otherwise, the remainder of the figures in this section plot a score delta, aggregating these deltas within model classes or corpus types.

Figure 3 tracks the influence of model class and data scale across the model types tested in our experiments, with SG score deltas on the y-axis. The left-hand panel shows the difference in SG score by model class. We find that model class clearly influences SG score: for example, the error bars (bootstrapped 95% confidence intervals of the mean) for RNNG and LSTM do not overlap. The right-hand panel shows the difference in SG score delta by training dataset, and shows a much more minor increase in mean SG score as training data increases.

We tested the influence of these factors quantitatively using a linear mixed-effects regression model, predicting suite-level performance as a feature of model architecture and training dataset size (represented as log-number of words). Both features made statistically significant contributions to SG score (both $p<0.001$). However, predictor ablation indicates that architecture affects regression model fit more (AIC=–581 when dataset size is ablated; AIC=–574 when architecture is ablated).$n$-grams and/or GPT-2 could arguably be expected to have qualitatively different sensitivity to training dataset size (the latter due to byte-pair encoding), so we repeated the analyses here and in Section 4.3 excluding both architectures individually as well as simultaneously. In all cases the same qualitative patterns described in the main text hold.

Beyond the above analysis, our GPT-2 results offer another striking example of the influence of model architecture relative to data scale. Figure 2 shows that our controlled BLLIP-md and BLLIP-lg GPT-2 models achieve roughly the same SG score as the pre-trained GPT-2 model, despite being trained on less than 1% of the data used by the pre-trained model. This suggests diminishing returns to training data scale for syntactic generalization performance.

3 Circuit-level effects on SG score

Figure 4 shows the breakdown at the circuit level by model architecture (left) and training dataset size (right). The right panel demonstrates little effect of dataset size on SG score delta within most circuits, except for Agreement, on which the models trained on our smallest dataset fare poorly. In the left panel we find substantial between-circuit differences across architectures. Linear mixed-effects analyses support this finding: interactions with circuit are significant for both training dataset size and model architecture, but stronger for the latter (AIC=–654 and AIC=–623 when size and architecture are respectively ablated).

While model inductive biases separate clearly in performance on some circuits, they have little effect on performance on Licensing. This minimally suggests that Licensing taps into a distinct syntactic process within language models. One potential explanation for this is that the interactions tested by Licensing involve tracking two co-varying tokens where the downstream token is optional (see e.g. Hu et al., 2020).

We show the circuit-level breakdown of absolute SG scores for all models (including off-the-shelf) in Figure 5. In general, the models that obtain high SG scores on average (as in Figure 1) also perform well across circuits: pre-trained GPT-2 and GPT-2-XL outperform all other models on each circuit, including Licensing, on which JRNN, GRNN, and most of our custom-trained models perform particularly poorly. Again, we highlight the impressive performance of RNNG: it achieves comparable average performance to GRNN on all circuits, despite being trained on a fraction of the data size.

4 Stability to modifiers

We separately investigate the degree to which models’ syntactic generalizations are robustly stored in memory. For five test suites (Center Embedding, Cleft, MVRR, NPZ-Ambiguous, NPZ-Object), we designed minimally edited versions where syntactically irrelevant intervening content was inserted before the critical region. An ideal model should robustly represent syntactic features of its input across these modifier insertions.

In Figure 6 we plot models’ average scores on these five test suites (dark bars) and their minimally edited versions (light bars), evaluating how robust each model is to intervening content. Among models in our controlled experiments, we see that model class clearly influences the degree to which predictions are affected by intervening content (compare e.g. the stability of RNNG to that of ON-LSTM). Some off-the-shelf models, such as GPT-2-XL, perform near ceiling on the original five test suites and are not affected at all by intervening content.

5 Effects of model pre-processing

The GPT-2 models trained and evaluated in this paper use a sub-word vocabulary learned by byte-pair encoding (BPE; Sennrich et al., 2016) to represent their inputs, while all other models represent and compute over word-level inputs. This byte-pair encoding was taken from the pre-trained GPT-2 model trained on a much larger corpus. The results reported for these models thus conflate a choice of model class (a deep Transformer architecture) and preprocessing standard (sub-word tokenization computed on a larger corpus). Some preliminary work suggests that sub-word tokenization is indeed responsible for much of the larger GPT-2 models’ success: we find that GPT-2 models trained on word-level representations of BLLIP-lg and BLLIP-md achieve good perplexity measures, but degrade sharply in SG score.

Peculiarities of the GPT-2 training regime may be responsible for its particularly bad performance on the smaller corpora. Its sub-word vocabulary was held constant across training corpora, meaning that the model vocabulary size also remained constant across corpora, unlike the other models tested. The poor performance of GPT-2 models trained on smaller corpora may thus be due to overparameterization, and not due to fundamental problems with the model architecture at small data scales. We leave a thorough investigation of the role of sub-word tokenization to future work.

Discussion

This work addresses multiple open questions about syntactic evaluations and their relationship to other language model assessments. Our results dissociate model perplexity and performance in syntactic generalization tests, suggesting that the two metrics capture complementary features of language model knowledge. In a controlled evaluation of different model classes and datasets, we find model architecture plays a more important role than training data scale in yielding correct syntactic generalizations. Our circuit-level analysis reveals consistent failure on Licensing but inconsistent behavior on other circuits, suggesting that different syntactic circuits make use of different underlying processing capacities. In addition to the insight these results provide about neural NLP systems, they also bear on questions central to cognitive science and linguistics, putting lower bounds on what syntactic knowledge can be acquired from string input alone.

Targeted syntactic evaluation is just one in a series of complementary methods being developed to assess the learning outcomes of neural language processing models. Other methods include classifying sentences as grammatical or ungrammatical (Warstadt et al., 2019b), decoding syntactic features from a model’s internal state (Belinkov et al., 2017; Giulianelli et al., 2018), or transfer learning to a strictly syntactic task such as parsing or POS tagging (Hewitt and Manning, 2019). As each task brings an explicit set of assumptions, complementary assessment methods can collectively provide greater insight into models’ learning outcomes.

Although this paper, together with Warstadt et al. (2020), report what is to our knowledge the largest-scale targeted syntactic evaluations to date, we emphasize that they are only first steps toward a comprehensive understanding of the syntactic capabilities of contemporary language models. This understanding will be further advanced by new targeted-evaluation test suites covering a still wider variety of syntactic phenomena, additional trained models with more varied hyperparameters and randomization seeds, and new architectural innovations. Humans develop extraordinary grammatical capabilities through exposure to natural linguistic input. It remains to be seen to just what extent contemporary artificial systems do the same.

Acknowledgments

The authors would like to thank the anonymous reviewers and Samuel R. Bowman for their feedback, Miguel Ballesteros for advice and technical guidance, and Tristan Thrush for technical assistance. J.H. is supported by the NIH under award number T32NS105587 and an NSF Graduate Research Fellowship. J.G. is supported by an Open Philanthropy AI Fellowship. R.P.L. gratefully acknowledges support from the MIT-IBM Watson AI Lab, a Google Faculty Research Award, and a Newton Brain Science Award.

References

Appendix 0.A Syntactic coverage of test suites

In order to assess the coverage of our syntactic tests, we manually inspected the “Ideas, Rules and Constraints introduced in this Chapter” section for each chapter in Carnie (2012), a standard introductory syntax textbook. We included entries from these sections which are theory-neutral and refer to observable linguistic data. For example, we do not include affix lowering (Chapter 7) or theta criterion (Chapter 8) because these phenomena presuppose a commitment to one particular syntactic analysis.

We found that our tests covered 16 of the 47 phenomena presented ($\sim$34%). Of the 15 chapters surveyed, our tests assessed phenomena in 11 ($\sim$73%). We did not assess coverage from the last two chapters of the book, which explore alternative syntactic formalisms. The outcome of our manual inspection is given in Table 5.

A ✓indicates that some aspect of that phenomena was tested in one or more of our suites. ✓does not necessarily mean that the test suite was designed explicitly for the purpose of testing that phenomena, but merely that the phenomena was implicated in model success. For example, we place a ✓next to Parts of Speech because differentiation between verbs and nouns is necessary for models to succeed in the Cleft Structure tests.

Appendix 0.B Description of test suites

In this work we have assembled a large number of test suites inspired by the methodology of experimental sentence-processing and psycholinguistic research. Each test suite contains a number of items, and each item appears in several conditions: across conditions, a given item will differ only according to a controlled manipulation designed to target a particular feature of grammatical knowledge. For each suite we define a success criterion, which stipulates inequalities among conditional probabilities of sentence substrings.

In the main paper, a model’s accuracy for a test suite is computed as the percentage of the test suite’s items for which it satisfies the criterion. In this appendix, we briefly describe each test suite and the criterion used to determine whether a given model succeeds on each item of the test suite.

Following and building on linguistic traditions, we annotate examples as follows. Examples marked with a * violate a well-established grammatical constraint, and are ungrammatical. Examples marked with ? or ?? are not necessarily ungrammatical, but are marginal: for example, they may require an unusual interpretation of a word in order for the sentence to be grammatical. (More ?’s is roughly intended to indicate more severe marginality). Examples marked with ! are not ungrammatical, but induce severe processing difficulty that is measurable in real-time human sentence processing. For all test suites, we include references to established literature on the relevant grammatical and/or sentence-processing phenomena.

B.1.2 Success criteria

We provide chance accuracy on the assumption that the order of probabilities among conditions for a given item is random. In some cases, exactly determining chance accuracy may require further assumptions about the distribution of these probabilities; in this case we provide an upper bound on chance accuracy.

B.2 Center embedding

Center embedding, the ability to embed a phrase in the middle of another phrase of the same type, is a hallmark feature of natural language syntax. Center-embedding creates nested syntactic dependencies, which could pose a challenge for some language models. To succeed in generating expectations about how sentences will continue in the context of multiple center embedding, a model must maintain a representation not only of what words appear in the preceding context but also of the order of those words, and must predict that upcoming words occur in the appropriate order. In this test suite we use verb transitivity and subject–verb plausibility to test model capabilities in this respect. For example, (A) below is a correct center-embedding, but (B) is not:

The painting${}_{\text{N}_{1}}$ that the artist${}_{\text{N}_{2}}$ painted${}_{\text{V}_{2}}$ deteriorated${}_{\text{V}_{1}}$. [correct]

??The painting${}_{\text{N}_{1}}$ that the artist${}_{\text{N}_{2}}$ deteriorated${}_{\text{V}_{1}}$ painted${}_{\text{V}_{2}}$. [incorrect]

Here, Ni and $V_{i}$ correspond to matched subject–verb pairs.

In the with-modifier version of the test suite, we postmodify N2 with a relative clause to increase the linear distance over which the nested dependencies must be tracked, potentially leading to a harder test suite:

The painting${}_{\text{N}_{1}}$ that the artist${}_{\text{N}_{2}}$ who lived long ago painted${}_{\text{V}_{2}}$ deteriorated${}_{\text{V}_{1}}$. [correct]

#The painting${}_{\text{N}_{1}}$ that the artist${}_{\text{N}_{2}}$ who lived long ago deteriorated${}_{\text{V}_{1}}$ painted${}_{\text{V}_{2}}$. [incorrect]

The probability of the verb sequence in the correct variant should be higher than the probability of the verb sequence in the incorrect variant:

In full form, this criterion for the example item in the no-modifier version of this test suite would be:

Chance performance on these center-embedding test suites would be 50%.

Miller and Chomsky (1963);Wilcox et al. (2019a)

B.3 Pseudo-clefting

The pseudo-cleft construction involves (i) an extraction of a targeted constituent from a sentence and (ii) a constituent that provides the semantic contents of the targeted constituent and must match it in syntactic category, where (i) and (ii) are linked by the copula. The pseudo-cleft construction can target both NPs and VPs; in the latter case, the VP of the free relative becomes an inflected form of do. This means that a free relative subject plus the copula can set up a requirement for the syntactic category that comes next. If the free relative clause has a do VP without a direct object, then the main-clause postcopular predicate can be a VP ((A) below). Otherwise, the postcopular predicate must be an NP ((C) below):

What the worker did was $\overbrace{\text{board the plane}}^{\text{VP}}$.

?What the worker did was $\overbrace{\text{the plane}}^{\text{NP}}$.

What the worker repaired was $\overbrace{\text{the plane}}^{\text{NP}}$.

*What the worker repaired was $\overbrace{\text{board the plane}}^{\text{VP}}$.

The postcopular predicate should be more surprising when its syntactic category mismatches the cleft, averaging across VP and NP postcopular predicates:

Chance is 50%. A more stringent criterion would be to apply this requirement separately for each of NP and VP postcopular predicates:

However, it is often possible to use an NP postcopular predicate with a do cleft through semantic coercion (e.g., in (B) “did” can be interpreted as “fixed” or “was responsible for”), so we felt that this latter criterion might be too stringent.

B.4 Filler–gap dependencies

Consider the following sentence, in which all arguments and adjuncts appear “in situ” (in the syntactic position at which they are normally interpreted semantically):

I know that our uncle grabbed the food in front of the guests at the holiday party.

A filler–gap dependency can be created by extracting any of a number of elements from the subordinate clause, including our uncle (subject extraction), the food (object extraction) or the guests (extraction from a prepositional phrase). These possibilities serve as the basis for several test suites on filler–gap dependencies.

Ross (1967); Crain and Fodor (1985); Stowe (1986); Wilcox et al. (2018); Chowdhury and Zamparelli (2018); Chaves (2020)

B.4.1 Subject extractions

I know that $\overbrace{\text{our uncle}}^{\alpha}$ grabbed the food in front of the guests at the holiday party. [that, no gap]

*I know who $\overbrace{\text{our uncle}}^{\alpha}$ grabbed the food in front of the guests at the holiday party. [wh, no gap]

*I know that $\overbrace{\text{grabbed}}^{\beta}$ the food in front of the guests at the holiday party. [that, gap]

I know who $\overbrace{\text{grabbed}}^{\beta}$ the food in front of the guests at the holiday party. [wh, gap]

We require that a model successfully pass a two-part criterion for each item: the wh-filler should make the unextracted subject $\alpha$ more surprising in the no-gap conditions and should make the post-gap material $\beta$ less surprising in the gap conditions:

B.4.2 Object extractions

The logic of this test suite is the same as that for subject extraction above. Note that we use obligatorily transitive embedded verbs, so that omitting a direct object should be highly surprising when there is no filler, as in (C).

I know that our uncle grabbed $\overbrace{\text{the food}}^{\alpha}$ in front of the guests at the holiday party. [that, no gap]

*I know what our uncle grabbed $\overbrace{\text{the food}}^{\alpha}$ in front of the guests at the holiday party. [wh, no gap]

??I know that our uncle grabbed $\overbrace{\text{in front of}}^{\beta}$ the guests at the holiday party. [that, gap]

I know what our uncle grabbed $\overbrace{\text{in front of}}^{\beta}$ in front of the guests at the holiday party. [wh, gap]

B.4.3 Extraction from prepositional phrases

The logic of this test suite is the same as that for subject and object extractions above.

I know that our uncle grabbed the food in front of $\overbrace{\text{the guests}}^{\alpha}$ at the holiday party. [that, no gap]

*I know who our uncle grabbed the food in front of $\overbrace{\text{the guests}}^{\alpha}$ at the holiday party. [wh, no gap]

*I know that our uncle grabbed the food in front of $\overbrace{\text{at the holiday party}}^{\beta}$. [that, gap]

I know who our uncle grabbed the food in front of $\overbrace{\text{at the holiday party}}^{\beta}$. [wh, gap]

B.4.4 Tests for unboundedness

Filler–gap dependencies are “unbounded” in the sense that there is no limit to how many clausal levels above the gap the filler can be extracted. This serves as the basis for harder versions of the object-extracted test suites, involving three or four levels of clausal embedding. Example [that, no gap] sentences are given below:

I know that our mother said her friend remarked that the park attendant reported your friend threw the plastic into the trash can. [3 levels of embedding]

I know that our mother said her friend remarked that the park attendant reported the cop thinks your friend threw the plastic into the trash can. [4 levels of embedding]

These base sentences give rise to 4-condition test suites using the same manipulations as for the basic object-extraction test suite (Section 0.B.4.2), and the criterion for success is the same.

B.5 Main-verb/reduced-relative garden-path disambiguation

This is one of the best-studied instances of syntactic garden-pathing in the psycholinguistics literature. An example 4-condition item is given below:

!The child kicked in the chaos $\overbrace{\text{found}}^{\text{V}^{*}}$ her way back home. [reduced, ambig]

The child who was kicked in the chaos $\overbrace{\text{found}}^{\text{V}^{*}}$ her way back home.

The child forgotten in the chaos $\overbrace{\text{found}}^{\text{V}^{*}}$ her way back home.

The child who was forgotten in the chaos $\overbrace{\text{found}}^{\text{V}^{*}}$ her way back home.

Relative to the [reduced, ambig] condition, not reducing the relative clause should make $\text{V}^{*}$ less surprising, as should changing the participial verb to one that is the same form as a simple past-tense verb. Additionally, the effect of not reducing the relative clause on $\text{V}^{*}$ surprisal should be smaller for unambiguous participial verbs than for participial verbs:

Bever (1970); Ferreira and Clifton (1986); Trueswell et al. (1994); van Schijndel and Linzen (2018); Futrell et al. (2019)

B.6 Negative Polarity Licensing

The words any and ever, in their most common uses, are “negative polarity items” (NPIs): they can only be used in an appropriate syntactic-semantic environment—to a first approximation, in the scope of negation. For example, the determiner no can license NPIs, but its NP has to structurally command the NPI. Below, (A) and (D) are acceptable, because no is the determiner for the subject noun managers. There is no negation in (C) so the NPI is unlicensed and the sentence is unacceptable; crucially, however, (B) is unacceptable despite the presence of no earlier in the sentence, because no is embedded inside a modifier of the main-clause subject and thus does not command the NPI.

No managers that respected the guard have had $\overbrace{\text{any}}^{\text{NPI}}$ luck. [+neg,–distractor]

*The managers that respected no guard have had $\overbrace{\text{any}}^{\text{NPI}}$ luck. [–neg,+distractor]

*The managers that respected the guard have had $\overbrace{\text{any}}^{\text{NPI}}$ luck. [–neg,–distractor]

No managers that respected no guard have had $\overbrace{\text{any}}^{\text{NPI}}$ luck. [+neg,+distractor]

In the above test suite, the “distractor” position for no is inside a subject-extracted relative clause modifying the main-clause subject. We also used a variant test suite in which these relative clauses are object-extracted:

No managers that the guard respected have had $\overbrace{\text{any}}^{\text{NPI}}$ luck. [+neg,–distractor]

*The managers that no guard respected have had $\overbrace{\text{any}}^{\text{NPI}}$ luck. [–neg,+distractor]

*The managers that the guard respected have had $\overbrace{\text{any}}^{\text{NPI}}$ luck. [–neg,–distractor]

No managers that no guard respected have had $\overbrace{\text{any}}^{\text{NPI}}$ luck. [+neg,+distractor]

The above two test suites use any as the NPI; we also use test suites with ever as the NPI. Subject-extracted relative clause example:

No managers that respected the guard have $\overbrace{\text{ever}}^{\text{NPI}}$ gotten old. [+neg,–distractor]

*The managers that respected no guard have $\overbrace{\text{ever}}^{\text{NPI}}$ gotten old. [–neg,+distractor]

*The managers that respected the guard have $\overbrace{\text{ever}}^{\text{NPI}}$ gotten old. [–neg,–distractor]

No managers that respected no guard have $\overbrace{\text{ever}}^{\text{NPI}}$ gotten old. [+neg,+distractor]

Object-extracted relative clause example:

No managers that the guard respected have $\overbrace{\text{ever}}^{\text{NPI}}$ gotten old. [+neg,–distractor]

*The managers that no guard respected have $\overbrace{\text{ever}}^{\text{NPI}}$ gotten old. [–neg,+distractor]

*The managers that the guard respected have $\overbrace{\text{ever}}^{\text{NPI}}$ gotten old. [–neg,–distractor]

No managers that no guard respected have $\overbrace{\text{ever}}^{\text{NPI}}$ gotten old. [+neg,+distractor]

Changing the main-clause subject’s determiner from The to No should increase the probability of the NPI where it appears, regardless of whether there is a distractor no in the subject-modifying relative clause. Furthermore, when there is exactly one no in the sentence, the NPI should be higher-probability when it is in a licensing position rather than in a distractor position:

Ladusaw (1979); Vasishth et al. (2008); Giannakidou (2011); Marvin and Linzen (2018); Futrell et al. (2018)

B.7 NP/Z garden-path ambiguity

This is another well-studied syntactic garden-pathing configuration. In (A) below, the NP the waters introduces a local syntactic ambiguity: it could be (1) the direct object of crossed, in which case the sentence-initial subordinate clause has not yet ended, or (2) the subject of the main clause, in which case crossed is used intransitively and is the last word of the sentence-initial subordinate clause. (This was dubbed “NP/Z” by Sturt et al. (1999) because the subordinate-clause verb might have either an NP object or a Z(ero), i.e. null, object.) The next word, remained, is only compatible with (2); the ruling out of (1) generally yields increased processing difficulty for human comprehenders. Marking the end of the subordinate clause with a comma, as in (B), makes the sentence easier at $\text{V}^{*}$, as does an obligatorily intransitive subordinate-clause verb, as in (C).

!As the ship crossed the waters $\overbrace{\text{remained}}^{\text{V}^{*}}$ blue and calm. [trans,no comma]

As the ship crossed, the waters $\overbrace{\text{remained}}^{\text{V}^{*}}$ blue and calm. [trans,comma]

As the ship drifted the waters $\overbrace{\text{remained}}^{\text{V}^{*}}$ blue and calm. [intrans,no comma]

As the ship drifted, the waters $\overbrace{\text{remained}}^{\text{V}^{*}}$ blue and calm. [intrans,comma]

Similar to the main-verb/reduced-relative garden-pathing ambiguity, a model must pass a three-part criterion. Relative to (A), either marking the subordinate-clause end with a comma or using an obligatorily intransitive verb in the subordinate clause should reduce the surprisal of $\text{V}^{*}$. Furthermore, the surprisal-reduction effect of the comma should be smaller when the subordinate-clause verb is intransitive than when it is transitive:

We also use an NP/Z test suite where the second means of disambiguation is not changing the subordinate-clause verb to an intransitive, but rather giving the transitive subordinate-clause verb an overt direct object. For the above example item, the first two conditions are the same and the other two conditions would be:

As the ship crossed the sea the waters $\overbrace{\text{remained}}^{\text{V}^{*}}$ blue and calm.

As the ship crossed the sea, the waters $\overbrace{\text{remained}}^{\text{V}^{*}}$ blue and calm.

Finally, we create harder versions of both the above test suites by adding a postmodifier to the main-clause subject (in the above example, the waters becomes the waters of the Atlantic Ocean).

Frazier and Rayner (1982); Mitchell (1987); Pickering and Traxler (1998); Sturt et al. (1999); Staub (2007)

B.8 Subject–verb number agreement

This task tests a language model for how well it predicts the number marking on English finite present-tense verbs (whether it should be the third-person singular form, or the non-third-person-singular form, generally referred to as the plural form for simplicity, although technically this is the form for first- and second-person singular as well). In controlled, targeted versions of this test, multiple NP precede the verb: the verb’s actual subject, as well as a distractor NP with number that is different from that of the subject. A successful language model should place higher probability on the verbform matching that of the subject, not the distractor. We have three versions of this test suite: one where the distractor is in a prepositional phrase postmodifier of the subject:

The farmer near the clerks knows${}_{\text{V}_{\text{sg}}}$ many people.

*The farmer near the clerks know${}_{\text{V}_{\text{pl}}}$ many people.

The farmers near the clerk know${}_{\text{V}_{\text{pl}}}$ many people.

*The farmers near the clerk knows${}_{\text{V}_{\text{sg}}}$ many people.

one in which the distractor is in a subject-extracted relative clause postmodifier of the subject:

The farmer that embarrassed the clerks knows${}_{\text{V}_{\text{sg}}}$ many people.

*The farmer that embarrassed the clerks know${}_{\text{V}_{\text{pl}}}$ many people.

The farmers that embarrassed the clerk know${}_{\text{V}_{\text{pl}}}$ many people.

*The farmers that embarrassed the clerk knows${}_{\text{V}_{\text{sg}}}$ many people.

and one in which the distractor is in an object-extracted relative clause postmodifier of the subject:

The farmer that the clerks embarrassed knows${}_{\text{V}_{\text{sg}}}$ many people.

*The farmer that the clerks embarrassed know${}_{\text{V}_{\text{pl}}}$ many people.

The farmers that the clerk embarrassed know${}_{\text{V}_{\text{pl}}}$ many people.

*The farmers that the clerk embarrassed knows${}_{\text{V}_{\text{sg}}}$ many people.

Following Linzen et al. (2016) and Marvin and Linzen (2018), we require successful discrimination of the preferred upcoming verbform of the given lemma (rather than, for example, successful discrimination of the better context given a particular verbform). For success we require that a model successfully predicts the preferred verbform for both the singular- and plural-subject versions of an item:

Chance performance is thus 25%, though a context-insensitive baseline that places different probabilities on $\text{V}_{\text{sg}}$ and $\text{V}_{\text{pl}}$ would score 50%.

Bock and Miller (1991); Linzen et al. (2016); Marvin and Linzen (2018)

B.9 Reflexive pronoun licensing

The noun phrase with which a reflexive pronoun (herself, himself, themselves) corefers must command it in a sense similar to that relevant for negative-polarity items (Section 0.B.6). In the below example, the reflexive pronoun ending the sentence can only corefer to the subject of the sentence, author, with which it must agree in number: a singular subject requires a singular reflexive ${\text{R}_{\text{sg}}}$, and a plural subject requires a plural reflexive ${\text{R}_{\text{pl}}}$.

The author next to the senators hurt $\text{herself}_{\text{R}_{\text{sg.fem}}}$.

*The authors next to the senator hurt $\text{herself}_{\text{R}_{\text{sg.fem}}}$.

The authors next to the senator hurt $\text{themselves}_{\text{R}_{\text{pl}}}$.

*The authors next to the senator hurt $\text{themselves}_{\text{R}_{\text{pl}}}$.

We generated a pair of test suites—one in which the singular reflexive is herself, and another where the singular reflexive is himself, on the template of the above example, where the distractor NP is in a prepositional-phrase postmodifier of the subject NP. We also generated a similar pair of test suites where the distractor NP is inside a subject-extracted relative clause modifying the subject:

The author that liked the senators hurt $\text{herself}_{\text{R}_{\text{sg.fem}}}$.

*The authors that liked the senator hurt $\text{herself}_{\text{R}_{\text{sg.fem}}}$.

The authors that liked the senator hurt $\text{themselves}_{\text{R}_{\text{pl}}}$.

*The authors that liked the senator hurt $\text{themselves}_{\text{R}_{\text{pl}}}$.

and a pair of test suites where the distractor NP is inside an object-extracted relative clause modifying the subject:

The author that the senators liked hurt $\text{herself}_{\text{R}_{\text{sg.fem}}}$.

*The authors that the senator liked hurt $\text{herself}_{\text{R}_{\text{sg.fem}}}$.

The authors that the senator liked hurt $\text{themselves}_{\text{R}_{\text{pl}}}$.

*The authors that the senator liked hurt $\text{themselves}_{\text{R}_{\text{pl}}}$.

For each item in each test suite, we require that for both the singular and the plural versions of the reflexive pronoun the model assign higher conditional probability in the correct licensing context than in the incorrect licensing context:

Reinhart (1981); Marvin and Linzen (2018)

B.10 Subordination

Beginning a sentence with As, When, Before, After, or Because, implies that an immediately following clause is not the main clause of the sentence, as would have otherwise been the case, but instead is a subordinate clause that must be followed by the main clause. Ending the sentence without a main clause, as in (B), is problematic. Conversely, following an initial clause with a second clause MC (without linking it to the initial clause with and, but, despite, or a similar coordinator or subordinator), as in (C) below, is unexpected and odd.

The minister praised the building$\overbrace{\text{.}}^{\text{{End}}}$

*After the minister praised the building$\overbrace{\text{.}}^{\text{{End}}}$

??The minister praised the building$\overbrace{\text{, it started to rain.}}^{\text{{MC}}}$

After the minster praised the building$\overbrace{\text{, it started to rain.}}^{\text{{MC}}}$

In addition to the base test suite exemplified by the item above, we include three versions with longer and more complex initial clauses, which may make the test suite more difficult. In the first of these versions, we postmodify both the subject and object of the initial clauses with prepositional phrases:

the minister in the dark suit and white tie praised the new building on the town’s main square

In the second of these versions, the postmodifiers are subject-extracted relative clauses:

the minister who wore a black suit praised the new building that was built by the square

In the third of these versions, the postmodifiers are object-extracted relative clauses:

the minister who the mayor had invited praised the new building that the businessman had built downtown

Introducing a subordinator at the beginning of the sentence should make an ending without a second clause less probable, and should make a second clause more probable: