The Natural Stories Corpus

Introduction

It is becoming a standard practice to evaluate theories of human language processing by their ability to predict psychometric dependent variables such as per-word reaction time for standardized corpora of naturalistic text. Dependent variables that have been collected over fixed corpora include word fixation time in eyetracking (Kennedy et al., 2003), word reaction time in self-paced reading (Roark et al., 2009; Frank et al., 2013), BOLD signal in fMRI data (Bachrach et al., ms), and event-related potentials (Dambacher et al., 2006; Frank et al., 2015).

The more traditional approach to evaluating psycholinguistic models has been to collect psychometric measures on hand-crafted experimental stimuli designed to tease apart detailed model predictions. While this approach makes it easy to compare models on their accuracy for specific constructions and phenomena, it is hard to get a sense from experimental results of how models compare on their coverage of a broad range of phenomena. When it is standard practice to compare model predictions over standardized texts, then it is easier to evaluate coverage.

Although the fixed corpus approach has these advantages, the existing corpora currently used are based on naturally-occurring data, which is unlikely to include the kinds of sentences which can crucially distinguish between theories. Many of the most puzzling phenomena in psycholinguistics, and the phenomena which have been used to test models, have only been observed in extremely rare constructions, such as multiply nested preverbal relative clauses (Gibson and Thomas, 1999; Grodner and Gibson, 2005; Vasishth et al., 2010). Corpora of naturally-occurring text are unlikely to contain these constructions. More generally, models of human language comprehension are more likely to make distinct predictions for sentences that cause difficulty for humans, rather than for sentences that are easy to process. For instance, models of comprehension difficulty based on memory integration cost during parsing (Gibson, 2000; Lewis and Vasishth, 2005; Schuler et al., 2010; van Schijndel et al., 2013) will predict effects when the memory spans required for parsing are large, but most syntactic dependencies in naturally-occurring text are short (Temperley, 2007; Liu, 2008; Futrell et al., 2015). In general, processing difficulty might be rare for naturally-occurring text, because text written and edited in order to be easily understood.

Here we attempt to combine the strength of experimental approaches, which can test theories using targeted low-frequency structures, and corpus studies, which provide broad-coverage comparability between models. We introduce and release a new corpus, the Natural Stories Corpus, a series of English narrative texts designed to contain many low-frequency and psycholinguistically interesting syntactic constructions while still sounding fluent and coherent. The texts are annotated with hand-corrected Penn Treebank style phrase structure parses, and Universal Dependencies parses automatically generated from the phrase structure parses. We also release self-paced reading time data for all texts, and word-aligned audio recordings of the texts. We hope the corpus can form the basis for further annotation and become a standard test set for psycholinguistic models.

Related Work

Here we survey datasets which are commonly used to test psycholinguistic theories and how they relate to the current release.

Currently the most prominent psycholinguistic corpus for English is the Dundee Corpus (Kennedy et al., 2003), which contains 51,501 word tokens in 2,368 sentences from British newspaper editorials, along with eyetracking data from 10 participants. A dependency parse of the corpus is released in Barrett et al. . Like in the current work, the eyetracking data in the Dundee corpus is collected for sentences in context and so reflects influences beyond the sentence level. The corpus has seen wide usage, see for example Demberg and Keller (2008); Mitchell et al. (2010); Frank and Bod (2011); Fossum and Levy (2012); Smith and Levy (2013); van Schijndel and Schuler (2015); Luong et al. (2015).

The Potsdam Sentence Corpus (Kliegl et al., 2006) of German provides 1138 words in 144 sentences, with cloze probabilities and eyetracking data for each word. Like the current corpus, the Potsdam Sentence Corpus was designed to contain varied syntactic structures, rather than being gathered from naturalistic text. The corpus consists of isolated sentences which do not form a narrative, and during eyetracking data collection the sentences were presented in a random order. The corpus has been used to evaluate models of sentence processing based on dependency parsing (Boston et al., 2008, 2011) and to study effects of predictability on event-related potentials (Dambacher et al., 2006).

The MIT Corpus introduced in Bachrach et al. (ms) has similar aims to the current work, collecting reading time and fMRI data over sentences designed to contain varied structures. This dataset consists of four narratives with a total of 2647 tokens; it has been used to evaluate models of incremental prediction in Roark et al. (2009), Wu et al. (2010), and Luong et al. (2015).

The UCL Corpus (Frank et al., 2013) consists of 361 English sentences drawn from amateur novels, chosen for their ability to be understood out of context, with self-paced reading and eyetracking data. The goal of the corpus is to provide a sample of typical narrative sentences, complementary to our goal of providing a corpus with low-frequency constructions. Unlike the current corpus, the UCL Corpus consists of isolated sentences, so the psychometric data do not reflect effects beyond the sentence level.

Eyetracking corpora for other languages are also available, including the Postdam-Allahabad Hindi Eyetracking Corpus (Husain et al., 2014) and the Beijing Sentence Corpus of Mandarin Chinese (Yan et al., 2010).

Corpus Description

The Natural Stories corpus consists of 10 stories, comprising 10,245 lexical word tokens and 485 sentences in total. The stories were developed by A.V., E.F., E.G. and S.P. by taking existing publicly available texts and editing them to use many subject- and object-extracted relative clauses, clefts, topicalized structures, extraposed relative clauses, sentential subjects, sentential complements, local structural ambiguity (especially NP/Z ambiguity), idioms, and conjoined clauses with a variety of coherence relations. The original texts are listed in Table 1.

The mean number of lexical words per sentence is 21.1, around the same as the Dundee corpus (21.7). Figure 1 shows a histogram of sentence length in Natural Stories as compared to Dundee. The word and sentence counts for each story are given in Table 2. Each token has a unique code which is referenced throughout the various annotations of the corpus.

In Figure 2 we give a sample of text from the corpus (from the first story).

2 Parses

The texts were parsed automatically using the Stanford Parser (Klein and Manning, 2003) and hand-corrected. Trace annotations were added by hand. We provide the resulting Penn Treebank-style phrase structure parse trees. We also provide Universal Dependencies parses (Nivre, 2015) automatically converted from the corrected parse trees using the Stanford Parser.

3 Self-Paced Reading Data

We collected self-paced reading (SPR) data (Just et al., 1982) for the stories from 181 native English speakers over Amazon Mechanical Turk. Text was presented in a dashed moving window display; spaces were masked. Each participant read 5 stories per HIT. 19 participants read all 10 stories, and 3 participants stopped after one story. Each story was accompanied by 6 comprehension questions. We discarded SPR data from a participant’s pass through a story if the participant got less than 5 questions correct (89 passes through stories excluded). We also excluded RTs less than 100 ms or greater than 3000 ms. Figure 3 shows histograms of RTs per story.

In order to evaluate the reliability of the self-paced reading RTs and their robustness across experimental participants, we analyzed inter-subject correlations (ISCs). For each subject, we correlated the Spearman correlation of that subject’s RTs on a story with average RTs from all other subjects on that story. Thus for each story we get one ISC statistic per subject. Figure 4 shows histograms of these statistics per story.

3.2 Psycholinguistic Sanity Checks

In order to validate our RT data, we checked that basic psycholinguistic effects obtain in it. In particular, we examined whether the well-known effects of frequency, word length, and surprisal (Hale, 2001; Levy, 2008; Smith and Levy, 2013) had an effect on RTs. To do this, for each of the three predictors log frequency, log trigram probability, and word length, we fit a linear mixed effects regression model with subject and story as random intercepts (models with random slopes did not converge) predicting RT. Frequency and trigram probabilities were computed from Google Books N-grams, summing over years from 1990 to 2013. (These counts are also released with this dataset.) The results of the regressions are shown in Table 3; we report results from the maximal converging model. In keeping with well-known effects, increased frequency and probability both lead to faster reading times, and word length leads to slower reading times.

4 Syntactic Constructions

Here we give an overview of the low-frequency or marked syntactic constructions which occur in the stories. We coded sentences in the Natural Stories corpus for presence of a number of marked constructions, and also coded 200 randomly selected sentences from the Dundee corpus for the same features. The features coded are listed and explained in Appendix A. Figure 5 shows the rates of these marked constructions per sentence in the two corpora. From the figure, we see that the natural stories have especially high rates of nonlocal VP conjunction, nonrestrictive SRCs, idioms, adjective conjunction, noncanonical ORCs, local NP/S ambiguities, and it-clefts.

Conclusion

We have described a new psycholinguistic corpus of English, consisting of edited naturalistic text designed to contain many rare or hard-to-process constructions while still sounding fluent. We believe this corpus will provide an important part of a suite of test sets for psycholinguistic models, exposing their behavior in uncommon constructions in a way that fully naturalistic corpora cannot. We also hope that the corpus as described here forms the basis for further data collection and annotation.

Acknowledgments

This work was supported by NSF DDRI grant #1551543 to R.F., NSF grants #0844472 and #1534318 to E.G., and NIH career development award HD057522 to E.F. The authors thank the following individuals: Laura Stearns for hand-checking and correcting the parses, Suniyya Waraich for help with syntactic coding, Cory Shain and Marten van Schijndel for hand-annotating the parses, and Kyle Mahowald for help with initial exploratory analyses of the SPR data. The authors also thank Nancy Kanwisher for recording half of the stories (the other half was recorded by E.G.), Wade Shen for providing initial alignment between the audio files and the texts, and Jeanne Gallee for hand-correcting the alignment.

References

Appendix A Features coded for Section 3.4

Local/nonlocal VP conjunction: Conjunction of VPs in which the head verbs are adjacent (local) or not adjacent (nonlocal)

Local/nonlocal NP conjunction: Conjunction of VPs in which the head nouns are adjacent (local) or not adjacent (nonlocal).

Sentential conjunction: Conjunction of sentences.

CP conjunction: Conjunction of CPs with explicit quantifiers.

Restrictive/nonrestrictive SRC: Subject-extracted relative clauses with either restrictive or nonrestrictive semantics

Restrictive/nonrestrictive ORC: Object-extracted relative clauses with either restrictive or nonrestrictive semantics

No-relativizer ORC: An object-extracted relative clause without an explicit relativizer, e.g. The man I know

Noncanonical ORC: An object-extracted relative clause where the subject is not a pronoun.

Adverbial relative clause: An relative clause with an extracted adverbial, e.g. the valley where you would find the city of Bradford.

NP/S ambiguity: A local ambiguity where it is unclear whether a clause is an NP or the subject of a sentence. For example, I know Bob is a doctor.

Main Verb/Reduced Relative ambiguity (easy/hard): A local ambiguity between a main verb and a reduced relative clause. For example, The horse raced past the barn fell.

Nonlocal SV: The appearance of any material between a verb and the head of its subject.

Nonlocal Verb/DO: The appearance of any material between a verb and its direct object.

Idiom: Any idiomatic expression, such as busy as beavers.

Coding was performed by Suniyya Waraich, Edward Gibson, and Richard Futrell.