Who did What: A Large-Scale Person-Centered Cloze Dataset

Introduction

Researchers distinguish the problem of general knowledge question answering from that of reading comprehension [Hermann et al., 2015, Hill et al., 2016]. Reading comprehension is more difficult than knowledge-based or IR-based question answering in two ways. First, reading comprehension systems must infer answers from a given unstructured passage rather than structured knowledge sources such as Freebase [Bollacker et al., 2008] or the Google Knowledge Graph [Singhal, 2012]. Second, machine comprehension systems cannot exploit the large level of redundancy present on the web to find statements that provide a strong syntactic match to the question [Yang et al., 2015]. In contrast, a machine comprehension system must use the single phrasing in the given passage, which may be a poor syntactic match to the question.

In this paper, we describe the construction of a new reading comprehension dataset that we refer to as “Who-did-What”. Two typical examples are shown in Table 1.The passages here only show certain salient portions of the passage. In the actual dataset, the entire article is given. The correct answers are (3) and (2). The process of forming a problem starts with the selection of a question article from the English Gigaword corpus. The question is formed by deleting a person named entity from the first sentence of the question article. An information retrieval system is then used to select a passage with high overlap with the first sentence of the question article, and an answer choice list is generated from the person named entities in the passage.

Our dataset differs from the CNN and Daily Mail comprehension tasks [Hermann et al., 2015] in that it forms questions from two distinct articles rather than summary points. This allows problems to be derived from document collections that do not contain manually-written summaries. This also reduces the syntactic similarity between the question and the relevant sentences in the passage, increasing the need for deeper semantic analysis.

To make the dataset more challenging we selectively remove problems so as to suppress four simple baselines — selecting the most mentioned person, the first mentioned person, and two language model baselines. This is also intended to produce problems requiring deeper semantic analysis.

The resulting dataset yields a larger gap between human and machine performance than existing ones. Humans can answer questions in our dataset with an 84% success rate compared to the estimates of 75% for CNN [Chen et al., 2016] and 82% for the CBT named entities task [Hill et al., 2016]. In spite of this higher level of human performance, various existing readers perform significantly worse on our dataset than they do on the CNN dataset. For example, the Attentive Reader [Hermann et al., 2015] achieves 63% on CNN but only 55% on Who-did-What and the Attention Sum Reader [Kadlec et al., 2016] achieves 70% on CNN but only 59% on Who-did-What.

In summary, we believe that our Who-did-What dataset is more challenging, and requires deeper semantic analysis, than existing datasets.

Related Work

Our Who-did-What dataset is related to several recently developed datasets for machine comprehension. The MCTest dataset [Richardson et al., 2013] consists of 660 fictional stories with 4 multiple choice questions each. This dataset is too small to train systems for the general problem of reading comprehension.

The bAbI synthetic question answering dataset [Weston et al., 2016] contains passages describing a series of actions in a simulation followed by a question. For this synthetic data a logical algorithm can be written to solve the problems exactly (and, in fact, is used to generate ground truth answers).

The Children’s Book Test (CBT) dataset, created by ?), consists of 113,719 cloze-style named entity problems. Each problem consists of 20 consecutive sentences from a children’s story, a 21st sentence in which a word has been deleted, and a list of ten choices for the deleted word. The CBT dataset tests story completion rather than reading comprehension. The next event in a story is often not determined — surprises arise. This may explain why human performance is lower for CBT than for our dataset — 82% for CBT vs. 84% for Who-did-What. The 16% error rate for humans on Who-did-What seems to be largely due to noise in problem formation introduced by errors in named entity recognition and parsing. Reducing this noise in future versions of the dataset should significantly improve human performance. Another difference compared to CBT is that Who-did-What has shorter choice lists on average. Random guessing achieves only 10% on CBT but 32% on Who-did-What. The reduction in the number of choices seems likely to be responsible for the higher performance of an LSTM system on Who-did-What – contextual LSTMs (the attentive reader of Hermann et al., 2015) improve from 44% on CBT (as reported by Hill et al., 2016) to 55% on Who-did-What.

Above we referenced the comprehension datasets created from CNN and Daily Mail articles by ?). The CNN and Daily Mail datasets together consist of 1.4 million questions constructed from approximately 300,000 articles. Of existing datasets, these are the most similar to Who-did-What in that they consists of cloze-style question answering problems derived from news articles. As discussed in Section 1, our Who-did-What dataset differs from these datasets in not being derived from article summaries, in using baseline suppression, and in yielding a larger gap between machine and human performance. The Who-did-What dataset also differs in that the person named entities are not anonymized, permitting the use of external resources to improve performance while remaining difficult for language models due to suppression.

Dataset Construction

We now describe the construction of our Who-did-What dataset in more detail. We sketch the procedure below and provide more specific details in the appendix. To generate a problem we first generate the question by selecting a random article — the “question article” — from the Gigaword corpus and taking the first sentence of that article — the “question sentence” — as the source of the cloze question. The hope is that the first sentence of an article contains prominent people and events which are likely to be discussed in other independent articles. To convert the question sentence to a cloze question, we first extract named entities using the Stanford NER system [Finkel et al., 2005] and parse the sentence using the Stanford PCFG parser [Klein and Manning, 2003].

The person named entities are candidates for deletion to create a cloze problem. For each person named entity we then identify a noun phrase in the automatic parse that is headed by that person. For example, if the question sentence is “President Obama met yesterday with Apple Founder Steve Jobs” we identify the two person noun phrases “President Obama” and “Apple Founder Steve Jobs”. When a person named entity is selected for deletion, the entire noun phrase is deleted. For example, when deleting the second named entity, we get “President Obama met yesterday with XXX” rather than “President Obama met yesterday with Apple founder XXX”. This increases the difficulty of the problems because systems cannot rely on descriptors and other local contextual cues. About 700,000 question sentences are generated from Gigaword articles (8% of the total number of articles).

Once a cloze question has been formed we select an appropriate article as a passage. The article should be independent of the question article but should discuss the people and events mentioned in the question sentence. To find a passage we search the Gigaword dataset using the Apache Lucene information retrieval system [McCandless et al., 2010], using the question sentence as the query. The named entity to be deleted is included in the query and required to be included in the returned article. We also restrict the search to articles published within two weeks of the date of the question article. Articles containing sentences too similar to the question in word overlap and phrase matching near the blanked phrase are removed. We select the best matching article satisfying our constraints. If no such article can be found, we abort the process and move on to a new question. See the appendix for details.

Given a question and a passage we next form the list of choices. We collect all person named entities in the passage except unblanked person named entities in the question. Choices that are subsets of longer choices are eliminated. For example the choice “Obama” would be eliminated if the list also contains “Barack Obama”. We also discard ambiguous cases where a part of a blanked NE appears in multiple candidate answers, e.g., if a passage has “Bill Clinton” and “Hillary Clinton” and the blanked phrase is “Clinton”. We found this simple coreference rule to work well in practice since news articles usually employ full names for initial mentions of persons. If the resulting choice list contains fewer than two or more than five choices, the process is aborted and we move on to a new question.The maximum of five helps to avoid sports articles containing structured lists of results.

After forming an initial set of problems we then remove “duplicated” problems. Duplication arises because Gigaword contains many copies of the same article or articles where one is clearly an edited version of another. Our duplication-removal process ensures that no two problems have very similar questions. Here, similarity is defined as the ratio of the size of the bag of words intersection to the size of the smaller bag.

In order to focus our dataset on the most interesting problems, we remove some problems to suppress the performance of the following simple baselines:

First person in passage: Select the person that appears first in the passage.

Most frequent person: Select the most frequent person in the passage.

$n$-gram: Select the most likely answer to fill the blank under a 5-gram language model trained on Gigaword minus articles which are too similar to one of the questions in word overlap and phrase matching.

Unigram: Select the most frequent last name using the unigram counts from the 5-gram model.

To minimize the number of questions removed we solve an optimization problem defined by limiting the performance of each baseline to a specified target value while removing as few problems as possible, i.e.,

where $T(C)$ is the subset of the questions solved by the subset $C$ of the suppressed baselines, and $\alpha(C)$ is a keeping rate for question set $T(C)$. $C_{i}=1$ indicates $i$-th baseline is in the subset and $|b|$ is a number of baselines. Then $N$ is a total number of questions and $k$ is an upper bound for the baselins after suppression. $k$ is set to the random performance. The performance of these baselines before and after suppression are shown in Table 2. The suppression removed 49.9% of the questions.

Table 3 shows statistics of our dataset after suppression. We split the final dataset into train, validation, and test by taking the validation and test to be a random split of the most recent 20,000 problems as measured by question article date. In this way there is very little overlap in semantic subject matter between the training set and either validation or test. We also provide a larger “relaxed” training set formed by applying less baseline suppression (a larger value of $k$ in the optimization). The relaxed training set then has a slightly different distribution from the train, validation, and test sets which are all fully suppressed.

Performance Benchmarks

We report the performance of several systems to characterize our dataset:

Word overlap: Select the choice $c$ inserted to the question $q$ which is the most similar to any sentence $s$ in the passage, i.e., ${\rm CosSim}({\rm bag}(c+q),{\rm bag}(s))$.

Sliding window and Distance baselines (and their combination) from ?).

Semantic features: NLP feature based system from ?).

Attentive Reader: LSTM with attention mechanism [Hermann et al., 2015].

Stanford Reader: An attentive reader modified with a bilinear term [Chen et al., 2016].

Attention Sum (AS) Reader: GRU with a point-attention mechanism [Kadlec et al., 2016].

Gated-Attention (GA) Reader: Attention Sum Reader with gated layers [Dhingra et al., 2016].

Table 4 shows the performance of each system on the test data. For the Attention and Stanford Readers, we anonymized the Who-did-What data by replacing named entities with entity IDs as in the CNN and Daily Mail datasets.

We see consistent reductions in accuracy when moving from CNN to our dataset. The Attentive and Stanford Reader drop by up to 10% and the AS and GA reader drop by up to 17%. The ranking of the systems also changes. In contrast to the Attentive/Stanford readers, the AS/GA readers explicitly leverage the frequency of the answer in the passage, a heuristic which appears beneficial for the CNN and Daily Mail tasks. Our suppression of the most-frequent-person baseline appears to more strongly affect the performance of these latter systems.

Conclusion

We presented a large-scale person-centered cloze dataset whose scalability and flexibility is suitable for neural methods. This dataset is different in a variety of ways from existing large-scale cloze datasets and provides a significant extension to the training and test data for machine comprehension.

References

Appendix

We include pseudocode for generating questions (Alg. 1) and multiple choice answer sets (Alg. 2).