DREAM: A Challenge Dataset and Models for Dialogue-Based Reading Comprehension

Introduction

Recently a significant amount of research has focused on the construction of large-scale multiple-choice Lai et al. (2017); Khashabi et al. (2018); Ostermann et al. (2018) and extractive Hermann et al. (2015); Hill et al. (2016); Rajpurkar et al. (2016); Trischler et al. (2017) reading comprehension datasets (Section 2). Source documents in these datasets have generally been drawn from formal written texts such as news, fiction, and Wikipedia articles, which are commonly considered well-written, accurate, and neutral.

With the goal of advancing research in machine reading comprehension and facilitating dialogue understanding, we construct and present in this paper DREAM — the first multiple-choice Dialogue-based REAding comprehension exaMination dataset. We collect 10,197 questions for 6,444 multi-turn multi-party dialogues from English language exams that are carefully designed by educational experts (e.g., English teachers) to assess the comprehension level of Chinese learners of English. Each question is associated with three answer options, exactly one of which is correct. (See Table 1 for an example.) DREAM covers a variety of topics and scenarios in daily life such as conversations on the street, on the phone, in a classroom or library, at the airport or the office or a shop (Section 3).

Based on our analysis of DREAM, we argue that dialogue-based reading comprehension is at least as difficult as existing non-conversational counterparts. In particular, answering 34% of DREAM questions requires unspoken commonsense knowledge, e.g., unspoken scene information. This might be due to the nature of dialogues: for efficient oral communication, people rarely state obvious explicit world knowledge Forbes and Choi (2017) such as “Christmas Day is celebrated on December 25th”. Understanding the social implications of an utterance as well as inferring a speaker’s intentions is also regularly required for answering dialogue-based questions. The dialogue content in Table 1, for example, is itself insufficient for readers to recognize the intention of the female speaker (W) in the first question (Q$1$). However, world knowledge is rarely considered in state-of-the-art reading comprehension models Tay et al. (2018); Wang et al. (2018b).

Moreover, dialogue-based questions can cover information imparted across multiple turns involving multiple speakers. In DREAM, approximately 85% of questions can only be answered by considering the information from multiple sentences. For example, to answer Q$1$ in Table 3 regarding the date of birth of the male speaker (M), the supporting sentences (in bold) include “You know, tomorrow is Christmas Day” from the female speaker and “…I am more than excited about my birthday, which will come in two days” from the male speaker. Compared to “multiple-sentence questions” in traditional reading comprehension datasets, DREAM further requires an understanding of the turn-based structure of dialogue, e.g. for aligning utterances with their corresponding speakers.

As only 16% of correct answers are text spans from the source documents, we primarily explore rule-based methods and state-of-the-art neural models designed for multiple-choice reading comprehension (Section 4). We find first that neural models designed for non-dialogue-based reading comprehension Chen et al. (2016); Dhingra et al. (2017); Wang et al. (2018b) do not fare well: the highest achieved accuracy is $45.5\%$, only slightly better than the accuracy of a simple lexical baseline Richardson et al. (2013) ($44.6\%$). For the most part, these models fundamentally exploit only surface-level information from the source documents. Considering the above-mentioned challenges, however, we hypothesize that incorporating general world knowledge and aspects of the dialogue structure would allow a better understanding of the dialogues. As a result, we modify our baseline systems to include (1) general world knowledge in the form of such as ConceptNet relations Speer et al. (2017) and a pre-trained language model Radford et al. (2018), and (2) speaker information for each utterance. Experiments show the effectiveness of these factors on the lexical baselines as well as neural and non-neural machine learning approaches: we acquire up to $11.9\%$ absolute gain in accuracy compared to the highest performance achieved by the state-of-the-art reading comprehension model Wang et al. (2018b) that mainly relies on explicit surface-level information in the text (Section 5).

Finally, we see a significant gap between the best automated approach ($59.5\%$) and human ceiling performance ($98.6\%$) on the DREAM dataset. This provides yet additional evidence that dialogue-based reading comprehension is a very challenging task. We hope that it also inspires the research community to develop methods for the dialogue-based reading comprehension task.

Related Work

We divide reading comprehension datasets into three categories based on the types of answers.

In recent years, we have seen increased interest in large-scale cloze/span-based reading comprehension dataset construction Hermann et al. (2015); Hill et al. (2016); Onishi et al. (2016); Rajpurkar et al. (2016); Bajgar et al. (2016); Nguyen et al. (2016); Trischler et al. (2017); Joshi et al. (2017); Choi et al. (2018). We regard them as extractive since candidate answers are usually short spans from source documents. State-of-the-art neural models with attention mechanisms already achieve very high performance based on local lexical information. Recently researchers work on the construction of spoken span-based datasets Lee et al. (2018); Li et al. (2018) by applying text-to-speech technologies or recruiting human speakers based on formal written document-based datasets such as SQuAD Rajpurkar et al. (2016). Some span-based conversation datasets are constructed from a relatively small size of dialogues from TV shows Chen and Choi (2016); Ma et al. (2018).

Considering the limitations in extractive datasets, answers in abstractive datasets such as MS MARCO Nguyen et al. (2016), SearchQA Dunn et al. (2017), and NarrativeQA Kočiskỳ et al. (2018) are human generated based on source documents or summaries. Concurrently, there is a growing interest in conversational question answering such as CoQA Reddy et al. (2018). Since annotators tend to copy spans as answers Reddy et al. (2018), the majority of answers are still extractive in these datasets (Table 2). Compared to the datasets mentioned above, most of the correct answer options (83.7%) in DREAM are free-form text.

2 Multiple-Choice Datasets

We primarily discuss the multiple-choice datasets in which answer options are not restricted to extractive text spans in the given document. Instead, most of the correct answer options are abstractive (Table 2). Multiple-choice datasets involve extensive human involvement for problem generation during crowdsourcing (i.e., questions, correct answer option, and distractors). Besides surface matching, a significant portion of questions require multiple-sentence reasoning and external knowledge Richardson et al. (2013); Mostafazadeh et al. (2016); Khashabi et al. (2018); Ostermann et al. (2018).

Besides crowdsourcing, some datasets are collected from examinations designed by educational experts Penas et al. (2014); Shibuki et al. (2014); Tseng et al. (2016); Clark et al. (2016); Lai et al. (2017), which aim to test human examinees. There are various types of complicated questions such as math word problems, summarization, logical reasoning, and sentiment analysis. Since we can adopt more objective evaluation criteria such as accuracy, these questions are usually easy to grade. Besides, questions from examinations are generally clean and high-quality. Therefore, human performance ceiling on this kind of datasets is much higher (e.g., $94.5\%$ on RACE Lai et al. (2017) and $98.6\%$ on DREAM in accuracy) than that of datasets built by crowdsourcing.

In comparison, we present the first multiple-choice dialogue-based dataset from examinations that contains a large percentage of questions that require multiple sentence inference. To the best of our knowledge, DREAM also contains the largest number of questions involving commonsense reasoning compared to other examination datasets.

Data

In this section, we describe how we construct DREAM (Section 3.1) and provide a detailed analysis of this dataset (Section 3.2).

We collect dialogue-based comprehension problems from a variety of English language exams (including practice exams) such as National College Entrance Examination, College English Test, and Public English TestWe list all the websites used for data collection in the released dataset., which are designed by human experts to assess either the listening or reading comprehension level of Chinese English learners in high schools and colleges (aged $12$-$22$). All the problems in DREAM are freely accessible online for public usage. Each problem consists of a dialogue and a series of multiple-choice questions. To ensure every question is associated with exactly three answer options, we drop wrong option(s) randomly for questions with more than three options. We remove duplicate problems and randomly split the data at the problem level, with $60\%$ train, $20\%$ development, and $20\%$ test.

2 Data Analysis

We summarize the statistics of DREAM in Table 4 and data split in Table 5. Compared to existing datasets built from formal written texts, the vocabulary size is relatively small since spoken English by its nature makes greater use of high-frequency words and needs a smaller vocabulary for efficient real-time communication Nation (2006).

We categorize questions into two main categories according to the types of knowledge required to answer them: matching and reasoning.

Matching A question is entailed or paraphrased by exactly one sentence in a dialogue. The answer can be extracted from the same sentence. For example, we can easily verify the correctness of the question-answer pair (“What kind of room does the man want to rent?”, “A two-bedroom apartment.”) based on the sentence “M: I’m interested in renting a two-bedroom apartment”. This category is further divided into two categories word matching and paraphrasing in previous work Chen et al. (2016); Trischler et al. (2017).

Reasoning Questions that cannot be answered by the surface meaning of a single sentence belong to this category. We further define four subcategories as follows.

Summary Answering this kind of questions requires the whole picture of a dialogue, such as the topic of a dialogue and the relation between speakers (e.g., D$2$-Q$3$ in Table 3). Under this category, questions such as “What are the two speakers talking about?” and “What are the speakers probably doing?" are frequently asked.

Logic We require logical reasoning to answer questions in this category. We usually need to identify logically implied relations among multiple sentences in a dialogue. To reduce the ambiguity during the annotation, we regard a question that can only be solved by considering the content from multiple sentences and does not belong to the summary subcategory that involves all the sentences in a dialogue as a logic question. Following this definition, both D$2$-Q$1$ and D$2$-Q$2$ in Table 3 belong to this category.

Arithmetic Inferring the answer requires arithmetic knowledge (e.g., D$2$-Q$1$ in Table 3 requires $25-1+2=26$).

Commonsense To answer questions under this subcategory, besides the textual information in the dialogue, we also require additional commonsense knowledge that cannot be obtained from the dialogue. For instance, all questions in Table 3 fall under this category. D$2$-Q$1$ and D$2$-Q$2$ in Table 3 belong to both logic and commonsense since they require multiple sentences as well as commonsense knowledge for question answering. There exist multiple types of commonsense knowledge in DREAM such as the well-known properties of a highly-recognizable entity (e.g., D$2$-Q$1$ in Table 3), the prominent relationship between two speakers (e.g., D$2$-Q$3$ in Table 3), the knowledge of or shared by a particular culture (e.g., when a speaker says “Cola? I think it tastes like medicine.”, she/he probably means “I don’t like cola.”), and the cause-effect relation between events (e.g., D$1$-Q$1$ in Table 1). We refer readers to LoBue and Yates (2011) for detailed definitions.

Table 6 shows the question type distribution labeled by two human annotators on $25\%$ questions randomly sampled from the development and test sets. Besides the previously defined question categories, we also report the percentage of questions that require reasoning over multiple sentences (i.e., summary or logic questions) and the percentage of questions that require the surface-level understanding or commonsense/math knowledge based on the content of a single sentence. As a question can belong to multiple reasoning subcategories, the summation of the percentage of reasoning subcategories is not equal to the percentage of reasoning. The Cohen’s kappa coefficient is $0.67$ on the development set and $0.68$ on the test set.

Dialogues in DREAM are generally clean and mostly error-free since they are carefully designed by educational experts. However, it is not guaranteed that each dialogue is written or proofread by a native speaker. Besides, dialogues tend to be more proper and less informal for exam purposes. To have a rough estimation of the quality of dialogues in DREAM and the differences between these dialogues and more casual ones in movies or TV shows, we run a proofreading tool – Grammarlyhttps://app.grammarly.com. – on all the dialogues from the annotated $25\%$ instances of the development set and the same size ($20.7$k tokens) of dialogues from Friends, a famous American TV show whose transcripts are commonly used for dialogue understanding Chen and Choi (2016); Ma et al. (2018). As shown in Table 7, there exist fewer spelling mistakes and the overall score is slightly higher than that of the dialogues in Friends. Based on the evaluated instances, articles and verb forms are the two most frequent grammar error categories ($10$ and $8$, respectively, out of $23$) in DREAM. Besides, the language tends to be less precise in DREAM, indicated by the number of vocabulary suggestions. For example, experts tend to use expressions such as “really hot”, “really beautiful”, “very bad”, and “very important” instead of more appropriate yet more advanced adjectives that might hinder reading comprehension of learners with smaller vocabularies. According to the explanations provided by the tool, the readability scores for both datasets fall into the same category “Your text is very simple and easy to read, likely to be understood by an average 5th-grader (age 10)”.

Approaches

We formally introduce the dialogue-based reading comprehension task and notations in Section 4.1. To investigate the effects of different kinds of general world knowledge and dialogue structure, we incorporate them into rule-based approaches (Section 4.2) as well as non-neural (Section 4.3) and neural (Section 4.4) machine learning approaches. We describe in detail preprocessing and training in Section 4.5.

We start with a formal definition of the dialogue-based multiple-choice reading comprehension task. An $n$-turn dialogue $D$ is defined as $D=\{s_{1}\colon t_{1},s_{2}\colon t_{2},\ldots,s_{n}\colon t_{n}$}, where $s_{i}$ represents the speaker ID (e.g., “M” and “W”), and $t_{i}$ represents the text of the $i^{th}$ turn. Let $Q$ denote the text of question, and $O_{1..3}$ denote the text of three answer options. The task is to choose the correct one from answer options $O_{1..3}$ associated with question $Q$ given dialogue $D$. In this paper, we regard this task as a three-class classification problem, each class corresponding to an answer option.

For convenience, we define the following notations, which will be referred in the rest of this paper. Let $D^{s}$ denote the turns spoken by speaker $s$ in $D$. Formally, $D^{s}=\{s_{i_{1}}\colon t_{i_{1}},s_{i_{2}}\colon t_{i_{2}},\ldots,s_{i_{m}}\colon t_{i_{m}}\}$ where $\{i_{1},i_{2},\ldots,i_{m}\}=\{i\,|\,s_{i}=s\}$ and $i_{1}<i_{2}<\ldots<i_{m}$. In particular, $s=*$ denotes all the speakers. $W^{D^{s}}$ and $W^{O_{i}}$ denote the ordered set of the running words (excluding punctuation marks) in $D^{s}$ and $O_{i}$ respectively. Questions designed for dialogue-based reading comprehension often focus on a particular speaker. If there is exactly one speaker mentioned in a question, we use $s_{Q}$ to denote this target speaker. Otherwise, $s_{Q}=*$. For example, given the dialogue in Table 3, $s_{Q}=$“M” for Question $1$ and $2$, and $s_{Q}=*$ for Question $3$.

2 Rule-Based Approaches

We first attempt to incorporate dialogue structure information into sliding window (SW), a rule-based approach developed by Richardson et al. (2013). This approach matches a bag of words constructed from a question $Q$ and one of its answer option $O_{i}$ with a given document, and calculates the TF-IDF style matching score for each answer option.

Let $\hat{D}^{s}$, $\hat{Q}$, and $\hat{O}_{i}$ be the unordered set of distinct words (excluding punctuation marks) in $D^{s}$, $Q$, and $O_{i}$ respectively. Instead of only regarding dialogue $D$ as a non-conversational text snippet, we also pay special attention to the context that is relevant to the target speaker mentioned in the question. Therefore, given a target speaker $s_{Q}$, we propose to compute a speaker-focused sliding window score for each answer option $O_{i}$, by matching a bag of words constructed from $Q$ and $O_{i}$ with $D^{s_{Q}}$ (i.e., turns spoken by $s_{Q}$). Given speaker $s$, we formally define the sliding window score $sw$ of $O_{i}$ as:

where $\text{ic}^{s}(w)=\log\left(1+\frac{1}{\sum_{i}\mathds{1}(W_{i}^{D^{s}}=w)}\right)$, $T_{i}=\hat{O}_{i}\cup\hat{Q}$, and $W_{i}^{D^{s}}$ denotes the $i$-th word in $W^{D^{s}}$. Based on the above definitions, we can regard $sw_{i}^{*}$ as the general score defined in the original sliding window approach, and $sw_{i}^{s_{Q}}$ represents the speaker-focused sliding window score considering the target speaker $s_{Q}$.

Since sliding window score ignores long-range dependencies, Richardson et al. (2013) introduce a distance-based variation (DSW), in which a word-distance based score is subtracted from the sliding window score to arrive at the final score. Similarly, we calculate the speaker-focused distance-based score given a ($Q$, $O_{i}$) pair and $s_{Q}$, by counting the distance between the occurrence of a word in $Q$ and a word in $O_{i}$ in $D^{s_{Q}}$. More formally, given speaker $s$ and a set of stop wordsWe use the list of stop words from NLTK Bird and Loper (2004). $U$, the distance-based score $d$ of $O_{i}$ is defined as

where $I_{Q}^{s}=(\hat{Q}\cap\hat{D}^{s})-U$, $I_{O_{i}}^{s}=(\hat{O}_{i}\cap\hat{D}^{s})-\hat{Q}-U$, and $\delta_{i}^{s}$ is the minimum number of words between an occurrence of a question word and an answer option word in $W^{D^{s}}$, plus one. The formal definition of $\delta_{i}^{s}$ is as follows.

Based on the above definitions, we can regard $d_{i}^{*}$ as the distance-based score defined in the original sliding window approach, and $d_{i}^{s_{Q}}$ represents the speaker-focused distance-based score considering speaker $s_{Q}$. In addition, the final distance-based sliding window score of $O_{i}$ Richardson et al. (2013) can be formulated as

Compared to (4) that only focuses on the general (or speaker-independent) information (i.e., $sw_{i}^{*}$ and $d_{i}^{*}$), we can capture general and speaker-focused information (i.e. $sw_{i}^{s_{Q}}$ and $d_{i}^{s_{Q}}$) simultaneously by averaging them:

Since a large percentage of questions cannot be solved by word-level matching, we also attempt to incorporate general world knowledge into our rule-based method. We calculate $cs_{i}^{s}$, the maximum cosine similarity between $O_{i}$ and consecutive words of the same length in $W^{D^{s}}$, as:

where $\overline{x}$ is obtained by averaging the embeddings of the constituent words in $x$. Here we use ConceptNet embeddings Speer et al. (2017) since they leverage the knowledge graph that focuses on general world knowledge. Following (5), we capture both general and speaker-focused semantic information within a dialogue as follows.

To make the final answer option selection, our rule-based method combines (5) and (7):

3 Feature-Based Classifier

To explore what features are effective for dialogue understanding, we first consider a gradient boosting decision tree (GBDT) classifier. Besides the conventional bag-of-words based features, we primarily focus on features related to general world knowledge and dialogue structure.

Features inspired by rule-based approaches: we adopt the features introduced in Section 4.2, including speaker-independent scores (i.e., $sw_{i}^{*}$ and $d_{i}^{*}$) and speaker-focused scores (i.e., $sw_{i}^{s_{Q}}$ and $d_{i}^{s_{Q}}$).

Matching position: $p_{1..3}^{s_{Q}}$ and $p_{1..3}^{*}$, where $p_{i}^{s}$ is the last position (in percentage) of a word in $D^{s}$ that is also mentioned in $O_{i}$; if none of the words in $D^{s}$ is mentioned in $O_{i}$. We consider matching position due to our observation of the existence of concessions and negotiations in dialogues Amgoud et al. (2007). We assume the facts or opinions expressed near the end of a dialogue tend to be more critical for us to answer a question.

Pointwise mutual information (PMI): $pmi_{\max,1..3}^{s_{Q}}$, $pmi_{\max,1..3}^{*}$, $pmi_{\min,1..3}^{s_{Q}}$, $pmi_{\min,1..3}^{*}$, $pmi_{\text{avg},1..3}^{s_{Q}}$, and $pmi_{\text{avg},1..3}^{*}$, where $pmi_{f,i}^{s}$ is defined as

$C_{1}(w)$ denotes the word frequency of $w$ in external copora (we use Reddit posts Tan and Lee (2015)), and $C_{2}(w_{1},w_{2})$ represents the co-occurrence frequency of word $w_{1}$ and $w_{2}$ within a distance $<K$ in external copora. We use PMI to evaluate the relatedness between the content of an answer option and the target-speaker-focused context based on co-occurrences of words in external corpora, inspired by previous studies on narrative event chains Chambers and Jurafsky (2008).

ConceptNet relations (CR): $cr_{1..3,1..|R|}$. $R=\{r_{1},r_{2},\ldots\}$ is the set of ConceptNet relation types (e.g., “CapableOf” and “PartOf”). $cr_{i,j}$ is the number of relation triples ($w_{1}$, $r_{j}$, $w_{2}$) that appear in the ConceptNet Speer et al. (2017), where $w_{1}$ represents a word in answer option $O_{i}$, $w_{2}$ represents a word in $D$, and the relation type $r_{j}\in R$. Similar to the motivation of using PMI, we use CR to capture the association between an answer option and the source dialogue based on raw co-occurrence counts in the commonsense knowledge base.

ConceptNet embeddings (CE): besides the lexical similarity based on string matching, we also calculate $cs_{1..3}^{*}$ and $cs_{1..3}^{s_{Q}}$, where $cs_{i}^{*}$ and $cs_{i}^{s_{Q}}$ represent the maximum cosine similarity between $O_{i}$ and consecutive words of the same length in $D$ and $D^{s_{Q}}$ respectively (Expression 6 in Section 4.2). We use ConceptNet embeddings Speer et al. (2017) since they leverage the general world knowledge graph.

4 End-To-End Neural Network

Our end-to-end neural model is based on a generative pre-trained language model (LM). We follow the framework of finetuned transformer LM (FTLM) Radford et al. (2018) and make modifications for dialogue-based reading comprehension.

The training procedure of FTLM consists of two stages. The first stage is to learn a high-capacity language model on a large-scale unsupervised corpus of tokens $\mathcal{U}=\{u_{1},\ldots,u_{n}\}$ by maximizing the following likelihood:

where $k$ is the context window size, and the conditional probability $P$ is modeled by a multi-layer transformer decoder Liu et al. (2018) with parameters $\Theta$. In the second stage, the model is adapted to a labeled dataset $\mathcal{C}$, where each instance consists of a sequence of input tokens $x^{1},\ldots,x^{m}$ with a label $y$, by maximizing:

where $P(y\,|\,x^{1},\ldots,x^{m})$ is obtained by a linear $+$ softmax layer over the final transformer block’s activation, and $\lambda$ is the weight for language model. For multiple-choice reading comprehension, the input tokens $x^{1},\ldots,x^{m}$ come from the concatenation of a start token, dialogue, question, a delimiter token, answer option, and an end token; $y$ indicates if the answer option is correct. We refer readers to Radford et al. (2018) for more details.

Since the original FTLM framework already leverages rich linguistic information from a large unlabeled corpus, which can be regarded as a type of tacit general world knowledge, we investigate whether additional dialogue structure can further improve this strong baseline. We propose speaker embedding to better capture dialogue structure. Specifically, in the original framework, given an input context $(u_{-k},\ldots,u_{-1})$ of the transformer, the encoding of $u_{-i}$ is $\bm{we}(u_{-i})+\bm{pe}(i)$, where $\bm{we}(\cdot)$ is the word embedding, and $\bm{pe}(\cdot)$ is the position embedding. When adapting $\Theta$ to DREAM, we change the encoding to $\bm{we}(u_{-i})+\bm{pe}(i)+\bm{se}(u_{-i},s_{Q})$ where the speaker embedding $\bm{se}(u_{-i},s_{Q})$ is (a) $\bm{0}$ if the token $u_{-i}$ is not in the dialogue (i.e. it is either a start/end/delimiter token or a token in the question/option); (b) $\bm{e}_{target}$ if the token is spoken by $s_{Q}$; (c) $\bm{e}_{rest}$ if the token is in the dialogue but not spoken by $s_{Q}$. $\bm{e}_{target}$ and $\bm{e}_{rest}$ are trainable and initialized randomly. We show the overall framework in Figure 1.

5 Preprocessing and Training Details

For all the models, we conduct coreference resolution to determine speaker mentions of $s_{Q}$ based on simple heuristics. Particularly, we map three most common speaker abbreviations (i.e., “M”; “W” and “F”) that appear in dialogues to their eight most common corresponding mentions (i.e., “man”, “boy”, “he”, and “his”; “woman”, “girl”, “she”, and “her”) in questions. We keep speaker abbreviations unchanged, since neither replacing them with their corresponding full forms nor removing them contributes to the performance based on our experiments.

For the neural model mentioned in Section 4.4, most of our parameter settings follow Radford et al. (2018). We adopt the same preprocessing procedure and use their publicly released pre-trained language model, which is trained on the BooksCorpus dataset Zhu et al. (2015). We set the batch size to $8$, language model weight $\lambda$ to $2$, and maximum epochs of training to $10$.

For other models, we use the following preprocessing steps. We tokenize and lowercase the corpus, convert number words to numeric digits, normalize time expressions to $24$-hour numeric form, and deal with negation by removing interrogative sentences that receive “no” as the reply. We use the gradient boosting classifier implemented in the scikit-learn toolkit Pedregosa et al. (2011). We set the number of boosting iterations to $600$ and keep the rest of hyperparameters unchanged. The distance upper bound $K$ for PMI is set to $10$.

We perform several runs of machine learning models (Section 4.3 and Section 4.4) with randomness introduced by different random seeds and/or GPU nondeterminism and select the model or models (for ensemble) that perform best on the development set.

Experiment

We implement several baselines, including rule-based methods and state-of-the-art neural models.

Word Matching This strong baseline Yih et al. (2013) selects the answer option that has the highest count of overlapping words with the given dialogue.

Sliding Window We implement the sliding window approach (i.e., $\arg\max_{i}sw_{i}^{*}$) and its distance-based variation (i.e., $\arg\max_{i}sw_{i}^{*}-d_{i}^{*}$) Richardson et al. (2013) introduced in Section 4.2.

Enhanced Distance-Based Sliding Window (DSW$++$) We also use general world knowledge and speaker-focused information to improve the original sliding window baseline, formulated in Expression 8 (Section 4.2).

Stanford Attentive Reader This neural baseline compares each candidate answer (i.e., entity) representation to the question-aware document representation built with attention mechanism Hermann et al. (2015); Chen et al. (2016). Lai et al. (2017) adds a bilinear operation to compare document and answer option representations to answer multiple-choice questions.

Gated-Attention Reader The baseline models multiplicative question-specific document representations based on a gated-attention mechanism Dhingra et al. (2017), which are then compared to each answer option Lai et al. (2017).

Co-Matching This state-of-the-art multiple-choice reading comprehension model explicitly treats question and answer option as two sequences and jointly matches them against a given document Wang et al. (2018b).

Finetuned Transformer LM This is a general task-agnostic model introduced in Section 4.4, which achieves the best reported performance on several tasks requiring multi-sentence reasoning Radford et al. (2018).

We do not investigate other ways of leveraging pre-trained deep models such as adding ELMo representations Peters et al. (2018) as additional features to a neural model since recent studies show that directly fine-tuning a pre-trained language model such as FTLM is significantly superior on multiple-choice reading comprehension tasks Radford et al. (2018); Chen et al. (2019). We do not apply more recent extractive models such as AOA Cui et al. (2017) and QANet Yu et al. (2018) since they aim at precisely locating a span in a document. When adapted to solve questions with abstractive answer options, extractive models generally tend to perform less well Chen et al. (2016); Dhingra et al. (2017); Lai et al. (2017).

2 Results and Analysis

We report the performance of the baselines introduced in Section 5.1 and our proposed approaches in Table 8. We report the averaged accuracy of two annotators as the human performance. The proportion of valid questions (i.e., an unambiguous question with a unique correct answer option provided) that are manually checked by annotators on the annotated test and development sets is regarded as the human ceiling performance.

Surface matching is insufficient. Experimental results show that neural models that primarily exploit surface-level information (i.e., SAR, GAR, and CO) attain a performance level close to that of simple rule-based approaches (i.e., WM, SW, and DSW). The highest accuracy achieved by CO is $45.5\%$, a similar level of performance to the rule-based method DSW ($44.6\%$).

It is helpful to incorporate general world knowledge and dialogue structure. We see a significant gain $5.5\%$ in accuracy when enhancing DSW using general world knowledge from ConceptNet embeddings and considering speaker-focused information (Section 4.2). FTLM, which leverages rich external linguistic knowledge from thousands of books, already achieves a much higher accuracy $55.5\%$ compared to previous state-of-the-art machine comprehension models, indicating the effectiveness of general world knowledge. Experimental results show that our best single model FTLM$++$ significantly outperforms FTLM (p-value $=0.03$), illustrating the usefulness of additional dialogue structure. Compared to the state-of-the-art neural reader Co-Matching that primarily explore surface-level information ($45.5\%$), the tacit general world knowledge (in the pre-trained language model) and dialogues structure in FTLM$++$ lead to an absolute gain of $11.9\%$ in accuracy.

Ensembling different types of methods can bring further improvements. We employ the majority vote strategy to obtain the ensemble model performance. While GBDT$++$ ($52.8\%$) itself does not outperform FTLM$++$, GBDT$++$ can serve as a supplement to FTLM$++$ as they have diverse types of general world knowledge and model architectures. We achieve the highest accuracy $59.5\%$ by ensembling one GBDT$++$ and three FTLM$++$.

3 Ablation Tests

We conduct ablation tests to evaluate the individual components of our proposed approaches (Table 10). In Table 9, we summarize the involved types of dialogue structure and general world knowledge in our approaches.

Dialogue Structure Specifically, we observe $1.4\%$ drop in accuracy if we set the target speaker $s_{Q}$ to $*$ for all questions when we apply DSW$++$. We observe a similar performance drop when we remove speaker-focused features from GBDT$++$. In addition, removing speaker embeddings from FTLM$++$ leads to $1.7\%$ drop in accuracy (in this case, the model becomes the original FTLM). These results consistently indicate the usefulness of dialogue structure for dialogue understanding.

General World Knowledge We also investigate the effects of general world knowledge. The accuracy of DSW$++$ drops by $4.7\%$ if we remove ConceptNet embeddings (CE) by deleting the last term of Expression 8 in Section 4.2. Additionally, the accuracy of GBDT$++$ drops by $6.2\%$ if we remove all the general world knowledge features (i.e., ConceptNet embeddings/relations and PMI), leading to prediction failures on questions such as “What do we learn about the man?” whose correct answer option “He is health-conscious.” is not explicitly mentioned in the source dialogue “M: We had better start to eat onions frequently, Linda. W: But you hate onions, don’t you? M: Until I learned from a report from today’s paper that they protect people from flu and colds. After all, compared with health, taste is not so important.”. Moreover, if we train FTLM$++$ with randomly initialized transformer weights instead of weights pre-trained on the external corpus, the accuracy drops dramatically to $36.2\%$, which is only slightly better than a random baseline.

4 Error Analysis

Impact of Longer Turns The number of dialogue turns has a significant impact on the performance of FTLM$++$. As shown in Figure 2, its performance reaches the peak while the number of turns ranges from to $10$ while it suffers severe performance drops when the given dialogue contains more turns. Both DSW$++$ ($56.8\%$) and GBDT$++$ ($57.4\%$) outperform FTLM$++$ ($55.7\%$) when the number of turns ranges from $10$ to $48$. To deal with lengthy context, it may be helpful to first identify relevant sentences based on a question and its associated answer options rather than using the entire dialogue context as input.

Impact of Confusing Distractors For $54.5\%$ of questions on the development set, the fuzzy match score Sikes (2007) of at least one distractor answer option against the dialogue is higher than the score of the correct answer option. For questions that all models (i.e., DSW$++$, GBDT$++$, and FTLM$++$) fail to answer correctly, $73.0\%$ of them contain at least one such confusing distractor answer option. The causes of this kind of errors can be roughly divided into two categories. First, the distractor is wrongly associated with the target speaker/s mentioned in the question (e.g., answer option A and C in D$2$-Q$3$ in Table 3). Second, although the claim in the distractor is supported by the dialogue, it is irrelevant to the question (e.g., D$1$-Q$1$-B in Table 1). A promising direction to solve this problem could be the construction of speaker-focused event chains Chambers and Jurafsky (2008) and dialogue-specific coreference resolution systems for more reliable evidence collection in a dialogue.

Impact of Question Types We further report the performance of the best single model FTLM$++$ and the GBDT$++$ baseline on the categories defined in Section 3.2 (Table 11). Not surprisingly, both models perform worse than random guessing on math problems. While most of the problems can be solved by one single linear equation, it is still difficult to apply recent neural math word problem solvers Huang et al. (2018); Wang et al. (2018a) due to informal dialogue-based problem descriptions and the requirement of commonsense inference. For example, given the dialogue: “W: The plane arrives at 10:50. It is already 10:40 now. Be quick! M: Relax. Your watch must be fast. There are still twenty minutes left.”, we need prior knowledge to infer that the watch of the man is showing incorrect time $10$:$40$. Instead, $10$:$50$ should be used as the reference time with the time interval “twenty minutes left” together to answer the question “What time is it now?”.

Results show that GBDT$++$ is superior to the fine-tuned language model on the questions under the category matching ($68.1\%$ vs. $57.0\%$) and the latter model is more capable of answering implicit questions (e.g., under the category summary, logic, and commonsense) which require aggregation of information from multiple sentences, the understanding of the entire dialogue, or the utilization of world knowledge. Therefore, it might be useful to leverage the strengths of individual models to solve different types of questions.

Conclusion and Future Work

We present DREAM, the first multiple-choice dialogue-based reading comprehension dataset from English language examinations. Besides the multi-turn multi-party dialogue context, $85\%$ of questions require multiple-sentence reasoning, and $34\%$ of questions also require commonsense knowledge, making this task very challenging. We apply several popular reading comprehension models and find that surface-level information is insufficient. We incorporate general world knowledge and dialogue structure into rule-based and machine learning methods and show the effectiveness of these factors, suggesting a promising direction for dialogue-based reading comprehension. For future work, we are interested in problem generation for dialogues and investigating whether it will lead to more gains to pre-train a deep language model such as FTLM over large-scale dialogues from movies and TV shows instead of the BookCorpus dataset Zhu et al. (2015) used by previous work Radford et al. (2018).

Acknowledgments

We would like to thank the editors and anonymous reviewers for their helpful feedback. We also thank Hai Wang from Technological Institute at Chicago for useful discussions and valuable comments.

References