Learning to Ask: Neural Question Generation for Reading Comprehension

Introduction

Question generation (QG) aims to create natural questions from a given a sentence or paragraph. One key application of question generation is in the area of education — to generate questions for reading comprehension materials Heilman and Smith (2010). Figure 1, for example, shows three manually generated questions that test a user’s understanding of the associated text passage. Question generation systems can also be deployed as chatbot components (e.g., asking questions to start a conversation or to request feedback Mostafazadeh et al. (2016)) or, arguably, as a clinical tool for evaluating or improving mental health Weizenbaum (1966); Colby et al. (1971).

In addition to the above applications, question generation systems can aid in the development of annotated data sets for natural language processing (NLP) research in reading comprehension and question answering. Indeed the creation of such datasets, e.g., SQuAD Rajpurkar et al. (2016) and MS MARCO Nguyen et al. (2016), has spurred research in these areas.

For the most part, question generation has been tackled in the past via rule-based approaches (e.g., Mitkov and Ha (2003); Rus et al. (2010). The success of these approaches hinges critically on the existence of well-designed rules for declarative-to-interrogative sentence transformation, typically based on deep linguistic knowledge.

To improve over a purely rule-based system, Heilman and Smith (2010) introduced an overgenerate-and-rank approach that generates multiple questions from an input sentence using a rule-based approach and then ranks them using a supervised learning-based ranker. Although the ranking algorithm helps to produce more acceptable questions, it relies heavily on a manually crafted feature set, and the questions generated often overlap word for word with the tokens in the input sentence, making them very easy to answer.

Vanderwende (2008) point out that learning to ask good questions is an important task in NLP research in its own right, and should consist of more than the syntactic transformation of a declarative sentence. In particular, a natural sounding question often compresses the sentence on which it is based (e.g., question 3 in Figure 1), uses synonyms for terms in the passage (e.g., “form” for “produce” in question 2 and “get” for “produce” in question 3), or refers to entities from preceding sentences or clauses (e.g., the use of “photosynthesis” in question 2). Othertimes, world knowledge is employed to produce a good question (e.g., identifying “photosynthesis” as a “life process” in question 1). In short, constructing natural questions of reasonable difficulty would seem to require an abstractive approach that can produce fluent phrasings that do not exactly match the text from which they were drawn.

As a result, and in contrast to all previous work, we propose here to frame the task of question generation as a sequence-to-sequence learning problem that directly maps a sentence from a text passage to a question. Importantly, our approach is fully data-driven in that it requires no manually generated rules.

More specifically, inspired by the recent success in neural machine translation Sutskever et al. (2014); Bahdanau et al. (2015), summarization Rush et al. (2015); Iyer et al. (2016), and image caption generation Xu et al. (2015), we tackle question generation using a conditional neural language model with a global attention mechanism Luong et al. (2015a). We investigate several variations of this model, including one that takes into account paragraph- rather than sentence-level information from the reading passage as well as other variations that determine the importance of pre-trained vs. learned word embeddings.

In evaluations on the SQuAD dataset Rajpurkar et al. (2016) using three automatic evaluation metrics, we find that our system significantly outperforms a collection of strong baselines, including an information retrieval-based system Robertson and Walker (1994), a statistical machine translation approach Koehn et al. (2007), and the overgenerate-and-rank approach of Heilman and Smith (2010). Human evaluations also rated our generated questions as more grammatical, fluent, and challenging (in terms of syntactic divergence from the original reading passage and reasoning needed to answer) than the state-of-the-art Heilman and Smith (2010) system.

In the sections below we discuss related work (Section 2), specify the task definition (Section 3) and describe our neural sequence learning based models (Section 4). We explain the experimental setup in Section 5. Lastly, we present the evaluation results as well as a detailed analysis.

Related Work

Reading Comprehension is a challenging task for machines, requiring both understanding of natural language and knowledge of the world Rajpurkar et al. (2016). Recently many new datasets have been released and in most of these datasets, the questions are generated in a synthetic way. For example, bAbI Weston et al. (2016) is a fully synthetic dataset featuring 20 different tasks. Hermann et al. (2015) released a corpus of cloze style questions by replacing entities with placeholders in abstractive summaries of CNN/Daily Mail news articles. Chen et al. (2016) claim that the CNN/Daily Mail dataset is easier than previously thought, and their system almost reaches the ceiling performance. Richardson et al. (2013) curated MCTest, in which crowdworker questions are paired with four answer choices. Although MCTest contains challenging natural questions, it is too small for training data-demanding question answering models.

Recently, Rajpurkar et al. (2016) released the Stanford Question Answering Datasethttps://stanford-qa.com (SQuAD), which overcomes the aforementioned small size and (semi-)synthetic issues. The questions are posed by crowd workers and are of relatively high quality. We use SQuAD in our work, and similarly, we focus on the generation of natural questions for reading comprehension materials, albeit via automatic means.

Question Generation has attracted the attention of the natural language generation (NLG) community in recent years, since the work of Rus et al. (2010).

Most work tackles the task with a rule-based approach. Generally, they first transform the input sentence into its syntactic representation, which they then use to generate an interrogative sentence. A lot of research has focused on first manually constructing question templates, and then applying them to generate questions Mostow and Chen (2009); Lindberg et al. (2013); Mazidi and Nielsen (2014). Labutov et al. (2015) use crowdsourcing to collect a set of templates and then rank the relevant templates for the text of another domain. Generally, the rule-based approaches make use of the syntactic roles of words, but not their semantic roles.

Heilman and Smith (2010) introduce an overgenerate-and-rank approach: their system first overgenerates questions and then ranks them. Although they incorporate learning to rank, their system’s performance still depends critically on the manually constructed generating rules. Mostafazadeh et al. (2016) introduce visual question generation task, to explore the deep connection between language and vision. Serban et al. (2016) propose generating simple factoid questions from logic triple (subject, relation, object). Their task tackles mapping from structured representation to natural language text, and their generated questions are consistent in terms of format and diverge much less than ours.

To our knowledge, none of the previous works has framed QG for reading comprehension in an end-to-end fashion, and nor have them used deep sequence-to-sequence learning approach to generate questions.

Task Definition

In this section, we define the question generation task. Given an input sentence $\mathbf{x}$, our goal is to generate a natural question $\mathbf{y}$ related to information in the sentence, $\mathbf{y}$ can be a sequence of an arbitrary length: $[y_{1},...,y_{|\mathbf{y}|}]$. Suppose the length of the input sentence is $M$, $\mathbf{x}$ could then be represented as a sequence of tokens $[x_{1},...,x_{M}]$. The QG task is defined as finding $\mathbf{\overline{y}}$, such that:

where $P\left(\mathbf{y}|\mathbf{x}\right)$ is the conditional log-likelihood of the predicted question sequence $\mathbf{y}$, given the input $\mathbf{x}$. In section 4.1, we will elaborate on the global attention mechanism for modeling $P\left(\mathbf{y}|\mathbf{x}\right)$.

Model

Our model is partially inspired by the way in which a human would solve the task. To ask a natural question, people usually pay attention to certain parts of the input sentence, as well as associating context information from the paragraph. We model the conditional probability using RNN encoder-decoder architecture Bahdanau et al. (2015); Cho et al. (2014), and adopt the global attention mechanism Luong et al. (2015a) to make the model focus on certain elements of the input when generating each word during decoding.

Here, we investigate two variations of our models: one that only encodes the sentence and another that encodes both sentence and paragraph-level information.

Similar to Sutskever et al. (2014) and Chopra et al. (2016), we factorize the the conditional in equation 1 into a product of word-level predictions:

where probability of each $y_{t}$ is predicted based on all the words that are generated previously (i.e., $y_{<t}$), and input sentence $\mathbf{x}$.

with $\mathbf{h}_{t}$ being the recurrent neural networks state variable at time step $t$, and $\mathbf{c}_{t}$ being the attention-based encoding of $\mathbf{x}$ at decoding time step $t$ (Section 4.2). $\mathbf{W}_{s}$ and $\mathbf{W}_{t}$ are parameters to be learned.

here, LSTM is the Long Short-Term Memory (LSTM) network Hochreiter and Schmidhuber (1997). It generates the new state $\mathbf{h}_{t}$, given the representation of previously generated word $y_{t-1}$ (obtained from a word look-up table), and the previous state $\mathbf{h}_{t-1}$.

The initialization of the decoder’s hidden state differentiates our basic model and the model that incorporates paragraph-level information.

For the basic model, it is initialized by the sentence’s representation $\mathbf{s}$ obtained from the sentence encoder (Section 4.2). For our paragraph-level model, the concatenation of the sentence encoder’s output $\mathbf{s}$ and the paragraph encoder’s output $\mathbf{s}^{\prime}$ is used as the initialization of decoder hidden state. To be more specific, the architecture of our paragraph-level model is like a “Y”-shaped network which encodes both sentence- and paragraph-level information via two RNN branches and uses the concatenated representation for decoding the questions.

2 Encoder

The attention-based sentence encoder is used in both of our models, while the paragraph encoder is only used in the model that incorporates paragraph-level information.

We use a bidirectional LSTM to encode the sentence,

where $\overrightarrow{\mathbf{b}_{t}}$ is the hidden state at time step $t$ for the forward pass LSTM, $\overleftarrow{\mathbf{b}_{t}}$ for the backward pass.

To get attention-based encoding of $\mathbf{x}$ at decoding time step $t$, namely, $\mathbf{c}_{t}$, we first get the context dependent token representation by $\mathbf{b}_{t}=[\overrightarrow{\mathbf{b}_{t}};\overleftarrow{\mathbf{b}_{t}}]$, then we take the weighted average over $\mathbf{b}_{t}$ ($t=1,...,|\mathbf{x}|)$,

The attention weight are calculated by the bi-linear scoring function and softmax normalization,

To get the sentence encoder’s output for initialization of decoder hidden state, we concatenate last hidden state of the forward and backward pass, namely, $\mathbf{s}=[\overrightarrow{\mathbf{b}_{|\mathbf{x}|}};\overleftarrow{\mathbf{b}_{1}}]$.

Given sentence $\mathbf{x}$, we want to encode the paragraph containing $\mathbf{x}$. Since in practice the paragraph is very long, we set a length threshold $L$, and truncate the paragraph at the $L$th token. We call the truncated paragraph “paragraph” henceforth.

Denoting the paragraph as $\mathbf{z}$, we use another bidirectional LSTM to encode $\mathbf{z}$,

With the last hidden state of the forward and backward pass, we use the concatenation $[\overrightarrow{\mathbf{d}_{|\mathbf{z}|}};\overleftarrow{\mathbf{d}_{1}}]$ as the paragraph encoder’s output $\mathbf{s}^{\prime}$.

3 Training and Inference

Giving a training corpus of sentence-question pairs: $\mathcal{S}=\left\{\left(\mathbf{x}^{(i)},\mathbf{y}^{(i)}\right)\right\}_{i=1}^{S}$, our models’ training objective is to minimize the negative log-likelihood of the training data with respect to all the parameters, as denoted by $\theta$,

Once the model is trained, we do inference using beam search. The beam search is parametrized by the possible paths number $k$.

As there could be many rare words in the input sentence that are not in the target side dictionary, during decoding many UNK tokens will be output. Thus, post-processing with the replacement of UNK is necessary. Unlike Luong et al. (2015b), we use a simpler replacing strategy for our task. For the decoded UNK token at time step $t$, we replace it with the token in the input sentence with the highest attention score, the index of which is $\operatorname*{arg\,max}_{i}a_{i,t}$.

Experimental Setup

We experiment with our neural question generation model on the processed SQuAD dataset. In this section, we firstly describe the corpus of the task. We then give implementation details of our neural generation model, the baselines to compare, and their experimental settings. Lastly, we introduce the evaluation methods by automatic metrics and human raters.

With the SQuAD dataset Rajpurkar et al. (2016), we extract sentences and pair them with the questions. We train our models with the sentence-question pairs. The dataset contains 536 articles with over 100k questions posed about the articles. The authors employ Amazon Mechanical Turks crowd-workers to create questions based on the Wikipedia articles. Workers are encouraged to use their own words without any copying phrases from the paragraph. Later, other crowd-workers are employed to provide answers to the questions. The answers are spans of tokens in the passage.

Since there is a hidden part of the original SQuAD that we do not have access to, we treat the accessible parts ($\sim$90%) as the entire dataset henceforth.

We first run Stanford CoreNLP Manning et al. (2014) for pre-processing: tokenization and sentence splitting. We then lower-case the entire dataset. With the offset of the answer to each question, we locate the sentence containing the answer and use it as the input sentence. In some cases (< 0.17% in training set), the answer spans two or more sentences, and we then use the concatenation of the sentences as the input “sentence”.

Figure 2 shows the distribution of the token overlap percentage of the sentence-question pairs. Although most of the pairs have over 50% overlap rate, about 6.67% of the pairs have no non-stop-words in common, and this is mostly because of the answer offset error introduced during annotation. Therefore, we prune the training set based on the constraint: the sentence-question pair must have at least one non-stop-word in common. Lastly we add to the beginning of the sentences, and to the end of them.

We randomly divide the dataset at the article-level into a training set (80%), a development set (10%), and a test set (10%). We report results on the 10% test set.

Table 1 provides some statistics on the processed dataset: there are around 70k training samples, the sentences are around 30 tokens, and the questions are around 10 tokens on average. For each sentence, there might be multiple corresponding questions, and, on average, there are 1.4 questions for each sentence.

2 Implementation Details

We implement our models The code is available at https://github.com/xinyadu/nqg. in Torch7 http://torch.ch/ on top of the newly released OpenNMT system Klein et al. (2017).

For the source side vocabulary $\mathcal{V}$, we only keep the 45k most frequent tokens (including , and placeholders). For the target side vocabulary $\mathcal{U}$, similarly, we keep the 28k most frequent tokens. All other tokens outside the vocabulary list are replaced by the UNK symbol. We choose word embedding of 300 dimensions and use the glove.840B.300d pre-trained embeddings Pennington et al. (2014) for initialization. We fix the word representations during training.

We set the LSTM hidden unit size to 600 and set the number of layers of LSTMs to 2 in both the encoder and the decoder. Optimization is performed using stochastic gradient descent (SGD), with an initial learning rate of 1.0. We start halving the learning rate at epoch 8. The mini-batch size for the update is set at 64. Dropout with probability 0.3 is applied between vertical LSTM stacks. We clip the gradient when the its norm exceeds 5.

All our models are trained on a single GPU. We run the training for up to 15 epochs, which takes approximately 2 hours. We select the model that achieves the lowest perplexity on the dev set.

During decoding, we do beam search with a beam size of 3. Decoding stops when every beam in the stack generates the token.

All hyperparameters of our model are tuned using the development set. The results are reported on the test set.

3 Baselines

To prove the effectiveness of our system, we compare it to several competitive systems. Next, we briefly introduce their approaches and the experimental setting to run them for our problem. Their results are shown in Table 2.

IR stands for our information retrieval baselines. Similar to Rush et al. (2015), we implement the IR baselines to control memorizing questions from the training set. We use two metrics to calculate the distance between a question and the input sentence, i.e., BM-25 Robertson and Walker (1994) and edit distance Levenshtein (1966). According to the metric, the system retrieves the training set to find the question with the highest score.

MOSES+ Koehn et al. (2007) is a widely used phrase-based statistical machine translation system. Here, we treat sentences as source language text, we treat questions as target language text, and we perform the translation from sentences to questions. We train a tri-gram language model on target side texts with KenLM Heafield et al. (2013), and tune the system with MERT on dev set. Performance results are reported on the test set.

DirectIn is an intuitive yet meaningful baseline in which the longest sub-sentence of the sentence is directly taken as the predicted question. We also tried using the entire input sentence as the prediction output, but the performance is worse than taking sub-sentence as the prediction, across all the automatic metrics except for METEOR. To split the sentence into sub-sentences, we use a set of splitters, i.e., {“?”, “!”, “,”, “.”, “;”}.

H&S is the rule-based overgenerate-and-rank system that was mentioned in Section 2. When running the system, we set the parameter just-wh true (to restrict the output of the system to being only wh-questions) and set max-length equal to the longest sentence in the training set. We also set downweight-pro true, to down weight questions with unresolved pronouns so that they appear towards the end of the ranked list. For comparison with our systems, we take the top question in the ranked list.

Seq2seq Sutskever et al. (2014) is a basic encoder-decoder sequence learning system for machine translation. We implement their model in Tensorflow. The input sequence is reversed before training or translating. Hyperparameters are tuned with dev set. We select the model with the lowest perplexity on the dev set.

4 Automatic Evaluation

We use the evaluation package released by Chen et al. (2015), which was originally used to score image captions. The package includes BLEU 1, BLEU 2, BLEU 3, BLEU 4 Papineni et al. (2002), METEOR Denkowski and Lavie (2014) and ROUGEL Lin (2004) evaluation scripts. BLEU measures the average $n$-gram precision on a set of reference sentences, with a penalty for overly short sentences. BLEU-$n$ is BLEU score that uses up to $n$-grams for counting co-occurrences. METEOR is a recall-oriented metric, which calculates the similarity between generations and references by considering synonyms, stemming and paraphrases. ROUGE is commonly employed to evaluate $n$-grams recall of the summaries with gold-standard sentences as references. ROUGEL (measured based on longest common subsequence) results are reported.

5 Human Evaluation

We also perform human evaluation studies to measure the quality of questions generated by our system and the H&S system. We consider two modalities: naturalness, which indicates the grammaticality and fluency; and difficulty, which measures the sentence-question syntactic divergence and the reasoning needed to answer the question. We randomly sampled 100 sentence-question pairs. We ask four professional English speakers to rate the pairs in terms of the modalities above on a 1–5 scale (5 for the best). We then ask the human raters to give a ranking of the questions according to the overall quality, with ties allowed.

Results and Analysis

Table 2 shows automatic metric evaluation results for our models and baselines. Our model which only encodes sentence-level information achieves the best performance across all metrics. We note that IR performs poorly, indicating that memorizing the training set is not enough for the task. The baseline DirectIn performs pretty well on BLEU and METEOR, which is reasonable given the overlap statistics between the sentences and the questions (Figure 2). H&S system’s performance is on a par with DirectIn’s, as it basically performs syntactic change without paraphrasing, and the overlap rate is also high.

Looking at the performance of our three models, it’s clear that adding the pre-trained embeddings generally helps. While encoding the paragraph causes the performance to drop a little, this makes sense because, apart from useful information, the paragraph also contains much noise.

Table 3 shows the results of the human evaluation. We see that our system outperforms H&S in all modalities. Our system is ranked best in 38.4% of the evaluations, with an average ranking of 1.94. An inter-rater agreement of Krippendorff’s Alpha of 0.236 is achieved for the overall ranking. The results imply that our model can generate questions of better quality than the H&S system. An interesting phenomenon here is that human raters gave higher score for our system’s outputs than the human questions. One potential explanation for this is that our system is trained on all sentence-question pairs for one input sentence, while we randomly select one question among the several questions of one sentence as the human generated question, for the purpose of rating. Thus our system’s predictions tend to be more diverse.

For our qualitative analysis, we examine the sample outputs and the visualization of the alignment between the input and the output. In Figure 3, we present sample questions generated by H&S and our best model. We see a large gap between our results and H&S’s. For example, in the first sample, in which the focus should be put on “the largest.” Our model successfully captures this information, while H&S only performs some syntactic transformation over the input without paraphrasing. However, outputs from our system are not always “perfect”, for example, in pair 6, our system generates a question about the reason why birds still grow, but the most related question would be why many species still grow. But from a different perspective, our question is more challenging (readers need to understand that birds are one kind of species), which supports our system’s performance listed in human evaluations (See Table 3). It would be interesting to further investigate how to interpret why certain irrelavant words are generated in the question. Figure 4 shows the attention weights ($\alpha_{i,t}$) for the input sentence when generating each token in the question. We see that the key words in the output (“introduced”, “teletext”, etc.) aligns well with those in the input sentence.

Finally, we do a dataset analysis and fine-grained system performance analysis. We randomly sampled 346 sentence-question pairs from the dev set and label each pair with a category. The IDs of the questions examined will be made available at https://github.com/xinyadu/nqg/blob/master/examined-question-ids.txt. The four categories are determined by how much information is needed to ask the question. To be specific, “w/ sentence” means it only requires the sentence to ask the question; “w/ paragraph” means it takes other information in the paragraph to ask the question; “w/ article” is similar to “w/ paragraph”; and “not askable” means that world knowledge is needed to ask the question or there is mismatch of sentence and question caused by annotation error.

Table 4 shows the per-category performance of the systems. Our model which encodes paragraph information achieves the best performance on the questions of “w/ paragraph” category. This verifies the effectiveness of our paragraph-level model on the questions concerning information outside the sentence.

Conclusion and Future Work

We have presented a fully data-driven neural networks approach to automatic question generation for reading comprehension. We use an attention-based neural networks approach for the task and investigate the effect of encoding sentence- vs. paragraph-level information. Our best model achieves state-of-the-art performance in both automatic evaluations and human evaluations.

Here we point out several interesting future research directions. Currently, our paragraph-level model does not achieve best performance across all categories of questions. We would like to explore how to better use the paragraph-level information to improve the performance of QG system regarding questions of all categories. Besides this, it would also be interesting to consider to incorporate mechanisms for other language generation tasks (e.g., copy mechanism for dialogue generation) in our model to further improve the quality of generated questions.

Acknowledgments

We thank the anonymous ACL reviewers, Kai Sun and Yao Cheng for their helpful suggestions. We thank Victoria Litvinova for her careful proofreading. We also thank Xanda Schofield, Wil Thomason, Hubert Lin and Junxian He for doing the human evaluations.

References