Improving Unsupervised Question Answering via Summarization-Informed Question Generation
Chenyang Lyu, Lifeng Shang, Yvette Graham, Jennifer Foster, Xin Jiang, Qun Liu
Introduction
The aim of Question Generation (QG) is the production of meaningful questions given a set of input passages and corresponding answers, a task with many applications including dialogue systems as well as education Graesser et al. (2005). Additionally, QG can be applied to Question Answering (QA) for the purpose of data augmentation Puri et al. (2020) where labeled
Early work on QG focused on template or rule-based approaches, employing syntactic knowledge to manipulate constituents in declarative sentences to form interrogatives Heilman and Smith (2009, 2010). Although template-based methods are capable of generating linguistically correct questions, the resulting questions often lack variety and incur high lexical overlap with corresponding declarative sentences. For example, the question generated from the sentence Stephen Hawking announced the party in the morning, with Stephen Hawking as the candidate answer span, could be Who announced the party in the morning?, with a high level of lexical overlap between the generated question and the declarative sentence. This is undesirable in a QA system Hong et al. (2020) since the strong lexical clues in the question would make it a poor test of real comprehension.
Neural seq2seq models Sutskever et al. (2014) have come to dominate QG Du et al. (2017), and are commonly trained with
We subsequently propose a new unsupervised approach that frames QG as a summarization-questioning process.
By employing freely available summary data, we firstly apply dependency parsing, named entity recognition and semantic role labeling to summaries, before applying a set of heuristics that generate questions based on parsed summaries. An end-to-end neural generation system is then trained employing the original news articles as input and the heuristically generated questions as target output.
An example is shown in Figure 1. The summary is used as a bridge between the questions and passages. Because the questions are generated from the summaries and not from the original passages, they have less of a lexical overlap with the passages. Crucially, however, they remain semantically close to the passages since the summaries by definition contain the most important information contained in the passages. A second advantage of this QG approach is that it does not rely on the existence of a QA dataset, and it is arguably easier to obtain summary data in a given language than equivalent QA data since summary data is created for many purposes (e.g. news, review and thesis summaries) whereas many QA datasets are created specifically for training a QA system.
In order to explore the effectiveness of our method, we carry out extensive experiments. We provide an extrinsic evaluation, and train an English QG model using news summary data. We employ our QG model to generate synthetic QA data to train a QA model in an unsupervised setting and test the approach with six English QA datasets: SQuAD1.1, Natural Questions, TriviaQA, NewsQA, BioASQ and DuoRC Rajpurkar et al. (2016); Kwiatkowski et al. (2019); Joshi et al. (2017); Trischler et al. (2017); Tsatsaronis et al. (2015); Saha et al. (2018). Experiment results show that our approach substantially improves over previous unsupervised QA models even when trained on substantially fewer synthetic QA examples.
Our contributions can be summarized as follows:
We propose a novel unsupervised QG approach that employs summary data and syntactic/semantic analysis, which to our best knowledge is the first work connecting text summarization and question generation in this way;
We employ our QG model to generate synthetic QA data achieving state-of-the-art performance even at low volumes of synthetic training data.
Related Work
Traditional approaches to QG mostly employ linguistic templates and rules to transform declarative sentences into interrogatives Heilman and Smith (2009). Recently, Dhole and Manning (2020) showed that, with the help of advanced neural syntactic parsers, template-based methods are capable of generating high-quality questions from texts.
Neural seq2seq generation models have additionally been widely employed in QG, with QG data usually borrowed from existing QA datasets Du et al. (2017); Sun et al. (2018); Ma et al. (2020). Furthermore, reinforcement learning has been employed by Zhang and Bansal (2019); Chen et al. (2019); Xie et al. (2020) to directly optimize discrete evaluation metrics such as BLEU Papineni et al. (2002). Lewis et al. (2020) and Song et al. (2019) show that a large-scale pre-trained model can achieve state-of-the-art performance for supervised QG Dong et al. (2019); Narayan et al. (2020).
BLEU Papineni et al. (2002), ROUGE Lin (2004) and Meteor Banerjee and Lavie (2005) metrics are commonly borrowed from text generation tasks to evaluate QG. Even with respect to original text generation tasks, however, the use of such metrics has been questioned Callison-Burch et al. (2006); Reiter (2018). Such metrics are particularly problematic for QG evaluation since multiple plausible questions exist for a given passage and answer. Consequently, there has been a shift in focus to evaluating QG using an extrinsic evaluation that generates synthetic QA pairs for the purpose of evaluating their effectiveness as a data augmentation or unsupervised QA approach Alberti et al. (2019); Puri et al. (2020); Shakeri et al. (2020).
In unsupervised QA, the QA model is trained using synthetic data based on a QG model instead of an existing QA dataset. Instead of resorting to existing QA datasets, unsupervised QG methods have been employed, such as Unsupervised Neural Machine Translation Lewis et al. (2019). Fabbri et al. (2020) and Li et al. (2020) propose template/rule-based methods for generating questions and employ retrieved paragraphs and cited passages as source passages to alleviate the problems of lexical similarities between passages and questions. Alberti et al. (2019); Puri et al. (2020); Shakeri et al. (2020) additionally employ existing QA datasets to train a QG model. Although related, this work falls outside the scope of unsupervised QA.
Methodology
Diverging from supervised neural question generation models trained on existing QA datasets, the approach we propose employs synthetic QG data, that we create from summary data using a number of heuristics, to train a QG model. We provide an overview of the proposed method is shown in Figure 2. We then employ the trained QG model to generate synthetic QA data that is further employed to train an unsupervised QA model.
In order to avoid generating trivial questions that are highly similar to corresponding declarative statements, we employ summary data as a bridge connecting the generated question and the original article.Data we employ in experiments is news summary data originally from BBC News Narayan et al. (2018) and the news articles are typically a few hundred words in length. The process we employ involves, firstly Dependency Parsing (DP) of summary sentences, followed by Named-Entity Recognition (NER) and finally Semantic Role Labeling (SRL). DP is firstly employed as a means of identifying the main verb (root verb), in addition to other constituents such as auxiliaries. NER is then responsible for tagging all entities in the summary sentence to facilitate discovery of the most appropriate question words to generate. The pivotal component of linguistic analysis is then SRL, employed to obtain all semantic frames for the summary sentence. Each frame consists of a verb followed by a set of arguments which correspond to phrases in the sentence. An argument could comprise, for example, an Agent (who initiates the action described by the verb), a Patient (who undergoes the action), and a set of modifier arguments such as a temporal ARG-TMP or locative argument ARG-LOC. Questions are then generated from the arguments according to argument type and NER tags, which means that wh-words can be determined jointly.
Returning to the example in Figure 1: given the SRL analysis [U2’s lead singer Bono ARG-0] has [had VERB] [emergency spinal surgery ARG-1] [after suffering an injury while preparing for tour dates ARG-TMP]., the three questions shown in Figure 1 can be generated based on these three arguments.
The pseudocode for our algorithm to generate questions is shown in Algorithm 1.
We first obtain all dependency edges and labels (), NER tags () and SRL frames () of a summary sentence. We then iterate through all arguments in the frame of the (the verb whose dependency label is ) and identify appropriate wh-words () for each argument using the function according to its argument type and the NER tags of entities in the argument. We follow Dhole and Manning (2020) to use the standard wh-words in English associated with appropriate argument types and NER tags. We then decompose the current main verb to its base form () and appropriate auxiliary words () in the function, before finally inserting the wh-words and the auxiliary verbs in appropriate positions using the . As can be seen from Figure 1, a single summary sentence generates multiple questions when its SRL frame has multiple arguments.
2 Training a Question Generation Model
The summarization data we employ consists of
To train the question generation model, we concatenate each passage and answer to form a sequence: passage
where is the i-th token in the question, and and are context and answer, respectively.
Experiments
We test our idea of using summaries in question generation by applying the questions generated by our QG system in unsupervised QA. We describe the details of our experiment setup, followed by our unsupervised QA results on six English benchmark extractive QA datasets.
We test the proposed method using news summary data from XSUM Narayan et al. (2018), crawled from BBC news website.www.bbc.com XSUM contains 226,711
We employ AllenNLPhttps://demo.allennlp.org/ Gardner et al. (2017) to obtain dependency trees, named entities and semantic role labels for summary sentences, before further employing this knowledge to generate questions from summaries following the algorithm described in Section 3.1. We remove any generated
Articles longer than 480 tokens (exceeding the maximum BART input length);
Articles in which fewer than 55% of tokens in the answer span are not additionally present in the passage (to ensure sufficient lexical overlap between the answer and passage);
Questions shorter than 5 tokens (very short questions are likely to have removed too much information)
For the dataset in question, this process resulted in a total of 14,830
For training the QG model, we employ implementations of BART Lewis et al. (2020) from Huggingface Wolf et al. (2019). The QG model we employ is BART-base. We train the QG model on the QG data for 3 epochs with a learning rate of , using the AdamW optimizer Loshchilov and Hutter (2019).
1.2 Unsupervised QA
We carry out experiments on six extractive QA datasets, namely, SQuAD1.1 Rajpurkar et al. (2016), NewsQA Trischler et al. (2017), Natural Questions Kwiatkowski et al. (2019), TriviaQA Joshi et al. (2017), BioASQ Tsatsaronis et al. (2015) and DuoRC Saha et al. (2018). We employ the official data of SQuAD1.1, NewsQA and TriviaQA and for Natural Questions, BioASQ and DuoRC, we employ the pre-processed data released by MRQA Fisch et al. (2019).
To generate synthetic QA training data, we make use of Wikidumps https://dumps.wikimedia.org/ by firstly removing all HTML tags and reference links, then extracting paragraphs that are longer than 500 characters, resulting in 60k paragraphs sampled from all paragraphs of Wikidumps. We employ the NER toolkits of Spacyhttps://spacy.io/ Honnibal et al. (2020) and AllenNLPhttps://demo.allennlp.org/named-entity-recognition/named-entity-recognition Gardner et al. (2017) to extract entity mentions in the paragraphs. We then remove paragraph, answer pairs that meet one or more of the following three conditions: 1) paragraphs with less than 20 words and more than 480 words; 2) paragraphs with no extracted answer, or where the extracted answer is not in the paragraph due to text tokenization; 3) answers consisting of a single pronoun.
Paragraphs and answers are concatenated to form sequences of the form passage
The QA model we employ is BERT-large-whole-word-masking (which we henceforth refer to as BERT-large for ease of reference). Document length and stride length are 364 and 128 respectively, the learning rate is set to . Evaluation metrics for unsupervised QA are Exact Match (EM) and F-1 score.
2 Results
We use the 20k generated synthetic QA pairs to train a BERT QA model and first validate its performance on the development sets of three benchmark QA datasets based on Wikipedia – SQuAD1.1, Natural Questions and TriviaQA. The results of our method are shown in Tables 1 and 2. The unsupervised baselines we compare with are as follows:
Lewis et al. (2019) employ unsupervised neural machine translation Artetxe et al. (2018) to train a QG model; 4M synthetic QA examples were generated to train a QA model;
Li et al. (2020) employ dependency trees to generate questions and employed cited documents as passages.
For comparison, we also show the results of some supervised models fine-tuned on the corresponding training sets: Match-LSTM Wang and Jiang , BiDAF Seo et al. (2016), BERT-base and BERT-large Devlin et al. (2019).
SQuAD1.1 results are shown in Table 1. The results of all baseline models are taken directly from published work. As can be seen from results in Table 1, our proposed method outperforms all unsupervised baselines, and even exceeds the performance of one supervised model, Match-LSTM Wang and Jiang .
Results for Natural Questions and TriviaQA are shown in Table 2. The results of all baseline models were produced
using the released synthetic QA data to finetune a BERT-large model. Our method outperforms previous state-of-the-art unsupervised methods by a substantial margin, obtaining relative improvements over the best unsupervised baseline model of 47% with respect to EM, 10% F-1 on Natural Questions, and by 34% EM and 12% F-1 on TriviaQA.
In summary, our method achieves the best performance (both in terms of EM and F-1) out of three unsupervised models on all three tested datasets. Furthermore, this high performance is possible with as few as 20k training examples. Compared to previous work, this is approximately less than 10% of the training data employed Li et al. (2020).
We also validate our method’s efficacy on three out-of-domain QA datasets: NewsQA created from news articles, BioASQ created from biomedical articles, and DuoRC created from movie plots, for the purpose of evaluating the transferability of the Wikipedia-based synthetic data. Results in Table 3 show that our proposed method additionally outperforms the unsupervised baseline models on the out-of-domain datasets, achieving F1 improvements over previous state-of-the-art methods by 3.8, 4.5 and 5.4 points respectively. It is worth noting that our data adapts very well to DuoRC, created from movie plots where the narrative style is expected to require more complex reasoning. Experiment results additionally indicate that our generated synthetic data transfers well to domains distinct from that of the original summary data.
Analysis
In the unsupervised QA experiments, we extracted answers from Wikipedia passages before feeding them into our QG model to obtain questions. These
2 Effect of Different Heuristics
We additionally investigate the effect of a range of alternate heuristics employed in the process of constructing the QG training data described in Section 3.1. Recall that the QG data is employed to train a question generator which is then employed to generate synthetic QA data for unsupervised QA.
Naive-QG only employs summary sentences as passages (instead of the original articles) and generates trivial questions in which only the answer spans are replaced with the appropriate question words. For example, for the sentence Stephen Hawking announced the party in the morning, with the party as the answer span, the question generated by Naive-QG would be Stephen Hawking announced what in the morning? We employ the summary sentences as input and questions as target output to form the QG training data.
Summary-QG makes use of the original news articles of the summaries as passages rather than summary sentences to avoid high lexical overlap between the passage and question.
Summary-QG can work with the following heuristics:
Main Verb: we only generate questions based on the SRL frame of the main verb (root verb) in the dependency tree of the summary sentences, rather than using verbs in subordinate clauses;
Wh-Movement: we move the question words to the beginning of the sentence. For example, in the sentence Stephen Hawking announced what in the morning? we move what to the beginning to obtain what Stephen Hawking announced in the morning?;
Decomp-Verb: the main verb is decomposed to its base form and auxiliaries;
NER-Wh: we employ the NER tags to get more precise question words for an answer. For example, for the answer span NBA player Michael Jordan, the question words would be which NBA player instead of who or what.
We employ the QG data generated by these heuristics to train QG models, which leads to six BART-QG models. We then employ these six models to further generate synthetic QA data based on the same Wikipedia data and compare their performances on the SQuAD1.1 dev set. The results in Table 6 show that using articles as passages to avoid lexical overlap with their summary-generated questions greatly improves QA performance. Summary-QG outperforms Naive-QG by roughly 20 EM points and 16 F-1 points. The results for the other heuristics show that they continuously improve the performance, especially Wh-Movement and Decomp-Verb which make the questions in the QG data more similar to the questions in the QA dataset.
3 Effect of the Size of Synthetic QA Data
We investigate the effects of varying the quantity of synthetic QA data. Results in Figure 3 show that our synthetic data allows the QA model to achieve competitive performance even with fewer than 20k examples, which suggests that our synthetic data contains sufficient QA knowledge to enable models to correctly answer a question with less synthetic data compared to previous unsupervised methods. The data-efficiency of our approach increases the feasibility of training a QA system for a target domain where there is no labeled QA data available.
4 Few-shot Learning
We conduct experiments in a few-shot learning setting, in which we employ a limited number of labeled QA examples from the training set. We take the model trained with our synthetic QA data, the model trained with the synthetic QA data of Li et al. (2020) and a vanilla BERT model, with all QA models employing BERT-large Devlin et al. (2019). We train these models using progressively increasing amounts of labeled QA samples from Natural Questions (NQ) and SQuAD1.1 and assess their performance on corresponding dev sets. Results are shown in Figure 4 where with only a small amount of labeled data (less than 5,000 examples), our method outperforms Li et al. (2020) and BERT-large, clearly demonstrating the efficacy of our approach in a few-shot learning setting.
5 QG Error Analysis
Despite substantial improvements over baselines, our proposed approach inevitably still incurs error and we therefore take a closer look at the questions generated by our QG model. We manually examine 50 randomly selected questions, 31 (62%) of which were deemed high quality questions. The remaining 19 contain various errors with some questions containing more than one error, including mismatched wh-word and answer (12%), missing information needed to locate the answer (8%), factual errors (10%) and grammatical errors (8) (16%) Typical examples are shown in Table 4.
Conclusion
We propose an unsupervised question generation method which uses summarization data to 1) minimize the lexical overlap between passage and question and 2) provide a QA-dataset-independent way of generating questions. Our unsupervised QA extrinsic evaluation on SQuAD1.1, NQ and TriviaQA using synthetic QA data generated by our method shows that our method substantially outperforms previous methods for generating synthetic QA for unsupervised QA. Furthermore, our synthetic QA data transfers well to the out-of-domain datasets. Future work includes refining our question generation heuristics and applying our approach to other languages.
Acknowledgements
This work was funded by Science Foundation Ireland through the SFI Centre for Research Training in Machine Learning (18/CRT/6183). We also thank the reviewers for their insightful and helpful comments.
References
Appendix A Appendix
We also study the effects of different beam size in generating synthetic questions to the performance of downstream QA task. Experiments are conducted on SQuAD1.1 dev set using BERT-large, questions in the synthetic QA data are generated with different beam size using the same BART-QG model. The experimental results in Figure 5 show that the beam size is an important factor affecting the performance of unsupervised QA, the largest margin between the highest score (beam-15) and the lowest score (beam-1) in Figure 5 is close to 4 points on EM and F-1 score.
A.2 Question Type Distribution
We show the distribution of question types of QG data described in Section 4.1.1, training set of SQuAD1.1 and our synthetic QA data in Section 4.1.2 in Figure 6, question types are defined as What, When, Where, Who, Why, How. The QG data has more what, when, where questions, indicating the existence of more SRL arguments associated with such question types in the summary sentences.
A.3 Generated QA Examples
Some Wikipedia-based