RikiNet: Reading Wikipedia Pages for Natural Question Answering

Introduction

Machine reading comprehension (MRC) refers to the task of finding answers to given questions by reading and understanding some documents. It represents a challenging benchmark task in natural language understanding (NLU). With the progress of large-scale pre-trained language models Devlin et al. (2018), state-of-the-art MRC models Ju et al. (2019); Yang et al. (2019); Lan et al. (2019); Zhang et al. (2019); Liu et al. (2019) have already surpassed human-level performance on certain commonly used MRC benchmark datasets, such as SQuAD 1.1 Rajpurkar et al. (2016), SQuAD 2.0 Rajpurkar et al. (2018), and CoQA Reddy et al. (2019).

Recently, a new benchmark MRC dataset called Natural QuestionsNQ provides some visual examples of the data at https://ai.google.com/research/NaturalQuestions/visualization. (NQ) Kwiatkowski et al. (2019) has presented a substantially greater challenge for the existing MRC models. Specifically, there are two main challenges in NQ compared to the previous MRC datasets like SQuAD 2.0. Firstly, instead of providing one relatively short paragraph for each question-answer (QA) pair, NQ gives an entire Wikipedia page which is significantly longer compared to other datasets. Secondly, NQ task not only requires the model to find an answer span (called short answer) to the question like previous MRC tasks but also asks the model to find a paragraph that contains the information required to answer the question (called long answer).

In this paper, we focus on the NQ task and propose a new MRC model called RikiNet tailored to its associated challenges, which Reads the Wikipedia pages for natural question answering. For the first challenge of the NQ task mentioned above, RikiNet employs the proposed Dynamic Paragraph Dual-Attention (DPDA) reader which contains multiple DPDA blocks. In each DPDA block, we iteratively perform dual-attention to represent documents and questions, and employ paragraph self-attention with dynamic attention mask to fuse key tokens in each paragraph. The resulting context-aware question representation, question-aware token-level, and paragraph-level representations are fed into the predictor to obtain the answer. The motivations of designing DPDA reader are: (a) Although the entire Wikipedia page contains a large amount of text, one key observation is that most answers are only related to a few words in one paragraph; (b) The final paragraph representation can be used naturally for predicting long answers. We describe the details of DPDA reader in § 3.1.

For the second challenge, unlike prior works on NQ dataset Alberti et al. (2019b); Pan et al. (2019) that only predict the short answer and directly select its paragraph as long answer, RikiNet employs a multi-level cascaded answer predictor which jointly predict the short answer span, the long answer paragraph, and the answer type in a cascaded manner. Another key intuition motivating our design is that even if the relevant documents are not given, humans can easily judge that some questions have no short answers Borschinger et al. (2019). Take this question as a motivating example:“What is the origin of the Nobel prize?” The answer should be based on a long story, which cannot be easily expressed in a short span of entities. Therefore we also feed the question representation into the predictor as an auxiliary prior to answer type prediction. The details will be given in § 3.2.

On the NQ test set, our single model obtains 74.3 F1 scores on the long-answer task (LA) and 57.9 F1 scores on the short-answer task (SA) compared to the published best single model Alberti et al. (2019a) results of 66.8 F1 on LA and 53.9 F1 on SA. To the best of our knowledge, RikiNet is the first single model that outperforms the single human performance Kwiatkowski et al. (2019) on both LA and SA. Besides, our ensemble model obtains 76.1 F1 on LA and 61.3 F1 on SA, which achieves the best performance of both LA and SA on the official NQ leaderboard.

Preliminaries

Before we describe our model in detail, we first introduce the notations and problem formalization. Our paper considers the following NQ Kwiatkowski et al. (2019) task: Given a natural question $q$, a related Wikipedia page $p$ (in the top 5 search results returned by the Google search engine), the model outputs a paragraph within the Wikipedia page $p$ as the long answer which contains enough information to infer the answer to the question, and an entity span within the long answer that answers the question as the short answer. Also, the short answer of the 1% Wikipedia page is “yes” or “no”, instead of a short span. Both long answers and short answers can be $\mathtt{NULL}$ (i.e., no such answer could be found).

For each tuple $(q,d,c,s,e,t)$ of the data pair $(q,p)$, RikiNet takes $d$ and $q$ as inputs, and jointly predicts $c,s,e,t$. Finally we merge the prediction results of every tuple to obtain the final predicted long answer, short answer, and their confidence scores of the data pair $(q,p)$ for evaluation.

Methodology

We propose the RikiNet which Reads the Wikipedia pages for natural question answering. As shown in Fig. 1, RikiNet consists of two modules: (a) the dynamic paragraph dual-attention reader as described in §3.1, and (b) the multi-level cascaded answer predictor as described in §3.2.

Dynamic Paragraph Dual-Attention (DPDA) reader aims to represent the document span $d$ and the question $q$. It outputs the context-aware question representation, question-aware token-level document representation, and paragraph-level document representation, which will be all fed into the predictor to obtain the long and short answers.

1.2 Dynamic Paragraph Dual-Attention Block

As shown on the left in Fig. 1, DPDA reader contains multiple Dynamic Paragraph Dual-Attention (DPDA) blocks. The first block takes $Q_{0}$ and $D_{0}$ as the inputs. The outputs $Q_{(t)}$ and $D_{(t)}$ of the $t$-th block are then fed into the next block. Each block contains three types of layers: the dual-attention layer, the paragraph dynamic self-attention layer, and the question self-attention layer. The last DPDA block outputs the final question and document representations. We describe them in detail now.

To strengthen the information fusion from the question to the paragraphs as well as from the paragraphs to the question, we adapt a dual-attention mechanism, which has been shown effective in other MRC models Xiong et al. (2018); Seo et al. (2017); Xiong et al. (2017). We further tweak it by increasing the depth of attention followed by a residual connection He et al. (2016) and layer normalization Ba et al. (2016).

Similar to Xiong et al. (2017); Seo et al. (2017), we obtain the question-aware representation of the document by

where $[\cdot;\cdot]$ denotes concatenation. We also obtain the context-aware question representation in a dual way:

We finally apply the residual connection and layer normalization to both the question and the document representations with the linear transformations.

This layer uses a transformer self-attention block Vaswani et al. (2017) to further enrich the question representation:

where $T$ denotes the number of the DPDA blocks. This question embedding $\bm{q}$ will be further fed into the predictor for answer type prediction.

This layer is responsible for gathering information on the key tokens in each paragraph. The token-level representation $D_{(t)}$ is first given by:

It forces each token to only attend to the tokens within the same paragraph. Therefore, each paragraph representation focuses on its internal token information after the mean pooling of Eq. (2).

Based on the second motivation, we dynamically generate another attention mask to select key tokens before self-attention. We use a neural network $\mathcal{F}_{(t)}^{\Phi}$ called scorer with the $\operatorname*{Sigmoid}$ activation function to calculate the importance score for each token:

2 Multi-level Cascaded Answer Predictor

Due to the nature of the NQ tasks, a short answer is always contained within a long answer, and thus it makes sense to use the prediction of long answers to facilitate the process of obtaining short answers. As shown on the right in Fig. 1, we design a cascaded structure to exploit this dependency. This predictor takes the token representation $D_{(T)}$, the paragraph representation $L$, and the question embedding $\bm{q}$ as inputs to predict four outputs in a cascaded manner: (1) long answer $\rightarrow$ (2) the start position of the short answer span $\rightarrow$ (3) the end position of the short answer span $\rightarrow$ (4) the answer type. That is, the previous results are used for the next tasks as indicated by the notation “$\rightarrow$”.

Firstly, we use the long-answer prediction representation $H^{L}$ and the token representation $D_{(T)}$ as the inputs to predict the start position of the short answer. Then the prediction representation of the start position of the short answer will be re-used to predict the end position.

For training, we compute cross-entropy loss over the above mentioned output logits, and jointly minimize these four cross-entropy losses as:

During inference, we calculate the final long-answer score $\Psi^{L}$ for all the paragraphs within the Wikipedia page based on the long-answer logits $\bm{o}^{L}$ and the answer type logits $\bm{o}^{T}$. The long-answer score of paragraph $c$ can be written as

where $\bm{o}^{T}$ denotes the logits where the answer type is “NULL”(no answer), $\sum_{t=1}^{4}\bm{o}^{T}[t]$ denotes the sum of the logits where the answer type is not “NULL”. The answer type score can be seen as a bias of each document span in the Wikipedia page. Then we select the paragraph of the highest long-answer score $\Psi^{L}$ over the entire Wikipedia page as the long answer.

Similarly, the short-answer score of the corresponding span $(s,e)$ is calculate by

where $\bm{o}^{T}$ denotes the score where the answer type is “SHORT”(has short answer). We select the short answer span which has the highest short-answer score $\Psi^{S}$ within the long answer as the final short answer. We use the official NQ evaluation script to set two separate thresholds for predicting whether the two types of answers are answerable.

Experiments

We focus on the Natural Questions (NQ) Kwiatkowski et al. (2019) dataset in this work. The public release of the NQ dataset consists of 307,373 training examples and 7,830 examples for development data (dev set). NQ provides a blind test set contains 7,842 examples, which can only be accessed through a public leaderboard submission.

2 Implementation Details

As discussed in § 2, we generate multiple document spans by splitting the Wikipedia page with a sliding window. Following Pan et al. (2019); Alberti et al. (2019b), the size and stride of the sliding window are set to 512 and 192 tokens respectively. The average number of document spans of one Wikipedia page is about 22. Since most of the document span does not contain the answer, the number of negative samples (i.e., no answer) and positive samples (i.e., has answers) is extremely imbalanced. We follow Pan et al. (2019); Alberti et al. (2019b) to sub-sample negative instances for training, where the rate of sub-sampling negative instance is the same as in Pan et al. (2019). As a result, there are 469,062 training instances in total.

We use Adam optimizer Kingma and Ba (2015) with a batch size of $36$ for model training. The initial learning rate, the learning rate warmup proportion, the training epoch, the hidden size $h$, the number of blocks $T$, and the hyperparameter $K$ are set to $2\times 10^{-5}$, $0.1$, $2$, $1024$, $2$, and $256$ respectively. Our model takes approximately 24 hours to train with 4 Nvidia Tesla P40. Evaluation completed in about 6 hours on the NQ dev and test set with a single Nvidia Tesla P100.

We use the Google released BERT-large model fine-tuned with synthetic self-training Alberti et al. (2019a) to encode the document and question as described in § 3.1.1. We also compare the performance of RikiNet which uses the pre-trained RoBERTa large model Liu et al. (2019). It should be noted that our RikiNet is orthogonal to the choice of a particular pre-trained language model.

3 Main Results

We present a comparison between previously published works on the NQ task and our RikiNet. We report the results of the precision (P), the recall (R), and the F1 score for the long-answer (LA) and short-answer (SA) tasks on both test set and dev set in Tab. 1. The first two lines of Tab. 1 show the results of two multi-passage MRC baseline models presented in the original NQ paper Kwiatkowski et al. (2019). The third to sixth lines show the results of the previous state-of-the-art models. These models all employ the BERTlarge model and perform better than that two baselines. Our RikiNet-BERTlarge also employs the BERTlarge model, and its single model has achieved a significant improvement over the previously published best model on the test set (LA from 66.8 F1 to 74.3 F1, and SA from 53.9 F1 to 57.9 F1). To the best of our knowledge, this is the firstThe single RikiNet-BERTlarge model was submitted to the NQ public leaderboard on 7 Nov. 2019. single model that surpasses the single human performance Kwiatkowski et al. (2019) on both LA and SA tasks. We also provide a BERTjoint Alberti et al. (2019b) + RoBERTa large Liu et al. (2019) baseline on NQ, which only replaces the BERTlarge in BERTjoint method with RoBERTa large. To be expected, the BERTjoint + RoBERTa large performs better than original BERTjoint. Furthermore, our single model of RikiNet-RoBERTa large which employs RoBERTa large model also achieves better performance on both LA and SA, significantly outperforming BERTjoint + RoBERTa large. These results demonstrate the effectiveness of our RikiNet.

Since most submissions on the NQ leaderboard are ensemble models, we also report the results of our ensemble model, which consists of three RikiNet-RoBERTa large models with different hyper-parameters. At the time of submission (29 Nov. 2019), the NQ leaderboard shows that our ensemble model achieves the best performance on both LA (F1 76.1) and SA (F1 61.3).

4 Ablation Study

RikiNet consists of two key parts: DPDA reader and multi-level cascaded answer predictor. To get a better insight into RikiNet, we conduct an in-depth ablation study on probing these two modules. We report the LA and SA F1 scores on the dev set.

We keep the predictor and remove the component of the DPDA reader. The results are shown in Tab. 2. In (a), we remove the entire DPDA reader as introduced in § 3.1 except BERTlarge. In (b), (c), and (d), we remove the dual-attention layer, question self-attention layer, and paragraph dynamic self-attention layer as described in § 3.1.1 respectively. In (e) and (f), we remove the paragraph attention mask of Eq. (3) and the dynamic attention mask of Eq. (4) respectively. We can see that after removing the DPDA reader, the performance drops sharply. In addition, the paragraph dynamic self-attention layer has the greatest impact on performance. Moreover, both the paragraph attention mask and dynamic attention mask contribute to the performance improvement.

We also change the hyper-parameter $K$ and the number of blocks $T$. Results show that the setting of $K=384$ performs better than $K=512$ (i.e., no dynamic attention mask), and $K=256$ performs best. For the number of DPDA blocks $T$, the model achieves the best performance when $T=2$.

On the predictor side, we further remove or replace its component and report the results in Tab. 3. In (1) we remove the whole DPDA reader and predictor. In (2), we remove the way of multi-level prediction (i.e., training the model to predict long and short answer jointly) described in § 3.2, and follow the previous work Alberti et al. (2019b) to directly predict the short answer and then select its paragraph as the long answer. We can see that our multi-level prediction is critical to the long answer prediction. In (3) we only remove the cascaded structure but keep the multi-level prediction, which means that the prediction representations are no longer used as input for other predictions, the performance of both long and short answers drops about 1.0 F1 score. In (4) we change the ordering of cascaded process. That is instead of considering long answer first and then short answer as described in § 3.2, we consider the cascaded structure of short answer first and then long answer. However, we get slightly worse results in this way. In (5), we remove the question embedding which is used for answer type prediction. It can be observed that the question embedding contributes to performance improvement. In the variants of (6)-(9), we remove the dense prediction layers with $\operatorname*{Tanh}$ activation function and replace it with Bi-directional Long-Short Term Memory (Bi-LSTM) Hochreiter and Schmidhuber (1997); Schuster and Paliwal (1997) layers, transformer self-attention blocks, and dense prediction layers with Gaussian Error Linear Unit $\operatorname*{GELU}$ Hendrycks and Gimpel (2016) activation function but neither get better performance.

Overall, both proposed DPDA reader and multi-level cascaded answer predictor significantly improve the model performance.

Related Works

Natural Questions (NQ) dataset Kwiatkowski et al. (2019) has been recently proposed, where each question is paired with an entire Wikipedia page which is a long document containing multiple passages. Although BERT Devlin et al. (2018) based MRC models have surpassed human performance on several MRC benchmark datasets Lan et al. (2019); Devlin et al. (2018); Liu et al. (2019); Rajpurkar et al. (2018), a similar BERT method Alberti et al. (2019b) still has a big gap with human performance on NQ dataset.

There are several recently proposed deep learning approaches for multi-passage reading comprehension. Chen et al. (2017) propose DrQA which contains a document retriever and a document reader (DocReader). Clark and Gardner (2018) introduce Document-QA which utilizes TF-IDF for paragraph selection and uses a shared normalization training objective. De Cao et al. (2019) employ graph convolutional networks (GCNs) for this task. Zhuang and Wang (2019) design a gated token-level selection mechanism with a local convolution. In contrast, our RikiNet considers multi-level representations with a set of complementary attention mechanisms.

To solve the NQ task, Kwiatkowski et al. (2019) adapt Document-QA Clark and Gardner (2018) for NQ, and also utilizes DecAtt Parikh et al. (2016) for paragraph selection and DocReader Chen et al. (2017) for answer prediction. BERTjointAlberti et al. (2019b) modifies BERT for NQ. Besides, some works focus on using data augmentation to improve the MRC models on NQ. Alberti et al. (2019a) propose a synthetic QA corpora generation method based on roundtrip consistency. Glass et al. (2019) propose a span selection method for BERT pre-training (SSPT). More recently, Pan et al. (2019) introduce attention-over-attention Cui et al. (2017) into the BERT model. Pan et al. (2019) also propose several techniques of data augmentation and model ensemble to further improve the model performance on NQ. Although the use of data augmentation and other advanced pre-trained language models Lan et al. (2019) may further improve model performance, as this is not the main focus of this paper, we leave them as our future work. Our RikiNet is a new MRC model designed tailored to the NQ challenges and can effectively represent the document and question at multi-levels to jointly predict the answers, which significantly outperforms the above methods.

Conclusion

We propose the RikiNet, which reads the Wikipedia pages to answer the natural question. The RikiNet consists of a dynamic paragraph dual-attention reader which learns the token-level, paragraph-level and question representations, and a multi-level cascaded answer predictor which jointly predicts the long and short answers in a cascade manner. On the Natural Questions dataset, the RikiNet is the first single model that outperforms the single human performance. Furthermore, the RikiNet ensemble achieves the new state-of-the-art results at 76.1 F1 on long-answer and 61.3 F1 on short-answer tasks, which significantly outperforms all the other models on both criteria.

Acknowledgment

This work is supported by National Natural Science Fund for Distinguished Young Scholar (Grant No. 61625204) and partially supported by the Key Program of National Science Foundation of China (Grant No. 61836006).

References