Knowledge Guided Text Retrieval and Reading for Open Domain Question Answering

Introduction

Open-domain question answering systems aim to answer any question a user can pose, with evidence provided by either factual text such as Wikipedia (Chen et al., 2017; Yang et al., 2019) or knowledge bases (KBs) such as Freebase (Berant et al., 2013; Kwiatkowski et al., 2013; Yih et al., 2015). Textual evidence, in general, has better coverage but KBs more directly support making complex inferences. It remains an open question how to best make use of KBs without sacrificing coverage in text-based open domain QA. Previous work has converted KB facts to sentences to provide extra evidence (Weissenborn et al., 2017; Mihaylov and Frank, 2018), but do not explicitly use the KB graph structure. In this paper, we show that such structure can be highly beneficial for both retrieving text passages and fusing information across them in open-domain text-based QA, for example as shown in Figure 1.

We introduce a general approach for text-based open-domain QA that is knowledge guided: it retrieves and reads a passage graph, where vertices are passages of text and edges represent relationships that are derived either from an external knowledge base or co-occurrence in the same article. Our goal is to combine the high coverage of textual corpora with the structural information in knowledge bases, to improve both the retrieval coverage and accuracy of the resulting model. Unlike standard approaches that retrieve and read a set of passages Chen et al. (2017), our approach integrates graph structure at every stage to construct, retrieve and read a graph of passages.

Our approach first retrieves a passage graph by expanding a set of seed passages based on the graph structure of the knowledge base and the co-occurrence in text corpus (Figure 1). We then introduce a reader model that extends BERT (Devlin et al., 2019) and propagates information from related passages and their relations, enabling knowledge-rich cross-passage representations. Together, this approach allows for better coverage (e.g. the graph contains many passages that text-match retrieval would miss) and accuracy (e.g. by better combining information across related passages to find the best answer).

Experiments demonstrate significant improvements on three popular open-domain QA datasets: WebQuestions (Berant et al., 2013), Natural Questions (Kwiatkowski et al., 2019) and TriviaQA (Joshi et al., 2017). Our graph-based retrieval and reader models, together, improve accuracy consistently and significantly, outperforming the non-graph baselines by 2–11% and matching or exceeding the state-of-the-art in every case without an expensive end-to-end training regime. Through extensive ablations, we show that both graph-based retrieval and reader models substantially contribute to the performance improvements, even when we fix the other component.

Related Work

Text-based open-domain QA is a long standing problem Voorhees et al. (1999); Ferrucci et al. (2010). Recent work has focused on two-stage approaches that combine information retrieval with neural reading comprehension Chen et al. (2017); Wang et al. (2018); Das et al. (2019); Yang et al. (2019). We follow this tradition but introduce a new framework which retrieves and reads a graph of passages.

Other graph retrieval methods have been developed, either using entity name matching Ding et al. (2019); Xiong et al. (2019b); Godbole et al. (2019) or hyperlinks Asai et al. (2020). However, we are not aware of work integrating external knowledge bases or tightly coupling the approach with a graph reader, as we do in this paper. Moreover, most previous graph-based approaches evaluate on questions that are explicitly written to encourage reasoning based on a chain of entities, such as WikiHop (Welbl et al., 2017) or HotpotQA (Yang et al., 2018). In this work, we instead focus on naturally gathered questions which require much more diverse types of cross paragraph reasoning.

After retrieving evidence passages, most pipeline systems use a reading comprehension model to extract the answer. Previous work either concatenates retrieved passages into a single sequence Swayamdipta et al. (2018); Yang et al. (2018); Song et al. (2018) with no explicit model of how they are related, or reads each passage in parallel Clark and Gardner (2018); Alberti et al. (2019); Min et al. (2019b); Wang et al. (2019) with no ability to fuse the information they contain. To the best of our knowledge, reading passages by incorporating structural information across passages has not been studied previously. The most related models are Song et al. (2018) and Cao et al. (2019), which fuse information through entities detected by entity linking and coreference resolution on WikiHop. In contrast, our model fuses information across passages, to better model the overall relationships between the different blocks of text.

Other lines of research in open-domain QA include joint learning of retrieval and reader components (Lee et al., 2019) or direct phrase retrieval in a large collection of documents Seo et al. (2019). Although end-to-end training can further improve the performance of our approach, this paper only focuses on pipeline approaches since end-to-end training is computationally and memory expensive.

Question answering over knowledge bases has also been well studied Berant et al. (2013); Kwiatkowski et al. (2013); Yih et al. (2015), typically without using any external text collections. However, recent work has augmented knowledge bases with text from Wikipedia Das et al. (2017); Sun et al. (2018, 2019); Xiong et al. (2019a), to increase factual coverage when a given knowledge base is incomplete. In this paper, we study what can be loosely seen as an inverse problem. The model answers questions based on a large set of documents, and the knowledge base is used to better model relationships between different passages of text.

Approach

We present a new general approach for text-based open-domain question answering, which consists of a retrieval model GraphRetriever and a neural reader model GraphReader. The overall approach is illustrated in Figure 2. GraphRetriever retrieves a graph of passages in which vertices are passages and edges denote relationships between passages (Section 3.1). GraphReader reads the input passage graph and returns the answer (Section 3.2).

The goal is to answer the question based on a text corpus $\mathcal{C}$, which consists of a large collection of articles and each of them can be divided into multiple passages. We also assume an external knowledge base $\mathcal{K}=\{(e_{1},r,e_{2})\}$ exists where $e_{1},e_{2}$ are entities and $r$ is a relation, and there is a 1-1 mapping between the KB entities and articles in the text corpus. Specifically, we use Wikipedia as the text corpus $\mathcal{C}$ and Wikidata (Vrandečić and Krötzsch, 2014) as the knowledge base $\mathcal{K}$, as there exists an alignment between the two resources and Wikipedia has been widely used before in open-domain question answering research Chen et al. (2017); Seo et al. (2019).

1 GraphRetriever

Starting from seed passages $\mathcal{P}^{(0)}$, GraphRetriever expands the passage graph from $\mathcal{P}^{(m-1)}$ to $\mathcal{P}^{(m)}$ by iterating over the following two methods, until it includes $n$ passages.

First, the passage graph is updated by adding passages that are related to $\mathcal{P}^{(m-1)}$ according to a relation present in Wikidata. Specifically, if $p_{i}\in\mathcal{P}^{(m-1)}$ and $p_{j}$ are the first passages of Wikipedia articles that correspond to KB entities $e_{p_{i}}$ and $e_{p_{j}}$ such that $(e_{p_{i}},r_{i,j},e_{p_{j}})\in\mathcal{K}$, $p_{j}$ is added to the passage graph, being connected to $p_{i}$ through $r_{i,j}$. Although this may include some entities that are not closely related to the question, it still increases the coverage of entities related to the answer, as shown in Section 4.4.

Finally, we retrieve a passage graph consisting of $n$ passages: $\{p_{1},\dots,p_{n}\}$. The relations between the passages are denoted by $\{r_{i,j}\mid 1\leq i,j\leq n\}$, where $r_{i,j}$ is either a KB relation, child, parent or no_relation, indicating the relationship between a passage pair ($p_{i}$, $p_{j}$).

2 GraphReader

Our GraphReader takes a question $q$ and $n$ retrieved passages $p_{1},p_{2},\ldots,p_{n}$ (and their relations $r_{i,j}$) and aims to output an answer to the question as a text span in one of retrieved passages. Instead of processing each passage independently, our approach obtains knowledge-rich representations of passages by fusing information from linked passages across the graph structure.

Formally, given the question $q$ and a passage $p_{i}$, GraphReader first obtains a question-aware passage representation:

where $L$ is the maximum length of each passage, and $h$ is the hidden dimension. We use BERT (Devlin et al., 2019), although the approach could be applied with many other encoders.

Additionally, GraphReader encodes a relation $r_{i,j}$ through a relation encoder:

We consider the most frequent 98 relations and group the other relations as unk_releation, total to be 100 including no_relation. We directly learn an an embedding matrix to get a vector representation for each relation, which works well in practice since we have relatively few relations and many examples of each.

2.2 Fusing Passage Representations

We first consider binary relations which encodes whether a passage pair is related or not, without incorporating relations. Specifically,

where $\mathbf{W}_{f}$ and $\mathbf{b}_{f}$ are learnable parameters, and $\oplus$ is a concatenation.

We then consider a relation-aware composition function:

where $f$ is a composition function, $\oplus$ is a concatenation, and $\mathbf{W}_{f}$ and $\mathbf{b}_{f}$ are learnable parameters. We use concatenation for the composition function, $f(\mathbf{r}_{i,j},\mathbf{z}^{(m)}_{j})=\textsc{Concat}(\mathbf{r}_{i,j},\mathbf{z}^{(m)}_{j})$, for simplicity because it worked as well as more complex functions such as element-wise multiplication and bilinear mappings in our early experiments.

2.3 Answering Questions

For training, we use the maximum marginal likelihood objective by maximizing:

where $\mathcal{S}_{i}$ is a set of spans which correspond to the answer text in $p_{i}$. We tried using $\mathbf{Z}$ for span predictions, but did not see meaningful improvements. We hypothesize it is because span prediction given the correct passage is an easier task compared to choosing the right evidence passage.We observed that over 80% of the error cases the baseline model made are due to the incorrect passage selection on all datasets.

Experiments

We evaluate our model on three open-domain question answering datasets, where the evaluation metric is Exact Match. (1) WebQuestions (Berant et al., 2013) is originally a QA dataset designed to answer questions based on Freebase; the questions were collected through Google Suggest API. We follow Chen et al. (2017) and frame the problem as a span selection task over Wikipedia. (2) Natural Questions (Kwiatkowski et al., 2019) consists of questions collected using the Google search engine; questions with short answers up to 5 tokens are taken following Lee et al. (2019). (3) TriviaQA (Joshi et al., 2017) consists of questions from trivia and quiz-league websites. For all datasets, we only use question and answer pairs for training and testing, and discard the provided evidence documents which are part of reading comprehension tasks. We follow the data splits from Chen et al. (2017) for WebQuestions and Min et al. (2019a) on Natural Questions and TriviaQA.https://bit.ly/2q8mshc and https://bit.ly/2HK1Fqn. Table 1 shows the statistics of the datasets and the density of the graph (number of relations per passage) retrieved by GraphRetriever.

2 Baselines

For retrieval, we compare our GraphRetriever to a pure text-match based retrieval method which retrieves Wikipedia articles based on TF-IDF scores (Chen et al., 2017) and ranks their passages through BM25 (Robertson et al., 2009). This is to investigate if leveraging the knowledge base actually improves the retrieval component.

For reader, we compare our GraphReader with two competitive baselines which read each passage in parallel, ParReader and ParReader++. Both baselines obtain question-aware passage representations $\mathbf{P}_{1},\dots,\mathbf{P}_{n}$ as described in Section 3.2 with a different way of calculating $P_{\textrm{sel}}(i)$. ParReader computes $P_{\textrm{sel}}(i)$ using a binary classifier:

3 Implementation Details

4 Main Results

The main results are given in Table 2. We observed three overall trends: (1) GraphRetrieveroffers significant performance gains over text-match retrieval when we compare within the same reader across all datasets, e.g., 1–11% absolute gains with ParReader++. This indicates that graph-based retrieval provides passages with significantly better evidence to answer the question. (2) GraphReaderoutperforms two ParReader baselines consistently across all datasets, achieving 1–5% absolute gains. This result demonstrates that fusing information across passages is more effective than reading each passage in isolation. (3) GraphReaderusing relations offers some improvement over GraphReader with binary relations. The gains are smaller than expected, likely because the relations are inferred based from the text. In order to verify this hypothesis, we modify our reader to have an output layer for relation classification, and observe that the accuracy is over 80% for all datasets.

We also compare our results to the previous best models, both pipeline and end-to-end approaches for open-doman QA, in Table 2. Our best-performing model outperforms previously published pipeline models by 6–18%, showcasing the benefit of our graph retrieval and reader models. In particular, our models with GraphRetriever (both baseline and GraphReader) outperform the previous best graph-based retrieval model (Asai et al., 2020) To the best of our knowledge, Asai et al. (2020) (1) is the only graph-based approach evaluated on naturally found questions and (2) also outperforms other graph-based approaches on HotpotQA. by a large margin, despite the fact that they used a stronger BERT model than ours as the base model. Our model also outperforms or matches the end-to-end model (Lee et al., 2019) which is expensive to train as it uses an extra pretraining strategy. Although not explored in this paper, our framework can be trained end-to-end as well, which has a great potential to further advance the state-of-the-art.

Analyses

To better understand model performance, we report a number of ablation studies (Section 5.1) and a qualitative analysis (Section 5.2).

Table 3 compares text-match retrieval and GraphRetriever with ‘Text-match + Wikidata’, a variant of GraphRetriever where we take the union of text-match retrieval and Wikidata-based retrieval, each computed in isolation. Specifically, the text-match retrieval is the baseline described in Section 4.2, and Wikidata-based retrieval is done by obtaining seed passages through entity linking and updating the passage graph only through Wikidata. This variant can be seen as late combination between the text-match and Wikidata-based retrieval, whereas GraphRetriever provides early combination. Although ‘Text-match + Wikidata’ outperforms text-match retrieval by a large margin, our GraphRetriever significantly outperforms this method, showing the importance of jointly leveraging text-match and Wikidata for graph construction.

Table 4 compares the effect of using different relation types in the constructed graph of passages for GraphReader, showing results for the following settings: (a) fully connected, which connects all pairs of passages, (b) empty, which does not include any edges between passages, (c) cross-doc, which only includes edges between passages according to the Wikidata relations, (d) inner-doc, which only includes child and parent, and (e) cross+inner, which includes both cross-doc and inner-doc, corresponding to the graph constructed by our approach. Results indicate that cross-doc and inner-doc relations achieve good performance across two datasets. In particular, using relation information is better than ignoring relation information (fully connected, empty), demonstrating the importance of selecting a good set of graph edges.

Table 5 compares the performance of our graph-based method with two baseline readers where a concatenation of passage pairs is included as input, and ParReader++ reads each of them in isolation. First, ParReader++ (pairs from graph) concatenates passage pairs that are related in the input graph, along with the relation text. Second, ParReader++ (all pairs) concatenates all passage pairs. For these baselines, the concatenated passages are up to 300 tokens. We split each passage up to 145 tokens and set the relation text to be up to 10 tokens. For ParReader++ (pairs from graph), we use $n=20$ instead of $n=\{40,80\}$.This restrition is needed because there are too many passage pairs: even $n=20$ gives $20+190=210$ passages. It is worth noting that concatenating more passages into a single input is non-trivial due to the fixed input length of BERT. Details are provided in Appendix A. Results show that concatenating passages is not competitive, potentially because truncating each passage causes significant information loss.

2 Qualitative Results

Figure 3 shows a few examples from Natural Questions and WebQuestions. Appendix B lists additional examples. They include cases where our method incorporates knowledge-rich relationships between passages to find the correct evidence and answer the question.

In Example 1, text-match retrieval does not retrieve the article ‘Director of the United States Mint’ and fails to retrieve any passage about the director of the US Mint. However, GraphRetriever retrieves the correct evidence passage by using the relationship between ‘United States Mint’ and ‘Director of the United States Mint’, enabling GraphReader to successfully predict the answer. Similarly, in Example 3, Wikidata enables GraphRetriever to retrieve ‘Toyota Motor Sales, USA’ which contains the evidence to the question, whereas text-match retrieval fails to do so.For both Example 1 and 3, initial retrieved articles include some passages containing the evidence, but BM25 passage ranking misses them.

Although Example 2 appears to be easy to humans since the passage from ‘Saint Louis Park’ alone provides the enough evidence, ParReader++ with no relation information makes a wrong prediction, ‘St. Louis County’, potentially because of the similarity in names. However, Wikidata relation in Example 2, is located in, explicitly supports the evidence to answer the question, therefore, GraphReader which leverages graph information easily predicts the right answer.

In Example 3, ParReader++ makes a wrong prediction from the passage ‘Toyota’, potentially because this passage seems more related to the company. GraphReader, however, leverages the relationship between ‘Toyota’ and ‘Toyota Motor Sales, USA’, and predicts the correct answer. Similarly in Example 4, ParReader++ predicts “Ben Feldman” as an answer potentially due to the word “judged by”. However, leveraging relations in the graph has part and part of the series, GraphReader infers that two passages belong to the same series and ‘Drop Dead Diva (Season 3)’ mentions the judge more explicitly.

Conclusion

We proposed a general approach for text-based open-domain question answering that integrates graph structure at every stage to construct, retrieve and read a graph of passages. Our retrieval method leverages both text corpus and a knowledge base to find a relevant set of passages and their relations. Our reader then propagates information according to the input graph, enabling knowledge-rich cross-passage representations. Our approach consistently outperforms competitive baselines on three open-domain QA datasets, WebQuestions, Natural Questions and TriviaQA. We also included a detailed qualitative analysis to illustrate which components contribute the most to the overall system performance.

References

Appendix A Training details

All experiments are done in Python 3.5 and PyTorch 1.1.0 (Paszke et al., 2017). For BERT, we use the uncased version of BERT${}_{\text{BASE}}$ and pytorch-transformers (Wolf et al., 2019)github.com/huggingface/transformers. Specifically, given a question $Q$ and a passage $P_{i}$ where the title of the originated article is $T_{i}$, we form a sequence $S_{i}=Q:{\tt[SEP]}:{\tt<t>}:T_{i}:{\tt</t>}:P_{i}$, where : indicates a concatenation and ${\tt[SEP]}$ is a special token. This sequence is then fed into BERT and the hidden representation of the sequence from the last layer is chosen as a question-aware passage representation. For the embedding matrix for the relation encoder, we keep 100 relations (no_relation, UNK and top 98 relations), which cover over 95% of all relations on all datasets.

For ParReader, we use a batch size of $10$ on WebQuestions and $60$ on Natural Questions and TriviaQA. For ParReader++ and GraphReader, we use a batch size of $8$ on WebQuestions and $16$ on the rest two. For each fusion layer, we apply dropout (Srivastava et al., 2014) with a probability of $0.3$. For training, we evaluate the model on the development set periodically, and stop training when Exact Match score does not improve $10$ times. For all other hyperparameters not mentioned, we follow the default settings from pytorch-transformers.

As mentioned in Section 4.3, for each model, we experiment with $M=\{1,2,3\}$ and two sampling methods, and choose the number that gives the best result on the development set. The chosen hyperparameters for each model is reported in Table 6.

For inference, we experiment with the number of input passages $n=\{40,80\}$ and choose the best one on the development set for testing. We restrict the predicted span to be a Freebase entity string on WebQuestions, following Chen et al. (2017).

A.2 Details for baselines with passage concatenation.

We design two baselines which concatenate a passage pair.

First, ParReader++ (pairs from graph) concatenates passage pairs that are related in the input graph, along with the relation text. Specifically, if $p_{i}$ and $p_{j}$ are connected through $r_{i,j}$, all of $p_{i}$, $p_{j}$, $p_{i}:{\tt{[SEP]}}:r_{i,j}:{\tt{[SEP]}}:p_{j}$ are included as input passages of ParReader++. We limit the length of the passages and the relation text to be 145 and 10, respectively, so that the total length to be up to $300$. The number of the final input passages to the model will be $n+|\{r_{i,j}|r_{i,j}\neq\text{none}\}|$, where $0\leq|\{r_{i,j}|r_{i,j}\neq\text{none}\}|\leq\frac{n(n-1)}{2}$ (but typically much smaller than $\frac{n(n-1)}{2}$ as the input graph is very sparse).

Similarly, ParReader++ (all pairs) concatenate all passage pairs. For all $1\leq i<j\leq n$, $p_{i}$, $p_{j}$, $p_{i}:{\tt{[SEP]}}:p_{j}$ are included as input passages of ParReader++. As $p_{i}$ and $p_{j}$ may not have a relation, the relation text $r_{i,j}$ is omitted as an input. Again, the length limit for $p_{i}$ and $p_{j}$ is 145. The number of the final input passages to the model will be $n+\frac{n(n-1)}{2}$. As there are too many input passages for this baseline, we use $n=20$ instead of $n=\{40,80\}$.

For both baselines, as the limit for a single passage is different from other baselines (145 vs. 300), we run the retrieval again by following the same method but just split the article into passages with a different length limit.

Appendix B More qualitative analyses

Figure 4 depict more examples where our model predicts the correct answer. We describe how our model outperforms baselines for each of retrieval and reading component.

In Example 1, text-match retrieval fails to retrieve the evidence passages because it fails to capture ‘All That’ as a key entity. GraphRetriever, on the other hand, retrieves the article “All That” by entity linking. It is worth to note that this was particularly common, when the key entities are composed of common words that TF-IDF does not capture its importance, e.g., “Who plays letty in bring it on all or nothing?” or “Who sings does he love you with Reba?”.

In Example 1, entity linking was not enough for evidence to answer the question, because the article “All That” does not contain the singer of the theme song. Meanwhile, WikiData contains a triple $<$“All That”, composer, “TLC”$>$, allowing the retrieval of the passage “TLC”.

In Example 1, although GraphRetriever retrieves the evidence passage, ParReader++ which does not leverage the graph information predicts the wrong span by choosing the first person name in the passage from “All That”. GraphReader, however, leverages the relation composer and predicts the correct answer.

In Example 2, although the question appears to by easy for humans, ParReader++ retrieves “China” as an answer, potentially because it is the only country name mentioned in retrieve passages from the article “Nike, Inc.” However, GraphReader leverages the relation country and predicts the correct answer.

Example 3 requires to reason across multiple passages, as it asks about the first advent of the movie series that have similar titles. ParReader++, which reads each passage in isolation, predicts “2010” from the wrong passage. GraphReader, however, incorporates relations followed by and follows and successfully distinguishes the first movie.