NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned

Introduction

Open-domain question answering (QA) is emerging as a benchmark method of measuring computational systems’ abilities to retrieve, represent, and read knowledge (Voorhees and Tice, 2000; Chen et al., 2017; Seo et al., 2019; Lee et al., 2019). Recently, this task has been addressed by a diverse set of approaches that navigate multiple documents (Min et al., 2019b; Asai et al., 2019), index large corpora of text (Lee et al., 2019; Guu et al., 2020; Karpukhin et al., 2020), represent world knowledge in the parameters of a neural network (Roberts et al., 2020), or consolidate knowledge from multiple passages (Izacard and Grave, 2020). More comprehensive background is provided in Chen and Yih (2020).

The EfficientQA competition, held at NeurIPS 2020, required contestants to build self-contained systems that contain all of the knowledge required to answer open-domain questions. There were no constraints on how the knowledge is stored—it could be in documents, databases, the parameters of a neural network, or any other form. However, the competition encouraged systems that store and access this knowledge using the smallest number of bytes, including code, corpora, and model parameters. Specifically, EfficientQA had four tracks: 1) best accuracy overall (unconstrained); 2) best accuracy, system size under 6GiB; 3) best accuracy, system size under 500MiB; 4) smallest system to get 25% accuracy. These memory budgets were designed to encourage contestants to explore the trade-off between storing and accessing large, redundant, retrieval corpora, structured data stores, or the parameters of large learned models.

This paper summarizes the findings from the competition. Section 2 describes the competition in detail; Section 3 presents a description of the best performing submissions in each track; Section 4 introduces a new human evaluation of system accuracy, and Section 5 provides results and an analysis based on automatic and human evaluation. Finally, Section 6 pits the top systems against human trivia experts.

The top submissions in each of EfficientQA’s four tracks significantly outperformed the provided baselines. All top submissions use a retrieval corpus and a neural network answering module. However, the nature of the retrieval corpus and answering module differs drastically across the tracks (Table 1).

The top submissions to the unrestricted track and the 6GiB track (Section 3.1; 3.2) outperformed the state-of-the-art baselines from April 2020 by nearly 20%. They achieved this improvement by combining the state-of-the-art retrieval systems (Karpukhin et al., 2020; Mao et al., 2020) with answer generation (Izacard and Grave, 2020); leveraging the state-of-the-art in text generation (Raffel et al., 2020) and text encoding (Clark et al., 2020); modeling not only text but also tables and lists from Wikipedia; and combining the extractive and generative answer prediction. The top submissions to the 6GiB track additionally massively reduced the size of their indexed corpus and made use of the state-of-the-art in compression, with minimal impact on accuracy.

MiB and smallest tracks

To get under 500MB (Section 3.3), the systems made more drastic changes. The submission from NAVER Clova drastically reduced the size of their indexed corpus and reused a single Transformer model for the retriever and the reader, winning the 500MiB track according to the human evaluation. The even smaller UCLNLP-FB system took a novel approach in generating a large corpus of question-answer pairs, indexing it, and retrieving the most similar question to the input question. This approach, with two systems with different sizes in question-answer corpus, won both the 500MiB track and the smallest 25% track according to the automatic evaluation.

Automatic vs. human evaluation

The human evaluation supports the observation that automatic metrics often incorrectly penalize correct predictions in the open-domain setting (Voorhees and Tice, 2000; Roberts et al., 2020). We also investigate the effect of question ambiguity on evaluation—the questions from NQ are often ambiguous without the associated evidence document (Min et al., 2020). In Section 4 we define an annotation scheme that supports multiple estimations of accuracy, corresponding to different definitions of correctness for answers to ambiguous questions. Almost all systems’ accuracy increased by 20%-25% under the strictest definition of correctness. The increase doubled when we relaxed the definition of correctness to permit any semantically valid interpretation of the question. We present a discussion as well as suggestions for future evaluation (Section 5.2).

Competition Overview

The competition uses English questions and answers from the Natural Questions dataset (Kwiatkowski et al., 2019, NQ): real user questions issued to the Google search engine, along with reference answers from WikipediaNQ was collected over the course of 2017 and 2018. Most solutions to the open-domain variant use a Wikipedia dump from December 2018. See Lee et al. (2019) for details.. Real user questions in NQ are interpretable outside of a document context (unlike SQuAD (Rajpurkar et al., 2016)), and less amenable to traditional IR approaches than over-complete trivia questions (Joshi et al., 2017), as discussed in Lee et al. (2019). While the original NQ task was posed as a reading comprehension task, in which evidence documents are provided, recent work has adapted the NQ data to the open-domain setting (Lee et al., 2019; Min et al., 2019a, b; Asai et al., 2019; Guu et al., 2020; Roberts et al., 2020; Karpukhin et al., 2020) by taking examples with short answers (up to five tokens) and discarding evidence documents. In the open-domain setting, NQ contains 88k one-way annotated training examples and 4k five-way annotated development examples. For EfficientQA, we introduce a new test and development set constructed in the same way as the original NQ, but labeled slightly after (early 2019 rather than through 2018).This temporal change caused slight degradation in performance for baseline systems. Our test set was kept hidden from contestants, and submissions were made by uploading solutions to an automatic leaderboard.

System size

All submissions to the three restricted tracks were submitted as self-contained Dockerhttps://www.docker.com images. We defined system size to be the on-disk, at-rest size of this image. We chose this approach to avoid confusion about distinctions between data, model parameters, and code. However, this choice did lead to a significant engineering efforts for the very smallest systems, which were not able to build on the standard Docker templates for the predominant deep-learning libraries. While the organizers did provide a small reference system based on T5 (Roberts et al., 2020) that used TensorFlow servinghttps://www.tensorflow.org/tfx/serving/docker, this did not support most submissions, and the very smallest systems required clever compilation strategies on top of their modeling enhancements.

Evaluation metrics

We measured the performance of different systems through automatic and human evaluation. For automatic evaluation, the accuracy of each systems’ predicted answers is judged against reference annotations, annotated by five human workers. We follow the literature in using exact match between predicted and reference answers after minor normalization (Lee et al., 2019). Due to the ambiguities inherent in language and question-answering in general, five reference answers are often not exhaustive, and systems predict correct answers that are judged incorrect according to the automatic metrics. To rectify this, we sent predictions from each of the systems for further human rating by three raters, to get a better estimation of accuracy, as detailed in Section 4.

Competition schedule

The competition was announced in June 2020, along with baselines and tutorials. The official leaderboard for restricted settings was launched on September 14th, 2020, and participants had two months to submit their systems (November 14th, 2020). Finally, predictions from the top systems from each leaderboard were sent for human evaluation, which was completed in the end of November. The same set of top systems was invited to present their systems at the NeurIPS event on December 12th and in this paper. The human vs. computer competition was held on December 6th (details in Section 6).

In total, we had 39 submissions from 18 unique teams, seven of which were affiliated or co-affiliated with universities.

Systems

We describe a set of provided baselines and systems from participants. Systems from participants include the top 1–3 submissions per track based on automatic measure, considering margins between the submissions. Table 1 summarizes the key distinctions between the systems, and Figure 1 shows the memory footprint of each system component.

color=magenta]EC: how many teams participate in this track? Both of the top two submissions make use of the previous state-of-the-art system, such as DPR (Karpukhin et al., 2020) and a generative reader (Izacard and Grave, 2020). They are greatly enhanced by a better training objective, aggregating the answers from the extractive and generative models, or incorporating lists and tables from Wikipedia.

Both REALM (Guu et al., 2020) and DPR (Karpukhin et al., 2020) use the retrieval-reader framework. They retrieve the top $K$ passages from Wikipedia by mapping the question and each passage to dense vectors and employing a maximum inner product search. The top $K$ passages are then fed into an extractive reader which predicts the start and the end position of the answer. A key distinction between REALM and DPR is that REALM is trained with a self-supervised, joint objective, while DPR is trained in a pipeline manner with gold positives and distantly supervised negatives.

MS UnitedQA

The UnitedQA system consists of three components: Retrieval, Reading, and Re-ranking. First, it uses DPR to fetch top 100 passages from the English Wikipedia dump for a given question. Second, a hybrid approach, combining both generative and extractive readers, is used to produce answer candidates from the collection of retrieved passages. The generative reader is Fusion-in-Decoder (Izacard and Grave, 2020) based on T5 (Raffel et al., 2020), and the extractive reader is based on ELECTRA (Clark et al., 2020). Several techniques are applied to facilitate the training of the readers: posterior differential regularization (PDR, Cheng et al. (2020b), improved loss-term (Cheng et al., 2020a) are explored for the extractive model, and adversarial training (Ju et al., 2019) approach, attention bias (Lewis et al., 2020a) for the generative model. At the final re-ranking stage, it combines the generative and extractive model predictions with linear interpolation, and produce the final answer. More details of the system can be found in Cheng et al. (2021).

FB Hybrid

This system uses a retriever-reader architecture. The retriever is a combination of DPR and Generation Augmented Retrieval (GAR) (Mao et al., 2020), where each of them retrieves 50 passages. The DPR encoders are trained iteratively, where better hard negatives are mined at each step using the model at the previous step—two iterations are found to be sufficient. Its dense index includes lists and tables, as well as regular text passages. Specifically, 455907 tables and infoboxes are processed from the NQ training corpus using a trivial linearization, which concatanates the text representation of each row with a newline character.This linearization is found to be more effective than converting each row to a statement. Tables are chunked at 100 tokens with the header included in each chunk. The reader is Fusion-in-Decoder based on T5-large, which is given 100 passages from the retriever and generates an answer. More details of the system are in Oguz et al. (2020).

2 6GiB track

For this track, we provided two baselines. One retrieval-based, and one generative model. However, all of the top three submissions are retrieval-based, and reduce the memory footprint by drastically pruning the Wikipedia corpus, either by a learned model or based on a page view. They also made use of model quantization and the state-of-the-art in compression to limit their system’s footprint. They additionally enhanced the previous state-of-the-art as done in an unrestricted track.

We create a variant of DPR with pruned Wikipedia. Specifically, we only keep passages from the pages that are paired with the questions on the training set of NQ, reducing the number of passages from 21M to 1.6M.

Generative baseline: T5-XL+SSM

T5 (Raffel et al., 2020) is a text-to-text Transformer language model utilizing an encoder-decoder architecture that was pre-trained using a “span corruption” objective on Common Crawl. This approach is based on Roberts et al. (2020) which demonstrated a “closed-book” setting where the model was only able to access the knowledge stored in its parameters after fine-tuning on the task. Following Roberts et al. (2020), the model is additionally pre-trained using a “salient span masking” objective (Guu et al., 2020) before fine-tuning. The XL model has approximately 3B parameters.

FB system

This system is based on a retriever-reader approach. The retriever is an ensemble of a dense retriever and GAR. The dense retriever is initialized from BERT-base and is trained by distilling the cross-attention scores of the reader. GAR (Mao et al., 2020) is a retriever with generative query augmentation, based on BM25. The Lucenehttps://lucene.apache.org/ index is built on the fly. The reader is Fusion-in-Decoder initialized from T5 large.

The text corpus initially included 26M passages including the plain text and the lists, but went through an article-level filtering to include 18.8M passages. Specifically, it uses a linear classifier where each Wikipedia article is represented by its title and list of categories.

The model weights are stored using float16, taking 1.6GB of memory. The dense index is compressed by relying on three strategies, described in Izacard et al. (2020): 1) the vector representations go through dimension reduction from 768 to 256, 2) they are further quantized through product quantization, using 2 bits per dimension, and 3) the text of Wikipedia is compressed using lrziphttps://github.com/ckolivas/lrzip.

Ousia-Tohoku Soseki

Soseki is an open-domain QA system that adopts a two-stage approach consisting of Retriever for passage retrieval, and Reader for reading comprehension. Given a question, Retriever obtains top-k candidate passages from Wikipedia. Then, Reader selects the most relevant passage from them and extracts an answer from the passage.

Retriever is based on a re-implementation of DPR, where two BERT-base-uncased models are used to embed questions and passages. We also quantize passage embeddings to reduce the system size. Embeddings of Wikipedia passages are precomputed and stored using Faiss (Johnson et al., 2017). Reader is a reading comprehension model based on ELECTRA-large (Clark et al., 2020). We added one dense layer for selecting the most relevant passage, and two dense layers for detecting the start and end positions of the answer. All modules are trained on the NQ dataset.

To reduce the system size under 6GB, we compressed the models, passages, and other data files using ZPAQhttp://mattmahoney.net/dc/zpaq.html and excluded Wikipedia passages with less than 40 monthly page views, resulting in 18M passages. These tricks do not drop the accuracy.

BUT R2-D2

R2-D2 is composed of a dense retriever, a re-ranker and two readers. The dense retriever is based on RoBERTa (Liu et al., 2019) and is trained via an objective known from DPR. It retrieves $K=400$ passages from a pruned version of Wikipedia, reduced from 21M to 1.6M passages. Pruning is done via a simple binary classifier based on RoBERTa trained on the data created from the golden passages and negative passages randomly sampled from the index. This classifier obtains 90.2% accuracy. The re-ranker (based on Longformer (Beltagy et al., 2020)) concatenates retrieved passages, assigns a score for each passage, and selects $V=20$ passages. The extractive reader (based on ELECTRA) reads $V$ passages in a similar way as in Fajcik et al. (2020). It is trained via the marginal likelihood objective combined with the compound objective. The generative reader follows the Fusion-in-Decoder schema and generates the answers. R2-D2 aggregates the output from these two readers using two fusioning methods. First, it reranks the top spans from the extractive reader by feeding them to a generative reader and combines its log likelihood with the log likelihood from the extractive reader through a linear combination. They are then further aggregated with abstractive answers from the generative reader that are generated independently, through another linear combination. The parameters are stored in float16 and compressed using ZIP. More details of the system are in Fajcik et al. (2021).

3 500MiB track

The top performing submissions in this track made more drastic changes to get under 500MiB. They have completely different approaches from each other.

This is a smaller version of the T5-XL+SSM baseline above, with 512 hidden dimension instead of 4096, 6 layers instead of 24, and 8 attention heads instead of 64. It contains 60M parameters in total.

UCLNLP-FB system

This system is based on the approach from Lewis et al. (2020c), consisting of a database of question-answer pairs and a retriever which returns the answer of the most similar stored question to an input question. This approach is attractive as it performs well using low parameter-count models, and less space is needed to store question-answer pairs than full Wikipedia passages.

The Question-Answer Pair Generator is similar to Alberti et al. (2019); Lewis et al. (2019). First, a passage selection model $P(c)$ is trained to identify appropriate passages from Wikipedia. For high-probability passages (w.r.t. $P(c)$), the system performs Named Entity Recognition to extract likely answers $a$, and generates questions $q$ from $(c,a)$ using a BART (Lewis et al., 2020b) model $P(q\mid a,c)$ fine-tuned on NQ. Finally, the system filters question-answer pairs with a “global consistency filter”—if an open-domain QA system (Izacard and Grave, 2020) is given $q$ and generates an answer that is consistent with $a$, the $(q,a)$ pair is added to the database. This is because generated $(q,a)$ pairs are sometimes incorrect or unanswerable without the passage they were generated from. The final database consists of NQ, the EfficientQA development set and 2.3M generated question-answer pairs. The Retriever consists of TF-IDF and a bi-encoder retriever based on ALBERT-base (Lan et al., 2020). The Reranker is a cross-encoder module based on ALBERT-large; it reranks the top-10 question-answer pairs from the retriever, and the answer of the top-1 question-answer pair is chosen as the final answer.

The system is further compressed using TFLite, quantization, and Alpine Linux. More details can be found from Lewis et al. (2021).

NAVER RDR

RDR is a lightweight retrieve-and-read system, consisting of a single Transformer (MobileBERT (Sun et al., 2020)) that performs both retrieval and reading, an index of dense passage vectors and the filtered Wikipedia corpus. The Transformer serves as a dense retriever as DPR does with two differences: (1) a single Transformer is used to encode both the question and the passage, and (2) the embeddings are defined as the first 128 out of 512 dimensions of the output vectors from the Transformer that correspond to the [CLS] token. The same Transformer serves as an extractive reader by producing the scores of the start and the end of the answer span as well as the passage reranking score. A distillation technique is used for training, by minimizing the KL divergence between its start, end, and reranking scores and those of a fully trained DPR reader. The Transformer is further finetuned in an iterative manner as a retriever and as a reader.

The index consists of 1.2M 128-dimensional INT8 vectors (scalar-quantized and unsigned), which are the dense embeddings of the subset of the 21M passages in Wikipedia, filtered by a RoBERTa (Liu et al., 2019)-based binary classifier trained with logistic regression to exclude uninformative passages. Positives to train this classifier are the top 200 passages for each question on NQ dataset and EfficientQA development set, retrieved by Yang and Seo (2020), a DPR retriever further finetuned on hard negatives using knowledge distillation from a DPR reader. Negatives are uniformly drawn from the set of 21M passages, excluding positives. More details can be found from Yang and Seo (2021).

4 25% smallest track

Among the provided baselines, the smallest system with an accuracy of over 25% is T5-XL+SSM, the same system described in Section 3.2. This system is 5.65GB and achieves an accuracy of 28%.

UCLNLP-FB system (29M)

This system is the same system as UCLNLP-FB system in the 500MiB track with following minor modifications, to further decrease the memory: (1) the retriever is just TF-IDF instead of a combination of TF-IDF and a bi-encoder, (2) the reranker uses ALBERT-base instead of ALBERT-large, and (3) there are 40k generated question-answer pairs, instead of 2.3M.

Human Annotations of Correctness

The EfficientQA development and test sets have up to five reference answers per question. Due to the variability of language, these five answer strings are often not exhaustive, and systems predict correct answers that are judged incorrect according to the automatic metrics. To rectify this, we sent predictions from each of the systems in Section 3 for human rating to get a better estimation of accuracy.

Each system prediction was sent for rating by three separate annotators: 1) the annotator first works on understanding the meaning and intent of the question (with a web search if necessary). 2) The annotator then determines whether the question is ambiguous, i.e., whether the question can lead to multiple different answers depending on factors such as: when the query was asked; where the query was asked; some unspoken intent of the questioner; or the opinion of the person giving the answer. 3) Finally, the annotator determines whether each answer is “definitely correct” (correct given a usual interpretation of the question), “possibly correct” (could be correct, given some interpretation of the question), or “definitely incorrect”.

Since the original NQ data was collected more than a year before the start of the EfficientQA competition, the denotation of some questions may have changed over time (e.g. “who won the last season of bake-off”). Rather than determine a single correct point in time for these questions, we asked our annotators to assume that the query could have been asked at any time since the web has existed color=cyan]JBG: when was this, to be precise? and choose the “possibly correct” label for answers that may or may not have been correct when the question was asked.

The final rating is an aggregation of ratings from three annotators: if at least 2/3 raters determined it to be “definitely correct”, the label is “definitely correct”. If at least 2/3 raters determined it to be either “definitely correct” or “possibly correct”, the label is “possibly correct”. The pairwise agreements of the human ratings are 69.2% (Cohen’s $\kappa=53.8$) for 3-way ratings, 85.7% (Cohen’s $\kappa=71.4$) for whether the prediction is definitely correct or not, and 76.7% (Cohen’s $\kappa=53.4$) for whether the prediction is possibly correct or not.An agreement in whether the question is ambiguous or not is 61.3% (Cohen’s $\kappa=22.6$). We conjecture the low agreement is due to an intrinsic difficulty in finding ambiguity (Min et al., 2020), and consider the question to be ambiguous if at least one annotator rated it as ambiguous.

Results & Analyses

All of the five systems in the unrestricted track and the 6GiB track significantly outperform the state-of-the-art (Table 2) at the beginning of the competition—DPR (36.6%) and REALM (35.9%). Systems in the 6GiB track approach the unrestricted track’s accuracy; for instance, the accuracy FB system is comparable to the accuracy of the top systems in the unrestricted track. The improvements in the 500MiB track are also impressive; both the top two systems significantly beat T5-small (17.6%).

Human raters find 13% and 17% of the predictions that do not match the reference answers to be definitely correct or possibly correct, respectively, overall increasing the accuracy of the systems. Most systems showed 17–25% and 41–54% improvement in accuracy when using definitely correct and possibly correct human evaluation respectively, compared to automatic evaluation metric which only consider exact string match to existing reference answers. An exception is NAVER RDR, which achieves significantly larger improvements (32% and 71%, respectively). We also found that when the gap in automatic measure between systems is marginal (around or smaller than 1%), human evaluation may change the rankings between the models.

Agreement between system predictions

Figure 2 (left) shows the agreement between system predictions, based on exact match in automatic evaluation. The largest agreement is made between FB Hybrid and FB system, likely because they are both based on DPR and Fusion-in-Decoder. Agreements between systems in the unrestricted and the 6GiB tracks are generally higher, likely because they are all based on retrieval-reader framework and pruning Wikipedia does not hurt too much. Two systems in the 500MiB track have smaller agreement with the other systems, and agree with each other even less.

Ensemble oracle accuracy of the systems

Figure 2 (right) reports the ensemble oracle accuracy for each system pair, which considers a prediction to be correct if either system prediction is correct. The FB Hybrid & Ousia-Tohoku Soseki pair achieves the highest ensemble oracle accuracy, indicating that their system predictions are substantially different from each other compared to other pairs with top performing systems.

2 Analyses

color=cyan]JBG: Could we have a more informative title?

We present an analysis of the 50 sample questions where at least one prediction does not match with the gold answer, but is judged as correct by human raters, with a “definitely correct” label or a “possibly correct” label, respectively. The samples are largely divided into three classes: as valid as gold (judged as either definitely correct or possibly correct), valid but not the best (judged as possibly correct), and closer to invalid (judged as possibly correct). We describe fine-grained categories and their percentage The total is over 100%; one question may have multiple predictions that fall into different categories. ( definitely correct and possibly correct) here, with examples shown in Appendix A (Table 7).

The following describes categories on predictions that are as valid as the gold answers.

Semantically the same ( 60%, 22%): The prediction is semantically equivalent to the gold, e.g., “about 930 BCE” and “around 930 BCE”.A few cases in this category are due to inconsistent tokenization (e.g., “Mercedes-Benz” and “Mercedes - Benz”), which could potentially be automatically captured but is not handled by the automatic evaluation from the literature.

Open-ended question ( 6%, 4%): There is a large set of distinct, plausible answers, mainly because the question is vague or open-ended.

Ambiguous entity/event references ( 20%, 20%): There is a set of distinct answers because the question contains ambiguous references of entities or events. For instance, “Gold woman” in the example in Table 7 may refer to the fictional character (“Ayesh”) or the actress (“Elizabeth Debick”).

Different granularity ( 14%, 6%): The prediction and the gold answer have different granularity, such as the year (1982) vs. the month (October 1982), or the city (Pawtucket) vs. the state (Rhode Island).

Incorrect gold ( 2%, 6%): The gold annotation is incorrect.

The following describes categories on possibly correct predictions that are as valid but not the best answers.

Ambiguous answer type ( 20%): There is a unique entity or event that can be the answer to the question, but there is ambiguity on which exact text should be presented as the answer. For instance, both the episode title (“Somber News”) and the air date (“February 23 2013”) are the valid answer to the question in Table 7.

Answer is time-dependent ( 18%): The answer depends on the time of the question being asked. Questions in this category usually involve recurring events such as sports games or elections.

Finally, we present categories for predictions that are closer to invalid answers.

Conflicting information in Wikipedia ( 4%): While the gold is the only valid answer, the English Wikipedia contains incorrect information that supports that prediction is the answer to the question. Consider the question in Table 7. While “Fort Hood” is an incorrect answer by a fact, the Wikipedia page https://en.wikipedia.org/wiki/Fort_Hood states “Fort Hood is the most populous U.S. military installation in the world.”At the human vs. computer competition, the human team also predicted “Fort Hood”.

Plausible only in certain conditions ( 4%): Prediction may be valid in certain conditions, but is incorrect in general cases. For instance, for the question in the table, “president of India” may only be valid in India.

Mismatch with question intent ( 8%): The prediction is somewhat valid but supposedly not the intended answer to the question. For instance, the example question in Table 7 is supposedly asking for the answer that is different from the great depression.

Incorrect prediction ( 2%): The prediction is definitely incorrect.

Our two main takeaways are as follows. First, the automatic evaluation confirms the observation of Voorhees and Tice (2000) that it is insufficient in capturing semantically equivalent answers, which are responsible for 60% of the definitely correct predictions. Second, ambiguity arises frequently in the questions in different levels, allowing multiple, semantically different answers to be valid. This is a consistent with Min et al. (2020), which reported that around half of the questions in NQ contain ambiguity, due to ambiguous references of entities, events or properties, or time-dependency of the answer. Based on our human evaluation, annotations on ambiguity have low agreement rate (61.3%, Cohen’s $\kappa=22.6$), and predictions with the same level of plausibility are often marked as “definitely correct” or “possibly correct” by different human raters. We note the notions of “correctness” and “plausibility” are not binary, and are instead often dependent on pragmatic interpretation of the questioner’s intent. For example, the question “who has the most superbowl rings” could be read as “which person (including coaches) has the most superbowl rings”, “which player has the most superbowl rings”, “which team has the most superbowl rings”. All three annotators identified this question as being ambiguous but they disagreed about the validity of the different readings. The three raters were split three ways when rating the correct answer (“Pittsburgh Steelers”) for the last interpretation. Meanwhile there were no “incorrect” ratings, and 2/3 “definitely correct” ratings given to the correct answer (“Tom Brady”) for the second interpretation, despite the fact that two coaches have more superbowl rings. Clearly, the annotators are applying some personal interpretation of the questioner’s intent and answer plausibility.

While we believe that many real world questions do require some non-literal assumptions about the questioner’s intent, and we believe that the natural language processing community should not shy away from that task, we also acknowledge that there is work to be done in creating better, non-binary, definitions of correctness. Section 6 contrasts these interpretations of correctness with the more rigid definition used by the Trivia community.

To better understand the effect of ambiguity on the ranking of different solutions, we also evaluate system performance on a subset of the questions that are unambiguous. We define unambiguous questions to be those that (1) have at least three out of five reference answers contain valid short answersThis follows the original NQ approach of using annotator agreement to set a threshold for high quality answers., and (2) are not labeled as ambiguous by any of three human raters, resulting in 51.5% of the original set. Table 3 shows human evaluation on the original set and this subset of unambiguous questions. Most systems, except UCLNLP-FB system, achieve higher accuracy on unambiguous questions, with the first three systems achieving over or near 80%. Unsurprisingly, the gap between “definitely correct” accuracy and “possibly correct” accuracy is marginal on unambiguous questions.

Importantly, the overall rankings are unchanged when we restrict our evaluation set to only unambiguous answers. This suggests that, while question ambiguity may lead to disagreement between annotators at a per-example level, it is not adversely impacting our ability to consistently rank solutions. More analyses can be found in Appendix A.

Trivia Experts vs Computer Systems

The questions in NQ are posed by humans to computers, and the competition attracted some of the strongest and most efficient QA systems available today. However, humans also answer questions for fun and recreation (Jennings, 2006) and the ultimate goal of artificial intelligence is to create machines that answer questions as well as humans (Turing, 1995, known as the Turing test). Moreover, existing comparisons of human question answering ability often use unskilled humans (Rajpurkar et al., 2016), leading to claims of computers “putting millions of jobs at risk” (Cuthbertson, 2018). Or, in competitions with trivia experts (Ferrucci et al., 2010), arcane rules of competitions can tilt the playing field toward computers (Boyd-Graber and Börschinger, 2020) or use unnatural questions (Boyd-Graber et al., 2012; Rodriguez et al., 2019).

We advertised our competition to trivia enthusiasts on social media. Teams of up to eight players applied to be part of the competition. We selected five teams to participate in the preliminary competition (results in Section 6.2).

To create a fair competition and to showcases all of the tiers of the efficient QA competition, we offered three ways to answer each question where either humans or computers have more resources to answer a question (Table 5).

To complement the 500MiB systems, humans had to instantly signal when they knew the answer to a question. This reflects instant recall of a fact by a single individual. In the next phase in competition with the 6GiB systems, both humans and computers had more resources: the human team could discuss the answer for thirty seconds, arguing why they believe their answer is correct and computers had over ten times the memory. Finally, to focus one reading comprehension, unlimited systems faced off against the human teams who also had access to snippets from search results using the question as a query. As with the previous phase, they have thirty seconds to discuss their answer.

We selected questions for the human eval based on the following criteria:

Diverse over topic, ensuring there were questions about history, literature, philosophy, sports, and popular culture. This results in fewer questions about sports and popular culture than the standard NQ distribution.

Not tied to 2018; we excluded questions that depend on the date being asked.

Interesting questions. While not strictly adversarial, we wanted to showcase both human and computer ability, so we excluded questions that many humans would not know (e.g., “how many us states are there”) or questions with answers that are difficult to evaluate in the NQ framework (“how many words are in Les Misérables?”).

To avoid the issues described in Section 4, we avoided questions that were overly ambiguous (answer changes based on time the question was asked, unclear answer type, mismatch with question intention, etc).

Thus, for the human competition, we exclude the broader interpretation of correct adopted in Section 5.2 to a standard we called “actually correct”, consistent with traditional trivia interpretations Jennings (2006). A human judge researched all of the answers to the questions and evaluated whether an answer was correct or not. This allows for more permissive answers lines such as “to pass away” to be accepted for questions like “what does it mean to cross over the rainbow bridge” even though the answer line only lists “heaven”. Again, consistent with trivia best practice, we only required full names in the case of confusion (e.g., “Theodore Roosevelt” or “Franklin D. Roosevelt” instead of just “Roosevelt”).

2 Preliminary Competition

To select which of the human teams faced off against the top computers in the final competition and to let the humans teams practice this unconventional format, we had the human teams face off against the baseline systems: T5 (500MiB), DPR (6GiB), and REALM (Unlimited).

We set aside an hour for each team. Because better teams can get through more questions, to normalize comparisons per team, we computed the average margin per question (Table 5). Only two teams had higher scores than the computer baseline teams. The top team (Team B), which had multiple professors and trivia champions was clearly the best human team on this set of questions.

3 Final Game

The winning team (Team B) went up against the winning computer systems in a final match with fifty questions. Most questions were answered in Phase 1, and those that were not were typically answered in Phase 2 (Figure 3 (right)). Only four questions reached Phase 3; these questions that stumped both humans and computers:

Who said there are old pilots and there are bold pilots

What is the second largest city in ethiopia

Of these questions, only the second largest city in Ethiopia was answered correctly (the humans were working through a list of cities in Ethiopia they knew). The other questions represent questions that cannot be answered by Wikipedia (“old pilots”), questions that require language capabilities beyond question answering (“kimi no na wa”), or that have a more nuanced answer than NQ can provide. For example, on the questions “who said i’ll make mincemeat out of you”, the human team reasonably thought that the question meant who originated the phrase, while NQ’s answer was the cartoon character who is most associated with the phrase (Klondike Kat).

4 Reflection

For question answering, there are multiple interpretations of the phrase “human evaluation”. Human evaluation of answers are important for exposing problems in the dataset, revealing ambiguity, and measuring whether the answers are useful. However, for the ultimate questions of where we have achieved artificial intelligence (Turing, 1995), we need fair comparisons with skilled humans. Moreover, to measure whether question answering systems are useful for people, we need to create socio-technical systems where humans and computers can answer questions together (Feng and Boyd-Graber, 2019). More importantly, trivia games are fun. They help illustrate the strengths and weaknesses of the underlying dataset and the QA methods with a spoonful of sugar to make otherwise dull evaluations more interesting.

Conclusions and Future Work

color=cyan]JBG: This seems to repeat a lot of what’s been said before. Perhaps we can trim here or add new observations? color=BurntOrange]Sewon: Great point; maybe good to add discussions that are not too similar to discussions already mentioned? (or related to broader open-domain QA?)

The EfficientQA competition was held to encourage research in open-domain question answering that focuses on both accuracy and memory footprint. All top performing submissions have a retriever-reader framework. Submissions to the unrestricted and the 6GiB track are enhanced over the state-of-the-art that retrieves a subset of Wikipedia passages and employs the extractive and/or generative answer modules; systems in the 6GiB track additionally cleverly select a small subset of Wikipedia, only marginally sacrificing accuracy when combined with state-of-the-art compression. In more restricted tracks, systems explore novel approaches and achieve impressive improvements over the baselines. They either generate a large corpus of question-answer pairs and retrieves the closest question to the input question, or dramatically filter Wikipedia and use a single Transformer model for retrieval and answer extraction. Still, they are behind the unrestricted systems in performance by 20%, indicating significant room for improvements in memory-restricted settings.

A human analysis shows that automatic evaluations of QA systems are not sufficient for thorough evaluations. Human raters find 30% of the predictions that do not match reference answers but are nonetheless correct. This does not affect all systems equally: relative accuracy rises using definitely correct (between 18–32%) and possibly correct (between 42 and a whopping 71%). The rise is mainly due to automatic evaluation failing to capture semantically equivalent answers, time-dependence of the answers, or underlying ambiguity in the questions (Min et al., 2020).

Future work in efficient open-domain question answering should continue to explore the tradeoff between system size, accuracy, and abstaining (He et al., 2016; Rajpurkar et al., 2018). Moreover, it is important to continue to refine the quality of QA evaluation: not all annotation and not all annotators are created equal. Using trivia enthusiasts for annotation and human benchmarks is a fun and effective evaluation of relative computer QA ability. We would encourage future leaderboards to use some element of human verification in important evaluations (e.g., an annual bake off), removing ambiguity in questions (Min et al., 2020), crafting adversarial examples (Jia and Liang, 2017; Wallace et al., 2019; Dua et al., 2019; Bartolo et al., 2020), or evaluating whether a response is useful to a user (Fan et al., 2019; Feng and Boyd-Graber, 2019). color=magenta]EC: can we add a paragraph summarizing what resources will be released from the competition?

We thank all the participants for taking part and making this a successful competition. We thank Google for providing prizes for computer participants. Boyd-Graber is supported by nsf Grant iis-1822494. Any opinions, findings, conclusions, or recommendations expressed here are those of the authors and do not necessarily reflect the view of the sponsor.

References

Appendix A Analyses

Table 8 shows questions that are answered correctly by all seven systems (easy), or answered incorrectly by all seven systems (hard). A common feature of easy questions (Sugawara et al., 2018; Kaushik and Lipton, 2018) is that a sentence from Wikipedia provides an explicit support to the question. Such supporting sentences have high lexical overlap with the question and require little paraphrasing (e.g. “the highest population” vs. “the most populous”). Even when the question is not well-formed, all systems may find the correct answer if there is high lexical overlap between the question and the supporting sentence (e.g. the fifth example question in Table 8).

Many of the hard questions have their answers in the tables (e.g. the first question in Table 8). This likely makes them “hard” because all systems except FB Hybrid do not consider the tables, and even FB Hybrid may miss such cases. In the other cases, systems make mistakes even when there is the text that supports the answer (e.g. the next two questions in Table 8). Though the reason for the mistake would be different depending on the systems, we conjecture that such supporting sentences are harder to retrieve or do not support the answer as explicitly as other examples. Occasionally, the question is not well-formed, such as the last one in Table 8.

Error Analysis

We further present an error analysis of the top two systems in the 500MiB track: UCLNLP-FB system and NAVER RDR. We choose these two systems because their approaches differ significantly from each other and the 500MiB track fits the main motivation of the EfficientQA competition. We randomly sample 100 questions from the development data. 40, 33 and 27 questions are answered correctly by both systems, one of the systems, and none of the systems, respectively.

Table 9 shows the breakdown of the predictions from UCLNLP-FB system, where 50 questions are answered correctly, and the other 50 are not. A majority of the error cases (47 out of 50) is due to retrieving an incorrect question. Out of 47, 25 cases retrieve the question with a different topic from the input questions, many of which discuss different entities (e.g., “don’t it make you feel like dancing” vs. “you make me feel like dancing”). The other 19 cases retrieve the question that discusses the same topic or contains the same key entity, but is asking about different details. For instance, in the example in Table 9, the input question asks the amount spent for a film, while the retrieved question ask the amount made by the film. It is also worth noting that the system sometimes gets the correct answer from the retrieved question with a different meaning from the input question, e.g., “What republican is running for mayor of phoenix” vs. “Who is the current mayor of phoenix”, which have the same answer because the current mayor of Phoenix is Republican.

Table 10 shows the breakdown of the predictions from NAVER RDR, where 63 questions are answered correctly, and 37 are not. Failure due to pruning the gold passage is rare, being responsible for only 3%. More of the failure cases are due to missing the gold passage, either when choosing 80 passages through dense retrieval (12%), or during cross-attention re-ranking 80 passages to decide on the top 1 passage (15%). Finally, in 7% of the cases, the top 1 passage contains the gold answer but the system fails to extract the correct answer. The gold passages in this category have valid but implicit support of the answer, e.g., the last example in Table 10.