Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

Introduction

Datasets are increasingly driving progress in AI, resulting in impressive solutions to several question-answering (QA) tasks (e.g., Rajpurkar et al., 2016; Joshi et al., 2017). However, many of these datasets focused on retrieval-style tasks, where surface-level cues alone were usually sufficient to identify an answer. This has not encouraged progress on questions requiring reasoning, use of commonsense knowledge, or other advanced methods for deeper text comprehension. The challenge presented here, called ARC (AI2 Reasoning Challenge), aims to address this limitation by posing questions that are hard to answer with simple baselines.

The ARC Dataset consists of a collection of 7787 natural science questions, namely questions authored for use on standardized tests. Standardized tests have previously been proposed as a Grand Challenge for AI Brachman (2005); Clark and Etzioni (2016) as they involve a wide variety of linguistic and inferential phenomena, have varying levels of difficulty, and are measurable, motivating, and ambitious. However, making this challenge a reality is difficult, as such questions are difficult obtain (most examination boards release only limited practice tests to the public). For ARC we have addressed this through several months of extensive search and investigation.

In addition, to encourage focus on advanced phenomena, we have partitioned ARC into a Challenge Set (2590 questions), containing questions answered incorrectly by both a retrieval-based algorithm and a word co-occurrence algorithm, and an Easy Set (5197 questions), containing the remainder. For example, two typical challenge questions are:

Which property of a mineral can be determined just by looking at it? (A) luster [correct] (B) mass (C) weight (D) hardness A student riding a bicycle observes that it moves faster on a smooth road than on a rough road. This happens because the smooth road has (A) less gravity (B) more gravity (C) less friction [correct] (D) more friction

Both these questions are difficult to answer via simple retrieval or word correlation. For example, there are no Web sentences of the form “luster can be determined by looking at something”; similarly, “mineral” is strongly correlated with “hardness” (an incorrect answer option). Rather, they require more advanced QA methods. We provide more example questions, and a catagorization of the knowledge and reasoning types that the questions appeal to, in Tables 4 and 5.

To help the community to engage with this dataset, we are also releasing a science text corpus and two baseline neural models as part of the ARC challenge:

The ARC Corpus, containing 14M science-related sentences with knowledge relevant to ARC. A sampled analysis suggests the corpus mentions knowledge relevant to 95% of the Challenge questions. (Use of the Corpus for the Challenge is optional).

Three neural baseline models, DecompAttn, BiDAF, and DGEM, for QA. These are multiple-choice QA adaptations of three neural models: the decomposable attention model Parikh et al. (2016), a top performer on SNLI; Bidirectional Attention Flow Seo et al. (2017b), a top performer on SQuAD; and the decomposed graph entailment model, a top performer on SciTail Khot et al. (2018). While these score well on the Easy Set, they are unable to perform significantly better than a random baseline on the Challenge Set, illustrating its challenging nature.

This challenge differs from the Kaggle-hosted 2016 Allen AI Science Challenge Schoenick et al. (2017) in three important ways ARC includes the publically releasable subset of the Kaggle questions (about 60% of the Kaggle set, making up 43% of the ARC set).. First, the creation of a Challenge partition is to avoid scores being dominated by the performance of simple algorithms, and thus encourage research on methods that the more difficult questions demand. Second, we provide a science corpus along with the questions to help get started (use of the corpus is optional, and systems are not restricted to this corpus). Finally, the questions, corpus, and models are all publically available.

The paper is organized as follows. We first discuss related work, and then describe how the ARC dataset was collected and partitioned. We also provide an analysis of the types of problems present in the Challenge Set. We then describe the supporting ARC Corpus, and illustrate how it contains knowledge relevant to Challenge questions. Finally, we describe several baselines and their performance on ARC. Most notably, although some baseline algorithms perform well on the Easy Set (scoring up to 61%), none are able to perform significantly above random on the Challenge Set. We conclude by posing ARC as a challenge to the community. The ARC Dataset, Corpus, Models, and Leaderboard can be accessed at http://data.allenai.org/arc.

Related Work

There are numerous datasets available to drive progress in question-answering. Earlier reading comprehension datasets, e.g., MCTest Richardson (2013), SQuAD Rajpurkar et al. (2016), NewsQA Trischler et al. (2016), and CNN/DailyMail Hermann et al. (2015), contained questions whose answers could be determined from surface-level cues alone (i.e., answers were “explicitly stated”). TriviaQA Joshi et al. (2017) broadened this task by providing several articles with a question, and used questions authored independently of the articles. Again, though, the questions were largely factoid-style, e.g., “Who won the Nobel Peace Prize in 2009?”. Although systems can now perform well on these datasets, even matching human performance Simonite (2018), they can be easily fooled Jia and Liang (2017); the degree to which they truly understand language or domain-specific concepts remains unclear.

To push towards more complex QA tasks, one approach has been to generate synthetic datasets, the most notable example being the bAbI dataset Weston et al. (2015). bAbi was generated using a simple world simulator and language generator, producing data for 20 different tasks. It has stimulated work on use of memory network neural architectures Weston et al. (2014), supporting a form of multistep reasoning where a neural memory propagates information from one step to another (e.g. Henaff et al., 2016; Seo et al., 2017a). However, its use of synthetic text and a synthetic world limits the realism and difficulty of the task, with many systems scoring a perfect 100% on most tasks (e.g. Weston et al., 2014). In general, a risk of using large synthetic QA datasets is that neural methods are remarkably powerful at “reverse-engineering” the process by which a dataset was generated, or picking up on its idiosyncrasies to excel at it, without necessarily advancing language understanding or reasoning.

More recently, Welbl et al. (2017b) created the WikiHop dataset, containing questions that appear to require more than one Wikipedia document to answer (“multihop questions”). The dataset takes a step towards a more challenging task, but has several limitations: questions are binary predicates (e.g., date_of_birth(“jeanne c. stein”,?X)); the intended inference is typically a simple two-step chain (commonly a geographical substitution of a city for a country); and in many cases the correct answer can be guessed from the passage, without requiring multi-hop inference ($\sim$44% of the answerable questions are single-hop, according to the authors Welbl et al. (2017b)).

Datasets based on human standardized tests have also been used in AI for several years (e.g. Clark et al., 2016; Seo et al., 2014; Fujita et al., 2014; Strickland, 2013), and as part of the NTCIR QALab NII (2017), and for the 2016 Allen AI Science Challenge described in the Introduction Schoenick et al. (2017). However, there are two potentially significant challenges with these datasets. First, these datasets are often small (e.g., hundreds of questions), due to the scarcity of public, real-world test data. Second, because tests were designed for people rather than machines, large portions of these tests can be easily solved by simple AI methods (e.g. Davis, 2016; Clark et al., 2016). The result of this is that scores become dominated by the performance of simple algorithms (information retrieval, statistical correlations). This then biases research towards incrementally improving those algorithms, rather than exploring the larger AI challenges that the more difficult questions demand. Indeed, it is easy to mistake progress on these datasets as implying equal progress on easy and hard questions, while in reality progress may be heavily concentrated on easy questions alone Gururangan et al. (2018), leaving more difficult challenges unaddressed. The ARC Dataset addresses both of these limitations.

The ARC Dataset

The ARC dataset consists of 7787 science questions, all non-diagram, multiple choice (typically 4-way multiple choice). They are drawn from a variety of sources, and sorted into a Challenge Set of 2590 “hard” questions (those that both a retrieval and a co-occurrence method fail to answer correctly) and an Easy Set of 5197 questions. Table 1 summarizes the sizes of the train/dev/test partitions in the ARC dataset. The question vocabulary uses 6329 distinct words (stemmed).

Questions vary in their target student grade level (as assigned by the examiners who authored the questions), ranging from 3rd grade to 9th, i.e., students typically of age 8 through 13 years. Table 2 shows a break-down of the set based on grade level with absolute counts(#) and percentage(%) of the Challenge and Easy set. In practice, there is substantial overlap in difficulty among grade levels (also seen in the similar distribution of grade levels), as each grade level contains a mixture of easy and difficult questions.

Summary statistics of ARC are provided in Table 3, showing questions and answers vary considerably in length. Finally, Table 7 in the Appendix lists the variety of sources the questions were drawn from.

Operationally, we define a Challenge question as one that is answered incorrectly by both of two baseline solvers, described below. Although this only approximates the informal goal of it being a “hard” question, this definition nevertheless serves as a practical and useful filter, as reflected by the low scores of various baselines on the Challenge Set.

The first filter we apply is the IR solver from Clark et al. (2016), briefly described here for completeness. The IR solver uses the Waterloo corpus from Clark et al. (2016), a Web-based corpus of $5\times 10^{10}$ tokens (280GB). The solver searches to see if the question $q$ along with an answer option is explicitly stated in the corpus, and returns the confidence that such a statement was found. To do this, for each answer option $a_{i}$, it sends $q+a_{i}$ as a query to a search engine (we use Elasticsearch), and returns the search engine’s score for the top retrieved sentence $s$ where $s$ also has at least one non-stopword overlap with $q$, and at least one with $a_{i}$; this ensures $s$ has some relevance to both $q$ and $a_{i}$. This is repeated for all options $a_{i}$ to score them all, and the option with the highest score selected.

The Pointwise Mutual Information (PMI) Solver.

The second filter we apply is the PMI solver, also from Clark et al. (2016), again described here for completeness. This uses the same corpus as the IR solver, and formalizes a way of computing and applying associational knowledge. Given a question $q$ and an answer option $a_{i}$, it uses PMI or pointwise mutual information Church and Hanks (1989) to measure the strength of the associations between parts of $q$ and parts of $a_{i}$. Given a large corpus $C$, the PMI for two n-grams $x$ and $y$ is defined as

Here $p(x,y)$ is the joint probability that $x$ and $y$ occur together in $C$, within a certain window of text (we use a 10 word window). The term $p(x)p(y)$, on the other hand, represents the probability with which $x$ and $y$ would occur together if they were statistically independent. The ratio of $p(x,y)$ to $p(x)p(y)$ is thus the ratio of the observed co-occurrence to the expected co-occurrence. The larger this ratio, the stronger the association between $x$ and $y$.

The solver extracts unigrams, bigrams, trigrams, and skip-bigrams from the question $q$ and each answer option $a_{i}$. It outputs the answer with the largest average PMI, calculated over all pairs of question n-grams and answer option n-grams.

The Challenge Set

To illustrate the impact of using these algorithms as filters when defining the Challenge Set, consider the following example:

Which property of air does a barometer measure? (A) speed (B) pressure [correct] (C) humidity (D) temperature

The question was excluded from the Challenge Set because it is correctly answered by (here) both the IR and PMI algorithms (note that it would have been excluded even if it was answered correctly by just one of the solvers). The IR algorithm finds multiple sentences supporting the correct answer, e.g.,

Air pressure is measured with a barometer.

Air pressure will be measured with a barometer.

The aneroid barometer is an instrument that does not use liquid in measuring the pressure of the air.

A barometer measures the pressure of air molecules.

and similarly the PMI algorithm finds that “barometer” and “pressure” (and also “air” and “pressure”) co-occur unusually frequently (high PMI) in its corpus.

In contrast, consider the following question:

Which property of a mineral can be determined just by looking at it? (A) luster [correct] (B) mass (C) weight (D) hardness

This is incorrectly answered by both algorithms: There are no corpus sentences similar to “a material’s luster can be determined by looking at it”. Similarly, “mineral” co-occurs unusually frequently with several incorrect answer options (e.g., mass, hardness), confusing the PMI algorithm. This question is thus part of the Challenge Set, as it appears to require a more advanced answering method.

Question Types

ARC questions appeal to both different styles of knowledge, and different styles of reasoning. In Tables 4 and 5, we enumerate the broad classes of each that we observe in the ARC challenge, based on a sample of 100 questions. The relative sizes of these categories are shown in Figures 1 and 2. These sizes are necessarily approximate, as it requires a subjective judgement about the main challenge exhibited by different questions. Nevertheless, it helps to provide a rough atlas of the knowledge and reasoning space underlying ARC.

The ARC Corpus

In addition to the ARC question set, we are also releasing the ARC Corpus, a large corpus of science-related sentences mined from the Web. It contains 14M sentences (1.4GB of text), and mentions much of the knowledge required to answer the Challenge Questions. Although some of these mentions are indirect, and exploiting them is not trivial, it nevertheless provides a starting point for attacking the ARC Challenge. Note that use of the corpus is optional, and also that systems are not restricted to this corpus.

The ARC Corpus was created by utilizing a major search engine to run a large series of search queries relevant to science. Queries were automatically constructed by instantiating $\sim$100 hand-written templates for 80 science topics covered by US elementary and middle schools, the subject areas of ARC. For example, for the topic “celestial phenomena”, two templates “[astronomical-term] astronomy” and “[astronomical-term] astrophysics” were authored, and a list of (here, 360) terms for astronomical-term collected and used, resulting in 720 queries. The top several documents from each search were collected and de-duplicated, and then the content of these documents was stripped down to capture just the text in each document. The resulting text was then chunked into sentences. This was repeated for all templates. Note that some templates were parameterized by more than one parameter. From an informal analysis of a random sample of 805 documents that were collected, approximately 75% were judged as “science relevant”. The corpus was then augmented with the AristoMini corpusAlso available separately at http://allenai.org/data.html, an earlier corpus containing dictionary definitions from Wiktionary, articles from Simple Wikipedia tagged as science, and additional science sentences collected from the Web. From a vocabulary analysis, 99.8% of the ARC question vocabulary is mentioned in the ARC Corpus Only 11 question words, mainly proper nouns, do not appear in the corpus: Daphne, Sooj, LaKeisha, Quickgrow, Hypergrow, CCGCAT, nonsnow, Quickgrow, Coaccretion, HZn, MgBr.

In our baseline experiments discussed shortly, we find that if we change the corpus behind the IR solver from Waterloo to the ARC Corpus, this changes its Challenge Set score from near zero (by definition, Challenge questions are those that IR with the Waterloo corpus gets wrong) to a score similar to random guessing In fact slightly lower than random guessing, likely due to some corpus similarities where the same distractors in Waterloo were also present in theARC Corpus.. However, from an informal, sampled analysis, we find that this is more a limitation of the IR methodology than of the coverage of the ARC Corpus. The ARC Corpus, in fact, appears to mention knowledge relevant to approximately 95% of the ARC Challenge questions (from an analysis of a random sample of questions), even if simple retrieval methods are not able to exploit it to correctly answer the questions. For example, consider:

Scientists launch a rocket into space for a mission. Once the rocket escapes the gravitational pull of Earth, how will the mass and weight of the rocket be affected? (A) The mass and weight will change. (B) The mass and weight will stay the same. (C) The mass will stay the same, but the weight will change. [correct] (D) The mass will change, but the weight will stay the same.

While this particular scenario is of course not mentioned explicitly in the ARC Corpus, there are several statements about the relation between mass, weight, and gravity, for example:

The main difference is that if you were to leave the Earth and go to the Moon, your weight would change but your mass would remain constant.

Astronauts in orbit experience weightlessness just like objects in the falling aircraft.

Weight is the force that something feels due to gravity: so the brick would have a much larger weight near the earth’s surface than it does in deep space.

Such sentences provide evidence that weight but not mass will change when in space. Similarly, consider the question:

Which factor will prompt an animal’s fight-or-flight response? (A) population size (B) competition for food [correct] (C) seasonal temperatures (D) protection of the environment

While there are few sentences in the ARC Corpus that directly relate fight-or-flight with food competition, there are several that indicate competition is a kind of threat, e.g.,:

Insurgency, cross-border firing, competition with livestock for grazing ground, …all threaten the Markhor.

The main threats to the species include loss of habitat through deforestation, hunting for meat and competition with domestic livestock.

Habitat fragmentation threatens this species, as does possible competition with the introduced non-native Ring-necked Pheasant.

and similarly that threats can trigger the fight-or-flight response:

Whenever a reptile faces a threat they go into fight or flight mode…

The sympathetics command your fight or flight response, and when they fire, your heart rate and your breathing speed up, …preparing your body to face a threat…

Which of the following distinguishes the organisms in the kingdom Fungi from other eukaryotic organisms? (A) Fungi are unicellular. (B) Fungi reproduce sexually. (C) Fungi obtain nutrients by absorption. [correct] (D) Fungi make food through photosynthesis

Again, there is distributed evidence for the answer, including that (A) is incorrect:

There are unicellular fungi and multicellular fungi.

Fungi are heterotrophs that acquire their nutrients by absorption.

and that (C) is distinct for other eukaryotic organisms:

Animals are multicellular eukaryotes; they are chemosynthetic heterotrophs that ingest their food.

Although these descriptions of reasoning paths are informal, and clearly many others are possible, they illustrate that the ARC Corpus mentions knowledge relevant to a question, even if no single sentence alone provides the answer. Of course, this does not address the challenge of correctly identifying and reasoning with this knowledge, nor the challenge of injecting unstated commonsense knowledge that may also be required. Rather, our claim is that the Corpus contains substantial linguistic signal relevant to most of the ARC questions, and is a useful starting point for corpus-based attacks on the Challenge.

Baseline Performance

We ran several baseline QA systems on the Challenge and Easy Sets, including two neural models, DecompAttn and BiDAF (details below), that have near state-of-the-art performance on the well-known SNLI and SQuAD datasets respectively. We scored systems using the following scoring rubric: For each question, a system receives 1 point if it chooses the correct answer and $1/k$ if it reports a $k$-way tie (i.e., chooses multiple answers) that includes the correct answer. For a question set, the overall score of the system is the sum of the points it receives for all questions, divided by the number of questions and reported as a percentage. We report performance of the following systems:

IR (dataset definition). IR method, described earlier.

PMI (dataset definition). PMI method, described earlier.

Guess-all (“random”). A naïve baseline that selects all answer options as equally valid, thereby scoring $1/k$ for each question with $k$ answer choices. A system that chooses a single answer at random will also converge to this score after enough trials.

IR (ARC Corpus). The IR algorithm, rerun with the ARC Corpus. Note that changing the original corpus is expected to result in a different score, unless the two corpora are highly correlated. A corpus containing random strings, for instance, will have very low correlation with the original corpus and will result in a random-guessing score of around 25%.

TableILP Khashabi et al. (2016), which performs matching and reasoning using a semi-structured knowledge base of science knowledge, expressed in tables.

TupleInference Khot et al. (2017), which performs semi-structured matching of the question with retrieved sentences, where the structure consists of Open IE tuples.

DecompAttn, DGEM, and DGEM-OpenIE (Neural Entailment Models). We adapted two neural entailment models, DecompAttn Parikh et al. (2016) and DGEM Khot et al. (2018), to the task of answering multiple-choice questions. The models were trained on an extended version of the SciTail dataset Khot et al. (2018). To adapt these to multiple-choice QA, we first convert the question $q$ plus an answer option $a_{i}$ into a hypothesis sentence (or paragraph) $h_{i}$, use this as a search query to retrieve text sentences $t_{ij}$ from a corpus, then compute the entailment scores between $h_{i}$ and each $t_{ij}$. This is repeated for all answer options, and the option with the overall highest entailment score selected. Further details are given in the Appendix.

The DGEM model uses a structured representation of the hypothesis $h_{i}$, extracted with a proprietary parser plus Open IE. To create a releasable version of DGEM, we also evaluate (and release) a variant DGEM-OpenIE, a version of DGEM that only uses Open IE to create the structured representation of $h_{i}$, thus avoiding proprietary tools.

BiDAF (Reading Comprehension Model). We also adapted BiDAF Seo et al. (2017b), a top-performer on the SQuAD dataset Rajpurkar et al. (2016), to our task. As BiDAF is a direct answer system, we adapted it to multiple-choice QA following the approach used in several previous projects Khashabi et al. (2018b); Welbl et al. (2017a); Kembhavi et al. (2017) as follows: First, given a question, we create a single paragraph by concatenating a set of retrieved sentences. In this case, we use the same sentences retrieved by the entailment models for all answer options (above). We then use BiDAF to select an answer span from this paragraph, given the question. Finally, we pick the multiple-choice option that maximally overlaps that answer span (here defined as the option with the highest percentage of lemmatized, non-stopword tokens covered by the BiDAF answer span). BiDAF was trained on SQuAD then further tuned to science questions using continued training.

Results

Table 6 summarizes the scores obtained by various baseline algorithms on the test partitions of the Challenge and Easy sets.

The IR and PMI (dataset definition) solvers, by design, score near zero on the Challenge set. The slightly above-zero score is due to the solver occasionally picking multiple (tied) answers, resulting in a partial credit for a few questions. We include these questions in the Challenge set.

The most striking observation is that none of the algorithms score significantly higher than the random baseline on the Challenge set, where the 95% confidence interval is $\pm 2.5\%$. In contrast, their performance on the Easy set is generally between 55% and 65%. This highlights the different nature and difficulty of the Challenge set.

The poor performance of the non-IR solvers is partly explained by their correlation with the IR solver: the first step for nearly all of them (except TableILP, which uses non-sentential knowledge but has low knowledge coverage) is to use a simple IR method to obtain relevant sentences, and then process them in different ways such as extracting structure, attempting matching, attempting chaining, etc. However, the retrieval bias of the underlying IR methods is towards sentences that are all very similar to the question, and away from sentences that individually only partially match the question, but together fully explain the correct answer (e.g., through chaining). This suggests the need for a more advanced retrieval strategy for questions that require combining multiple facts, as well as new methods for combining that information.

Conclusion

Datasets have become highly influential in driving the direction of research. Recent datasets for QA have led to impressive advances, but have focused on factoid questions where surface-level cues alone are sufficient to find an answer, discouraging progress on questions requiring reasoning or other advanced methods. To help the field move towards more difficult tasks, we have presented the AI2 Reasoning Challenge (ARC), consisting of a new question set, text corpus, and baselines, and whose Challenge partition is hard for retrieval and co-occurence methods. We find that none of the baseline systems tested can significantly outperform a random baseline on the Challenge set, including two neural models with high performances on SNLI and SQuAD. Progress on ARC would thus be an impressive achievement, given its design, and be significant step forward for the community. To access ARC, view the leaderboard, and submit new entries, visit the ARC Website at http://data.allenai.org/arc.

The ARC Dataset, Corpus, three baseline neural models (DecompAttn, BiDAF, and DGEM), and the leaderboard, are all available from http://data.allenai.org/arc.

References

Appendix A Appendix A: Question Sources

Table 7 provides a brief description of the question sources used to assemble ARC.

Appendix B Appendix B: Question Answering via Entailment

Given a large corpus of sentences (such as the ARC Corpus), one natural way to answer multiple-choice questions is by identifying the answer choice entailed by a sentence in this corpus. We can, therefore, use an entailment model to answer multiple-choice questions by (1) converting a question and each answer choice into an assertion (2) computing the entailment scores for this assertion (hypothesis) from sentences in our corpus (premise) (3) return the answer with the highest support based on the entailment scores. We describe each step in further detail.

We convert a question, $q$ and answer choice, $a$ to an assertion, $h_{q,a}$ by first converting the question into a Fill-In-The-Blank (FITB) statement, $f_{q}$ (e.g. Plants produce ___). In certain cases, the questions were already in FITB form and in other cases, we replace the wh-word with a blank. We also drop phrases common in the multiple-choice question setting, e.g. “of the following”, “of these”. Given a FITB statement, we just replace the blank with the answer choice to generate the assertion $h_{q,a}=\textit{replace}(f_{q},\_\_\_,a)$. However, this simple approach can lead to non-grammatical sentences. E.g. “How many times does Earth rotate on its axis in one day?” will be converted to “One many times does Earth rotate on its axis in one day.”. A more sophisticated rule-based approach could be developed to handle each question type; but we leave this to future work.

Entailment Computation

Next, we use the entailment model to identify the support for the generated hypothesis, $h_{q,a}$. For a large-scale text corpus, it would not be computationally feasible to compute the support from each sentence in the corpus. Since sentences supporting the assertion are likely to have high lexical overlap, we use information retrieval as described earlier to find relevant sentences. We query an ElasticSearch index on this corpus with the question and answer choice as a query. We also require the matched sentence must contain the answer choice to avoid sentences that can not be used to answer questions. We filter long sentences ($>300$ characters), sentences with negation and noisy sentences from the retrieved sentences. Similar to SemanticILP Khashabi et al. (2018a), we pick the top 8 sentences (based on their ElasticSearch score, $score_{ir}(p_{q,a},q+a)$) across the answer choices for each question. These sentences, $P_{q}$ form the set of candidate premises used to compute the entailment score.

For an entailment model $g$, we can compute the score for each question, answer choice and premise as $p_{q,a}\in P_{q},score_{e}(q,a,p_{q,a})=entails_{g}(p_{q,a},h_{q,a})$. We present the results with two neural entailment models, Decomposable Attention Parikh et al. (2016) and Decomposed Graph Entailment Model (DGEM) Khot et al. (2018). Specifically for DGEM, we also extract graph structure from the hypothesis using Open IE and a chunker. Since the chunker uses proprietary software, we also evaluate and release DGEM-OpenIE that only uses the graph structure from Open IE and ignores sentences with no Open IE extraction. These models were trained on the Science Entailment dataset Khot et al. (2018), a dataset created in a similar manner where crowd-workers annotated the premises and hypotheses as entailing or not. Our released code https://github.com/allenai/arc-solvers can also be easily extended to use any other neural entailment model implemented in AllenNLP.

Answer Scoring

Given the scores for each premise $score_{e}(q,a,p_{q,a})$, we use the maximum supporting sentence score as the answer choice $score_{c}(q,a)=\max score_{e}(q,a,p_{q,a})$.