EXAMS: A Multi-Subject High School Examinations Dataset for Cross-Lingual and Multilingual Question Answering

Introduction

Research on science question answering has attracted a lot of attention in recent years (Clark, 2015; Schoenick et al., 2017; Clark et al., 2019). Such questions are challenging as they require domain and common sense knowledge (Clark et al., 2018), as well as complex reasoning and different forms of inference over a variety of knowledge sources (Khashabi et al., 2016, 2018). Indeed, a combination of these was required to achieve noticeable performance gains Clark et al. (2016). This inevitably made research in school-level science Question Answering (QA) hard for languages other than English due to the scarceness of resources (Clark et al., 2014; Khot et al., 2017, 2018; Bhakthavatsalam et al., 2020).

There has been a recent mini-revolution in QA, as well as in the field of Natural Language Processing (NLP) in general, due to the invention of the Transformer (Vaswani et al., 2017), and the subsequent rise of large-scale pre-trained models (Peters et al., 2018; Radford et al., 2018, 2019; Devlin et al., 2019; Lan et al., 2020; Yang et al., 2019; Liu et al., 2019c; Raffel et al., 2020). Nowadays, fine-tuning such models on task-specific data has become an essential element of any top-scoring QA system. Yet, for science QA, training on datasets from a different domain (Sun et al., 2019; Khashabi et al., 2020) and carefully selected background knowledge (Banerjee et al., 2019; Ni et al., 2019) could improve such models further.

The success of large-scale pre-trained models and the development of their multilingual versions (Devlin et al., 2019; Conneau et al., 2020) gives hopes for supposedly better performance in multilingual question answering. Therefore, several new datasets have been released for multilingual reading comprehension and open-domain question answering in the Wikipedia domain (Liu et al., 2019a; Lewis et al., 2020; Artetxe et al., 2020; Clark et al., 2020).

Here, we present $E{\chi}\alpha{\mu}s$, a new dataset and benchmark for multilingual and cross-lingual evaluation of models and methods for answering diverse school science questions (see Figure 1).

We advance the task of science Question Answering (QA) with multilingual and cross-lingual evaluations.

We collect a new challenging dataset $E{\chi}\alpha{\mu}s$ from multilingual high school examinations, which offers several advantages over existing datasets: (i) it covers various domains, (ii) it is nearly three times larger than pre-existing Science QA datasets, (iii) it extends multilingual QA tasks to more languages, (iv) the questions are written by experts, rather than translated or crowdsourced, (v) the questions are harder since they are from matriculation exams rather than 4-8th grade.

We use fine-grained evaluation – per subject and per language – which yields more precise comparison between models.

We perform extensive experiments and analysis using top-performing multilingual models (mBERT, XLM-R), and we show that $E{\chi}\alpha{\mu}s$ offers several challenges that such models would need to overcome in the future, including multi-lingual and cross-lingual knowledge retrieval, aggregation, and reasoning, among others.

We release our code, pre-trained models and data for research purposes.The $E{\chi}\alpha{\mu}s$ dataset and code are publicly available at http://github.com/mhardalov/exams-qa

Related Work

The work in Science Question Answering emerged in recent years with the development of several challenging datasets. The most notable is ARC (Clark et al., 2018), which is a QA reasoning challenge that contains both Easy and Challenge questions from 4th to 8th grade examinations in the Natural Science domain. As in $E{\chi}\alpha{\mu}s$, the questions in ARC are created by experts, albeit our dataset covers a wide variety of high school (8th-12th grade) subjects including but not limited to, Natural Sciences, Social Sciences, Applied Studies, Arts, Religion, etc. (see Section 3.2 for details). We provide definitions of the less known subjects in $E{\chi}\alpha{\mu}s$ in Appendix B.1.

The early versions of ARC (Clark, 2015; Schoenick et al., 2017) inspired several crowdsourced datasets: Welbl et al. (2017) proposed a scalable approach for crowdsourcing science questions given a set of basic supporting science facts. Dalvi et al. (2019) focused on specific phenomena including understanding science procedural texts, Mihaylov et al. (2018) and Khot et al. (2020) studied multi-step reasoning, given a set of science facts and commonsense knowledge, Tafjord et al. (2019), and Mitra et al. (2019) worked on reasoning about qualitative relationships, and declarative texts, among others. Unlike these English-only datasets, $E{\chi}\alpha{\mu}s$ offers questions in 16 languages. Moreover, it contains questions about multiple subjects, which are presumably harder as they were extracted mostly from matriculation examinations (8-12th grade). Finally, $E{\chi}\alpha{\mu}s$ contains over 24,000 questions, which is more than three times as many as in ARC.

Multilingual and Cross-lingual QA

Recently, several QA datasets have been created that cover languages other than English, but still focusing on one such language. Gupta et al. (2018) proposed a parallel QA task for English and Hindi, Liu et al. (2019b) collected a bilingual cloze-style dataset in Chinese and English. Jing et al. (2019) crowdsourced parallel paragraphs from novels in Chinese and English. A few datasets investigated multiple-choice school QA (Hardalov et al., 2019; Van Nguyena et al., 2020), albeit in a limited domain, and for lower school grades (1st-5th). Other efforts focused on building bi-lingual datasets that are similar in spirit to SQuAD (Rajpurkar et al., 2016) – extractive reading comprehension over open-domain articles. Such datasets are collected by crowdsourcing questions, following a procedure similar to (Rajpurkar et al., 2016), in Russian (Efimov et al., 2020), Korean (Lim et al., 2019), French (d’Hoffschmidt et al., 2020), or by translating existing English QA pairs to Spanish (Carrino et al., 2020).

Recently, some multilingual datasets, were released to the public. MLQA (Lewis et al., 2020), and XQuAD (Artetxe et al., 2020) use translations by professionals and extend the monolingual SQuAD (Rajpurkar et al., 2016) to 7 and 11 languages, respectively, thus forming cross-lingual evaluation benchmarks. Clark et al. (2020) collected an entirely new dataset (TyDi QA) of questions in 11 typologically diverse languages.

The task was to ask a question, and then the shortest span answering it from a list of paragraphs was selected. As these datasets are complementary, rather than making each other obsolete, hereby the recently released XTREME (Hu et al., 2020) benchmark combined them in a joint task. $E{\chi}\alpha{\mu}s$ differs from the aforementioned multilingual benchmarks in several aspects. First, we extend the multilingual QA efforts to a different, more challenging domain (Clark et al., 2018). Second, our datasets support more languages. Next, the questions in $E{\chi}\alpha{\mu}s$ are written by educational experts rather than non-expert annotators, making the evaluation results comparable to a top-performing student. Finally, our fine-grained evaluation for different subjects, languages, and combinations thereof allows for in-depth analysis and comparison.

E​χ​α​μ​s𝐸𝜒𝛼𝜇𝑠E{\chi}\alpha{\mu}s Dataset

We introduce $E{\chi}\alpha{\mu}s$, a new benchmark dataset for multilingual and cross-lingual question answering from high school examinations. In this section, we present the properties of the dataset, and we give details about the process of data collection, preparation and normalization, as well as information about the data splits, and the parallel questions.

We collected $E{\chi}\alpha{\mu}s$ from official state exams prepared by the ministries of education of various countries. These exams are taken by students graduating from high school, and often require knowledge learned through the entire course. The questions cover a large variety of subjects and material based on the country’s education system. Moreover, we do not focus only on major school subjects such as Biology, Chemistry, Geography, History, and Physics, but we also cover highly-specialized ones such as Agriculture, Geology, Informatics, as well as some applied and profiled studies. These characteristics make the questions in the dataset of very high variety, and not easily solvable, due to the need for highly specialized knowledge. Next, we discuss the cross-lingual and the multilingual properties of our dataset.

Some countries allow students to take official examinations in several languages. Such parallel examinations also exist in our dataset. In particular, there are 9,857 parallel question pairs spread across seven languages as shown in Table 2. The parallel pairs are coming from Croatia (Croatian, Serbian, Italian, Hungarian), Hungary (Hungarian, German, French, Spanish, Croatian, Serbian, Italian), and North Macedonia (Macedonian, Albanian, Turkish).

Multilinguality

Our dataset includes a total of 24,143 questions in 16 languages from eight language families. Each question is a 3-way to 5-way (3.96 on average) multiple-choice question with a single correct answer. Table 1 shows a breakdown for each language, where the number of subjects, questions, and the vocabulary size are shown as absolute numbers, while the question length, the choice length, and the number of choices are averaged. All statistics about the questions and the answer options are measured in terms of words. We see that we have a rich vocabulary with almost 160,000 unique words. Interestingly, there are $\sim$9,500 shared words between at least one pair of languages in our dataset, excluding numbers and punctuation. As expected, the overlapping words are mostly between closely related languages (bg-mk, bg-sr, es-it, es-pt, hr-sr, mk-sr). Other common shared words are subject-specific words such as person names (e.g., Abraham, Karl, Ivan), chemical compounds (e.g., NaOH, HCl), units (e.g., m/s, g/mol), etc. Then, there are cognates with the exact same spelling (homographs) even between unrelated languages, mostly words of Latin or Greek origin, e.g., temperatura (temperature) and forma (form). Finally, there are also false friends, whose meaning differs across languages, e.g., para can mean for (es/pt) vs. money (mk/tr/sq) vs. couple (pl); similarly, ser can mean be (es/pt) vs. cheese (pl) vs. after (vi).

2 Subjects and Categories

Each education system has its own specifics, resulting in some differences in curricula, topics, and even naming of the subjects. That being said, the original, non-normalized categories in our dataset are more than 40 for exams from just a few countries. Given the sparse nature of the subjects, we use a two-level taxonomy in order to categorize them into logically connected groups. The lower-level is a subject, and the higher level is a major group. We normalized the subject using a two-step algorithm: first, we put each subject (with its original naming) in a separate category, then, if the subject was general enough, e.g., Biology, History, etc., or there were no similar ones, we retained the category; otherwise, we merged all similar subjects together in a unifying category, e.g., Economics Basics, and Economics & Marketing. We repeated the aforementioned steps until there were no suitable merge candidates. As a result, we ended up with a total of 24 subjects (see Appendix B for more details), which we further grouped into three major categories, based on the main branches of science: Natural Science – “the study of natural phenomena”, Social Sciences – “the study of human behavior and societies”, Other – Applied Studies, Arts, Religion, etc. (see Figure 1). https://en.wikipedia.org/wiki/Branches_of_science

The distribution of the major categories is Natural Sciences (40.0%) and Social Sciences (44.0%) and 16.0% for Others (these are the actual numbers, not approximate). The remaining questions are labeled as Other as they are not suitable for the two main categories. Figure 2 presents the relative sizes of the subjects in the dataset.

3 Collection and Preparation

Here, we describe the process of collecting and preparing the data, as it is not trivial and it could be applied to other languages and examinations. First, we identified potential online sources of publicly available school exams starting from the Matriculation Examination page in Wikipedia. https://en.wikipedia.org/wiki/Matriculation_examination

For all languages in our dataset, the first step in the process of data collection was to download the PDF files per year, per subject, and per language (when parallel languages were available in the same source). We converted the PDF files to text and we used only those that were well-formatted and followed the document structure.

Then, we used Regular Expressions (RegEx) to parse the questions, their corresponding choices and the correct answer choice. In order to ensure that all our questions are answerable using textual input only, we removed questions that contained visual information. We did that using a manually curated list of words such as map, table, picture, graph, etc., in the corresponding language. Next, we performed data cleaning to ensure the quality of the generated dataset, by manually reviewing each question and its choices and ensuring that all options, text, and symbols (e.g., $\mu,\rightarrow,\alpha,\leftarrow$) were displayed correctly. As a result, we filtered out about 17% of the questions (the percentage varies based on the source, the language, and the subject). Finally, in order to remove frequency bias such as “most answers are B)”, we shuffled each question’s choices.

4 Data Splits

In our experiments, we aim at evaluating the multilingual and the cross-lingual question answering capabilities of different models. Therefore, we split the data in order to support both evaluation strategies: Multilingual and Cross-lingual.

In this setup, we want to train and to evaluate a given model with multiple languages, and thus we need multilingual training, validation and test sets. In order to ensure that we include as many of the languages as possible, we first split the questions independently for each language $L$ into TrainL, DevL, TestL with 37.5%, 12.5%, 50% of the examples, respectively.For languages with fewer than 900 examples, we only have TestL. We then unite all language-specific subsets into the multilingual sets TrainMul, DevMul, TestMul, and we used them for training, development, and testing.

Since we have parallel data for several languages (discussed in Section 3.1), in this setup, we ensure that the same parallel questions are only found in either training, development or testing, so that we do not leak the answer from training via some other language. In order to do that, we sample the questions with the assumptions and the ratios mentioned above, stratified per subject in the given language. The number of examples per language and the total number of multilingual sets are shown in the first three columns of Table 3.Sometimes, grouping parallel questions in the same split slightly violates the splitting ratios.

Cross-Lingual

In this setting, we want to explore the capability of a model to transfer its knowledge from a single source language $L_{src}$ to a new unseen target language $L_{tgt}$. In order to ensure that we have a larger training set, we train the model on 80% of $L_{src}$, we validate on 20% of the same language, and we test on a subset of $L_{tgt}$.To ensure that the cross-lingual evaluation is comparable to the multilingual one, we use the same subset of questions from language $L_{tgt}$ that are used in TestMul The last three columns of Table 3 show the number of examples used for training and validation with the corresponding language.

5 Reasoning and Knowledge Types

In order to give a better understanding of the reasoning, and the knowledge types in $E{\chi}\alpha{\mu}s$, we sampled and annotated 250 questions, all of which are from the multilingual Dev. For each question, we provided English translations as not all annotators were native speakers of the questions’ language. We followed the procedure and re-used the annotation types presented in earlier work (Clark et al., 2018; Boratko et al., 2018). However, as they were designed mainly for Nature Science questions, we extended them with two new annotation types: “Domain Facts and Knowledge” and “Negation” (see Appendix C for examples).

The relative sizes of the knowledge and the reasoning types are shown in Figures 3 and 4. Here, we must note that the sizes are approximate rather than exact, since the annotations are subjective and the distribution may vary.

Baseline Models

We divide our baselines into the following two categories: (i) models without additional training, and (ii) fine-tuned models. The first group contains common baselines, i.e., random guessing and information retrieval solver (Clark et al., 2016). In addition, we evaluate the knowledge contained in the pre-trained language model, i.e., mBERT (Devlin et al., 2019) and XLM-R (Conneau et al., 2020), and we use it as an answering mechanism. The second group of baselines compare the learning ability of state-of-the-art multilingual models on the task of multiple-choice question answering. Since we have multi-choice questions, we adopt accuracy as an evaluation measure, as this is standard for this setup.

This IR baseline is from Clark et al. (2016), and it ranks the possible options $o$ for each question $q$ based on the relevance score returned by a search engineWe build and use a separate index for each language using ElasticSearch.. In particular, for each option $o_{i}$, we form a query by appending the option’s text to the question’s ($q+o_{i}$), and we send this concatenation to the search engine. We then sum the returned scores for the top-10 hits, and we predict the choice with the highest score to be the correct answer. More detailed discussion can be found in Appendix D.

Pre-trained Model as a Knowledge Base (KB)

As we start to understand pre-trained BERT-like models better(Petroni et al., 2019; Rogers et al., 2020), we observe some interesting phenomena. Here, we evaluate the knowledge contained in the model by leveraging the standard masking mechanism used in pre-training. We tokenize each question-option pair into subwords, and then we replace all the pieces from the option with the special [MASK] token. Following the notation from Devlin et al. (2019), the input sequence can be written as follows:

where Q is the question, and M_O is the masked option. Following the notation above, we obtain a score for each option in the question based on the normalized log-probability for the entire masked sequence. (see Eq. 1).

We could probably obtain better results for that evaluation if we form the question-option pairs as a single statement, e.g., “What is the purpose of something? [SEP] [M_O] $\rightarrow$ The purpose of something is [M_O].”

2 Fine-Tuned Models

We are interested in evaluating the ability of pre-trained models to transfer science-based knowledge across languages when fine-tuned.

In order to evaluate the QA capability of these models, we follow the established approach in this setting (Devlin et al., 2019; Liu et al., 2019c; Sun et al., 2019), and we fine-tune them to predict the correct answer in a multi-choice setting, given a selected context. This setup feeds the pre-trained model with a text, tokenized using the corresponding tokenizer for the model in the format:

where C, Q and O are the tokenized knowledge context (see Appendix D), the question, and the option, respectively. Each question-option pair (Q+O) is evaluated, and the one with the highest confidence of being an answer is selected.

In our experiments, we used the Transformers library (Wolf et al., 2019). We experimented with the best-performing multilingual models: the Multilingual version of BERT, or mBERT Devlin et al. (2019), and the recently proposed XLM-RoBERTa, or XLM-R Conneau et al. (2020).

Multilingual BERT (Devlin et al., 2019) is a fundamental multilingual model trained on 104 languages with a vocabulary of 110K word-pieces, with a total of 172M parameters (12 layers, 768 hidden states, 12 heads).

XLM-RoBERTa (Conneau et al., 2020) is a recent multilingual model based on RoBERTa (Liu et al., 2019c). It is trained on 100 languages, with a larger vocabulary of 250K sentence pieces. It comes in two sizes: XLM-RBase (270M parameters, same architecture as mBERT, except vocab size), and XLM-R (550M parameters, 24 layers, 1,024 hidden states, 16 heads). For completeness, we include both in our experiments.

We fine-tuned the aforementioned models following the standard procedure for multiple-choice comprehension tasks, as described in (Devlin et al., 2019) and (Liu et al., 2019c), using the Transformers library (Wolf et al., 2019). The training details can be found in Appendix A.

Experiments and Results

In this section, we evaluate the performance of the baseline models described in Section 4 on the $E{\chi}\alpha{\mu}s$ dataset. In Table 4, we show the overall per-language performance of the evaluated models. The first group shows simple baselines: random guessing and IR over Wikipedia articles. IR is better than random guessing, but it is clear that most questions require reasoning beyond simple word matching. In the last group, we evaluate the knowledge contained in the models before and after the QA fine-tuning. First, we evaluate XLM-R as a knowledge base, and then we use the Full model but with the question–option pair only.

The next two groups show (i) how continuous fine-tuning of XLM-R on multi-choice machine reading comprehension and multi-choice science QA helps, and (ii) how the different models (XLM-R, XLM-RBase, and mBERT) compare. We follow a standard training scheme for such tasks: first we fine-tune on RACE (Lai et al., 2017) ($\sim$85k EN questions over documents), then on the AI2 English science datasets (we call them SciENs for shorter), including $\sim$9k EN questions with provided relevant contextsWe use the data described at http://leaderboard.allenai.org/arc/submission/blcotvl7rrltlue6bsv0, and, finally, on our multilingual training set (see Section 3.4) with retrieved relevant contexts from Wikipedia (see Appendix D), which is our desired multilingual evaluation setting and we call it Full. We can also see that training on the SciENs, which has mostly primary school questions from Natural Sciences, only yields +0.5% improvement on $E{\chi}\alpha{\mu}s$. Nevertheless, we see a 2.4% improvement with multilingual fine-tuning on $E{\chi}\alpha{\mu}s$ and +0.5% for English. In the third group, we compare the results from mBERT, XLM-RBase, and XLM-R after fine-tuning. Increasing the capacity of the model yields improvements: XLM-R scores 7.4% higher on $E{\chi}\alpha{\mu}s$, and more than 14% on English datasets, compared to its base version (XLM-RBase). However, mBERT and XLM-RBase have close performance, with mBERT having a small advantage in the multilingual setting.

Finally, we fine-tuned mBERT on $E{\chi}\alpha{\mu}s$ only. As expected, the performance drops by 3% absolute compared to the Full setup.

2 Knowledge Evaluation

The last two rows of Table 4 evaluate the knowledge in the best model, namely XLM-R. With XLM-R as KB (see Section 4.1) we see small improvement over the random baseline: +5% ARC Easy, 2% on R12, and just +1% on $E{\chi}\alpha{\mu}s$ and ARC Challenge. Furthermore, we evaluate the knowledge contained in the model after the Full fine-tuning by excluding the relevant knowledge context (ctx). This is better than the XLM-R as KB, but it still achieves inferior overall results, which shows that the stored knowledge is not enough, and that we need to explicitly obtain additional knowledge from an external source.

3 Cross-lingual Evaluation

Table 5 shows the results from the cross-lingual zero-shot transfer compared to the English-only baseline $en_{all}$, from XLM-R fine-tuned on SciEN. The languages are ordered by family, and then alphabetically. We further fine-tune on a single source language and we test on all other languages using the splits described in Subsection 3.4. The results show that the additional fine-tuning on a single language is mostly positive. This is notable when fine-tuning on a language with similar linguistic characteristics to the target language, e.g., Balto-Slavic: bg-sr, hr-mk, pl-mk, sr-bg.

We also see gains when the source language contains more questions from largely represented and harder subjects. Examples of such are the experiments showing the positive effects of training on Vietnamese and Macedonian as source languages; they both contain such subjects: Biology, History, Chemistry, Physics, and Geography.

This is an indication that the knowledge from the same or from related subjects in a non-related language is preferred over knowledge from non-related subjects from a related language. For the same reasons, Portuguese and Polish show negative effects of fine-tuning on some of the target languages. They contain mostly niche subjects such as Professional, Philosophy, Economics, Geology. We see a noticeable drop in accuracy for Portuguese almost everywhere, but it has positive effect on languages that contain similar subjects (Biology, Economics) or are from the same language family such as Spanish and Italian (for Portuguese). We see the opposite in the Lithuanian-Polish pair, languages from the same family (but different subjects) have negative, or no effect on each other. Finally, we analyze the results from language pairs containing parallel examples (the underlined values). Such pairs show consistent improvement (+5 to +10), which suggests that the model learns to align the parallel knowledge from the source language to the target language. However, we also must note that the effect is strongly dependent on the size of the overlapping sets.

4 Per-subject Fine-grained Evaluation

Fine-grained evaluation (Mihaylov and Frank, 2019; Xu et al., 2020) allows an in-depth analysis of the question answering models. One of the nice features of $E{\chi}\alpha{\mu}s$ is that it supports subject-related fine-grained evaluation. On Figure 5, the results are shown by subject group and per-subject for Natural Science.Per-subject results for Social Science and Other are available in Appendix E.

We can see that the Natural Science questions are the most challenging ones, which is mostly due to Chemistry and Physics. Those questions require very complex reasoning and knowledge such as understanding physical models, processes and causes, comparisons, algebraic skills and multi-hop reasoning (see Section 3.5). These skills are currently beyond the capabilities of the current QA models, and pose interesting challenges for future work (Welbl et al., 2018; Yang et al., 2018; Saxton et al., 2019; Lample and Charton, 2020). Informatics is another challenging subject, as it requires understanding programming code and positional numerical systems among others.

Discussion

Our results show that initial fine-tuning on a large monolingual out-of-domain multi-choice machine reading comprehension dataset (RACE (Lai et al., 2017)) performs much better than no training baselines for answering multilingual $E{\chi}\alpha{\mu}s$ questions. Moreover, additional training on English science QA in lower school levels has no significant effect on the overall accuracy. These results suggest that further investigation of fine-tuning with other multilingual datasets (Gupta et al., 2018; Lewis et al., 2020; Clark et al., 2020; Efimov et al., 2020; d’Hoffschmidt et al., 2020; Artetxe et al., 2020; Longpre et al., 2020) is needed in order to understand the domain transfer benefits to science QA in $E{\chi}\alpha{\mu}s$, even if they are not in a multi-choice setting (Khashabi et al., 2020). Using domain-adaptive and task-adaptive pre-training (Gururangan et al., 2020) to the multilingual science QA might offer further potential benefits.

Moreover, we need a better knowledge context for a given question–choice pair (the last row in Table 4). Knowing that the context retrieved from the noisy Wikipedia corpus is relevant for answering $E{\chi}\alpha{\mu}s$ questions, suggests that we need a better multilingual science corpus, similar to Clark et al. (2018); Pan et al. (2019); Bhakthavatsalam et al. (2020). We further need better multilingual knowledge selection and ranking (Banerjee et al., 2019). Finally, our cross-lingual experiments show that we can align the knowledge between languages from parallel examples, which poses a new question: Is it only due to keyword matching or could the model align full sentences?

Conclusion and Future Work

We presented $E{\chi}\alpha{\mu}s$, a new challenging cross-lingual and multilingual benchmark for science QA in 16 languages and 24 subjects from high school examinations.

We further proposed new fine-grained evaluation that allows precise comparison across different languages and school subjects. We performed various experiments and analysis with pre-trained multilingual models (XLM-R, mBERT), and we demonstrated that there is a need for better reasoning and knowledge transfer in order to solve some of the questions from $E{\chi}\alpha{\mu}s$. We hope that our publicly available data and code will enable work on multilingual models that can reason about question answering in the challenging science domain.

In future work, we plan to extend the dataset with more questions, more subjects, and more languages. We further plan to develop new models to address the specific challenges we identified.

Acknowledgments

We thank the anonymous reviewers for their helpful questions and comments. We thank the AI2 Aristo Team for providing the data splits used for pre-training on SciEN datasets.

This research is partially supported by Project UNITe BG05M2OP001-1.001-0004 funded by the OP “Science and Education for Smart Growth” and co-funded by the EU through the ESI Funds.

References

Appendix A Fine-Tuning and Hyper-parameters

In this work, we are interested in the cross-lingual transferability of multilingual models such as mBERT (Devlin et al., 2019) and XLM-RoBERTa (Conneau et al., 2020), each of which comes pre-trained on more than 100 languages. We evaluated the QA capabilities of these models, following the established protocol (Devlin et al., 2019; Liu et al., 2019c; Sun et al., 2019), namely we fine-tuned them to predict the correct answer in a multi-choice setting, given a selected context. The aforementioned setup feeds the pre-trained model with a text, processed using the model’s tokenizer in the following format:

where C, Q and O are the tokenized knowledge Context (see Appendix D), Question, and Option, respectively.

We used the Transformers library (Wolf et al., 2019). We fine-tuned mBERT, XLM-R, and XLM-RBase in three steps. We first fine-tuned the models with RACE (Lai et al., 2017), a multiple-choice reading comprehension dataset with around 85k questions for training. Then, we trained on the combination of ARC (Clark et al., 2018), OpenBookQA (Mihaylov et al., 2018), and Regents Living Environments, as in the AristoRoBERTaV7 ARC Challenge leaderboard entry https://leaderboard.allenai.org/arc/submission/blcotvl7rrltlue6bsv0; we refer to these datasets as SciENs (Science English datasets). We used the resulting pre-trained models as base models for our Multilingual and Cross-lingual evaluations (Section 5 in the paper). For the multilingual evaluation, we continued training the model, previously fine-tuned on the SciENs datasets, with our multilingual TrainMul set, validating on DevMul and testing on TestMul. For our cross-lingual evaluation, we continued training the SciENs model on separate languages, as described in Section 5.3.

In Table 6, we show the values of the hyper-parameters for each fine-tuning step and corresponding model. Note that these hyper-parameters were not obtained with an exhaustive search, and thus a better setting might exist for each model and dataset. Initially, we used the hyper-parameters for AristoRoBERTaV7 ARC Leaderboard submission for English-only RoBERTa (Liu et al., 2019c): epochs = 4, learning rate = 1e-5.

With these parameters alone, the models did not perform well, and thus we added a warmup of 0.1 and a weight decay of 0.06, which stabilized the training. In all experiments, we used the Adam optimizer with $\beta_{1}$=0.9, $\beta_{2}$=0.999, and $\epsilon$=1e-08.

We further performed manual tuning of the hyper-parameters: we experimented with variations thereof, depending on the performance on the corresponding development sets, and we ended up with the values in Table 6. Moreover, we adjusted the batch size and the accumulation steps depending on the availability of the GPUs on our cluster: Nvidia GTX 1080 Ti (Pascal, 11GB memory) or Nvidia Quadro RTX 6000 (24GB). For each examined setting, we trained for up-to 6 epochs, evaluating the model on the corresponding development set every 100 to 1000 update steps, depending on the dataset size and the effective batch size. For the final evaluations, we chose the model with the highest accuracy score on the corresponding development set.

Fine-tuning XLM-R (550M parameters) on Nvidia Quadro RTX 6000 (24GB) with the given hyper-parameters took around three hours per epoch when fine-tuned on RACE ($\sim$85k examples), 30 minutes per epoch when fine-tuned on SciENs ($\sim$9k examples), and 30 minutes on $E{\chi}\alpha{\mu}s$ on TrainMul ($\sim$8k examples). Fine-tuning XLM-RBase (270M parameters) and mBERT (172M parameters) on Nvidia GTX 1080 Ti (Pascal, 11GB memory) with the given hyper-parameters took roughly 2 to 2.5 hours per epoch when fine-tuned on RACE ($\sim$85k examples), 30 to 35 minutes per epoch when fine-tuned on SciENs ($\sim$9k examples), and additional 30 minutes on the $E{\chi}\alpha{\mu}s$ TrainMul ($\sim$8k examples).

Appendix B Subject Analysis

The Natural Science group contains five subjects. The corresponding question length is 16.4 characters and 3.9 answers on average. Some of the subjects are well-known and widely studied, such as Physics, Biology and Chemistry. They appear in at least 10 out of the 16 languages, covering 7 out of 8 language families. However, Geology is less common and is present for only 4 languages. Finally, Science is an isolated subject for Arabic. This group contributes a total of 9,962 questions in the entire dataset, as shown in Table 7. The major groups in the table are divided with a horizontal line for convenience.

The second subject group covers Social Sciences. Geography, History, Philosophy, Psychology and Ethics are more common, and thus are included in seven languages on average (see Table 7). The subject group’s average question length is 18.5 characters. The only sizeable deviation being for Citizenship, as most of the questions in this subject explain some social situation in detail.

The last and smallest of the three subject groups is Others. It combines subjects that cannot be categorized as exactly science-related (either social or natural). Those subjects are often specific for a particular country or culture and are fairly diverse. As expected, they are present for less languages (just two).

Next, we give a brief description of the less commonly known subjects included in our dataset.

covers questions about soil farming and preservation, small animals breeding and their general health care, and vehicle maintenance and repair.

Business & Economics

is a term used to combine five similar subjects related to business and economics. The questions in these subjects cover theoretical questions on economics basis, marketing questions, business questions with elements of accountancy, finances, and organizational studies.

Citizenship

is a specific subject from the Vietnamese school system, which tries to inform and give better perspective on different social situations, to educate students in how to perform better, and to be a more aware member of the society by analyzing different norms and personal morality.

Fine Arts

contains analytical and historical questions about different forms of art such as movies, music, art, etc.

Forestry

studies the craft of managing, using, conserving, and repairing forests, woodlands, and associated resources around them such as water sources and soil.

Geology

is the study of the Earth, with the general exclusion of present-day life, flow within the ocean, and the atmosphere. Questions from this subject cover branches of Geology such as Economical Geology, Marine Geology, Geomorphology, and Geophysics.

Informatics

consists of questions about basic hardware knowledge and software management as well as basics of different positional numeral systems (e.g., binary and hexadecimal).

Islamic Studies

refers to the academic studies of Islam, Quran excerpts, and Muslim morality. This a subject studied in the Qatari educational system during both middle and high school.

Landscaping

teaches about modifying the visible features of an area of land, trees and park decorations. It also contains questions about plants and soils.

Politics

covers Croatia’s political system, historical questions about the country’s development, as well as different regulations and laws, international relations and contracts.

Professional

subject is present in the Polish school system and covers knowledge on specific professions such as flight attendant, babysitter, care taker, office worker in terms of profession’s regulations, rules and established norms, etc.

Religion

subject covers Christianity studies such as Bible knowledge, related traditions, e.g., baptism, marriage, etc.

Tourism

covers hospitality management, as well as basis of business and traditions in Hungary and its neighbouring countries.

Science

which is used in the Arabic school system throughout middle and high grade studies combines general science questions from Biology, Chemistry, Physics Geology and their branches such as as Biophysics, Astrophysics, and Biochemistry.

Social

subject, similarly to Science, combines questions from political, cultural, historical and geographical studies.

Sociology

is the study of society, patterns of social relationships, social interaction, and culture that surrounds our everyday life.

Appendix C Reasoning and Knowledge Types

For our reasoning and knowledge type annotations, we followed the procedure and re-used the annotation types presented in Clark et al. (2018); Boratko et al. (2018). However, as they were designed mainly for Natural Science questions, we had to extend them with two new types:

(Knowledge) This skill requires specific expertise in properties and facts in a given domain, e.g., physical properties, characteristics of a chemical element.

Which of the following is an example of a priori knowledge?

C) I know that no brother is an only child. ✓

Negation

(Reasoning) is a direct statement of negation, and it is often combined with other reasoning types such as linguistic matching.

Example from Fine Arts (North Macedonia):

Which of the following works of art does not belong to the fine arts?

Appendix D Background Knowledge Corpus

Students need good textbooks to study before they can pass an exam, and the same holds for a good machine reading model. However, finding the information needed to answer a question, especially for questions in such a narrow domain as the subjects studied in high schools, usually requires a collection of specialized texts. The ARC Corpus (Clark et al., 2018) is an example of such a collection. It is built by querying a major search engine, and around 100 hand-written templates for 80 science topics covered by US elementary and middle schools. Albeit effective, this strategy relies on crafting templates for all language–subject pairs, making the task time-consuming if applied to multiple languages and subjects.

In our work, we used articles from Wikipedia to build a background knowledge corpus for each language. In particular, we parsed the text from the entire Wikipage, removing non-textual content, e.g., HTML tags, tables, etc. Following the common strategy used to solve similar tasks in English (Clark et al., 2018; Mihaylov et al., 2018), we split each document into sentences and we indexed them using an inverted index. In order to reduce the search space, and to mitigate the effect of known linguistic phenomena within the same language family, e.g., homonyms, partially shared alphabet, etc., we created a separate index for each language.

Table 8 describes the main characteristics of the indices created for each language from its Wikipedia dump.We used the official Wiki dumps from March 2020 for all languages. http://dumps.wikimedia.org/ We compared the size of our index to the one from ARC (Clark et al., 2018). The number of articles for each language is taken from Wikipedia’s official statistics The statistics are extracted from http://meta.wikimedia.org/wiki/List_of_Wikipedias. We also marked the language analysis applied on the index. Some of the languages in $E{\chi}\alpha{\mu}s$ are low-resource ones, especially the ones from the Balto-Slavic family, which is also clear from their Wikipedia sizes. In the table, we see that half of the languages have under one million articles, and Albanian even falls under 100K. Moreover, even more languages are comparable with the number of sentences in the ARC Corpus, which is also built from science books. Finally, some of the languages (Serbian, Croatian, Macedonian, and Albanian) are not processed with any language-specific ElasticSearch analyzers.

Appendix E Fine-Grained Evaluation

Figure 6 shows fine-grained evaluation for two subject groups: Social Science and Others. We can see that these subjects are less challenging than Natural Science. One reason is that many of the subjects in these two groups such as Business & Economics, Geography, and History can be answered using knowledge that is easily accessible in sources such as Wikipedia (e.g., “Who was the first prime minister of Poland after 1990?”), i.e., without the need for complex reasoning or calculations, which are often needed in order to answer questions in subjects such as Physics and Chemistry. Nevertheless, while seeing scores as high as 60% for some subjects and languages, the current multilingual QA models are still far from perfect, which leaves a lot of room for improvement.