MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering

Ankit Pal, Logesh Kumar Umapathi, Malaikannan Sankarasubbu

Data and Code Availability

The dataset to reproduce these experiments and the leaderboard to track the progress of MedMCQA is available at medmcqa.github.io

Introduction

Question Answering (QA) is an important and challenging research area in Natural Language Processing (NLP). QA systems enable efficient

access to the vast amount of information available that exists in text format.

In recent times, a significant amount of work has been done on constructing a question-answer dataset (Rajpurkar et al., 2016, 2018; Reddy et al., 2019; Kwiatkowski et al., 2019; Yang et al., 2015) reading comprehension datasets (Yang et al., 2018; Lai et al., 2017; Zellers et al., 2018; Yagcioglu et al., 2018; Dua et al., 2019; Bajaj et al., 2018; Huang et al., 2019), extractive question answering (Hermann et al., 2015; Trischler et al., 2017), healthcare domain QA (Jin et al., 2019; Šuster and Daelemans, 2018; Möller et al., 2020) and the organization of workshops & competitions such as the Question Answering in the medical domain & BioASQ Challenge (Abacha et al., 2019; Nentidis et al., 2020)

However, despite these successful efforts, automatic questions answering for real medical examination is still a challenge that is less explored. This type of real-world examination dataset on complex medical subjects like pharmacology, medicine, surgery, etc., is scarce. Apart from their scarcity, the requirement of a comprehensive understanding of the domain, matching human experts, makes them appealing for research pursuits. Before this attempt, very few works have been done to construct biomedical MCQA datasets (Vilares and Gomez-Rodr, 2019), and they are (1) mostly small, containing up to few thousand questions, and (2) cover a limited number of Medical topics and Subjects.

Thus, a large-scale, diverse medical QA dataset is needed to accelerate research and facilitate more consistent and effective open-domain QA models in Medical-QA. This paper addresses the aforementioned limitations by introducing MedMCQA, a new large-scale, Multiple-Choice Question Answering (MCQA) dataset designed to address real-world medical entrance exam questions. The dataset consists of 194k high-quality medical domain MCQs covering 2.4k healthcare topics and 21 medical subjects to provide a reliable and diverse benchmark. Apart from the question, the correct answer(s), and other options., it also consists of various ancillary data, the primary being a detailed explanation of the solution.

Questions are taken from AIIMS & NEET PG entrance exam MCQs, where graduate medical students are evaluated on their professional knowledge. Questions in these exams are challenging and generally require deeper domain and language understanding as it tests the 10+ reasoning abilities across a wide range of medical subjects & topics. Hence a model must be trained to find relevant information from the open domain knowledge base, reason over them, and choose the correct answer.

Fig.1 shows two example questions, their corresponding explanation, and answers from the study dataset.

An in-depth analysis & a thorough evaluation of the dataset are conducted. The baseline experiments on this dataset with the current state-of-the-art methods can only answer 47% of the question correctly, which is far behind the performance of human candidates (merit candidates of these exams score an average of 90% marks). Error analysis and results indicate possibilities for improvement in the current methods’ reasoning and medical domain question answering. It is believed that this dataset would be an appropriate testbed for future research in this direction.

In brief, the contributions of this study are as follows.

Diversity and difficulty This dataset offers several advantages over existing datasets: (i) Covers 2.4k healthcare topics and 21 medical subjects with an average token length of 12.77, the diversity of questions in MedMCQA demonstrate challenges unique to the dataset. (ii) It is larger than pre-existing Medical QA datasets, (iii) As these questions are from real-world and mock examinations, all the questions and candidate options are created by human experts. These questions are a comprehensive evaluation of a medical practitioner’s professional skills, (iv) The questions are difficult & challenging. They test the 10+ reasoning abilities of a model across a wide range of medical subjects & topics.

Quality Detailed statistics, analysis of the data, and fine-grained evaluation per medical subject are provided, yielding a more precise comparison between models. Each sample contains a question, correct answer(s), other options, and a detailed explanation of the solution.

Evaluation of quality Extensive experiments are conducted using high-performance pre-trained medical domain models. Error analysis is also provided to illustrate the major challenges of this task. The baseline experiments on this dataset with the most current state-of-the-art methods answer only 47% of the question correctly, which is far behind the human performance of 90%, indicating possibilities for improvement in models’ reasoning ability & constitutes a challenging benchmark for future research.

Reproducible exam-based split The dataset is split based on the exams instead of a question-based split (explained in section 2.4). This ensures that the evaluation is closer to the real-world examinations, model generalizability, and reusability. Individual Examinations tend to have similar questions or pattern of questions repeated periodically. Exam based split avoid this leakage of similar questions into test set, hence helping in generalizability of the dataset. The dataset code to reproduce the experiments & the leaderboard to track the progress of MedMCQA are available at medmcqa.github.io

The MedMCQA Dataset

In this section, properties of the MedMCQA dataset are presented. Data collection, preparation, preprocessing, and train/test/development splits are discussed.

2 Dataset collection

All India Institute of Medical Sciences (AIIMS PG) & National Eligibility cum Entrance Test (NEET PG) are the two medical entrance exams conducted by All India Institute for Medical Sciences (AIIMS) & National Board of Examinations (NBE), respectively, for providing admission to the postgraduate medical courses. The applicants must have obtained an Bachelor of Medicine and Bachelor of Surgery (MBBS) from a recognized institute to appear for the exams. The exams are used to evaluate the candidates in a structured format, namely, Diagnostic Reasoning and Treatment, Pharmacology, Psychology, Biology, Physical Examination, General Management Strategies, Medical Knowledge, and many other aspects of health and general attitude demeanor of the patient and the examiners. These exams are a comprehensive evaluation of the professional skills of a medical practitioner.

In this paper, the raw data is collected from open websites and books that put together several mock tests and online test series created by medical professionals. In addition to the collected data, AIIMS & NEET PG examination questions (1991- present) from the official websites are also used to create the MedMCQA.

The dataset contains MCQs with fine-grained human-labeled classes on various graduation level medical subjects. Each sample contains ID, question, correct answer, and options. Besides, an explanation of the solution is also provided.

3 Preprocessing & Quality Checks

To ensure that all the questions are answerable using textual input only, the following steps were taken to clean the raw data, considering questions from several data sources,

Questions with an inconsistent format were excluded, e.g., a question where the number of options was not four(excluding punctuation marks).

Questions with no best answer and missing or null candidates were also omitted.

Questions whose validity relied on external information were filtered, i.e., the articles and questions containing images or tables.

Questions containing the keywords “equation”, “India”, “graph”, “map” etc., were removed using a manually curated list of words.

Further, heuristic rules were also used. For example, in some cases, the question contained HTML tags, special symbols, URLs, extra whitespaces, and missing options. Different tools were used, e.g., a spell checker, an HTML parser, to identify and correct these cases.

A proofreading tool, ‘Grammarly’ was used for all the questions, options, and explanations in the dataset to fix the grammar, punctuation, and spelling mistakes. Appropriate suggestions from the tool were applied to the content with human supervision to improve the dataset’s quality. As a result, many errors could be corrected

Lastly, all duplicated questions were removed.

Additional data cleansing steps were carried out to ensure that the question has provided information that matches the data quality goals. The final dataset contains 193,155 questions.

4 Split Criteria

The goal of MedMCQA is to emulate the rigor of real word medical exams. To enable that, a predefined split of the dataset is provided. The split is by exams instead of the given questions. This also ensures the reusability and generalization ability of the models.

The training set of MedMCQA consists of all the collected mock & online test series, whereas the test set consists of all AIIMS PG exam MCQs (years 1991-present). The development set consists of NEET PG exam MCQs (years 2001-present) to approximate real exam evaluation.

In the dataset, leakages of similar questions from the training data to test and dev could artificially inflate the models’ performance. This is avoided by building the development and test set to include sufficiently different training data questions.

The Levenshtein distance between each pair of questions was computed in the entire dataset. If the similarity between the two documents was larger than 0.9, the question was excluded from the development and test set. The final dataset contains 183K train examples, 6K in the development set, and 4K in the test set.

Data statistics

This dataset covers many medical subjects based on the AIIMS & NEET PG entrance exams. The train, development, and test set consist of 182,822 , 4,183 & 6,150 questions with an average token length of 12.35, 13.91 & 9.68, respectively. The general statistics of preprocessed data are summarized in table 2

An additional informative statistic is the count of unique tokens in the dataset plotted in Fig. 4. Vocabulary size is a good measure of linguistic and domain complexity associated with a text corpus and influences the models’ performance. It is observed that the length of questions and the vocabulary size in the AIIMS PG exams (test set) are larger than that of the NEET PG exams (dev. set). Hence, it can be inferred that questions from AIIMS are more complex than NEET.

Data Analysis

An analysis of the dataset is presented in the subsequent sections. The difficulty and diversity of questions and the answers were analyzed to understand the MedMCQA dataset’s properties. The complexity of MedMCQA is demonstrated by considering the question and reasoning types covered in the dataset.

In clinical medicine, a diverse number of questions are possible as it is spread over a range of topics. For example, given the description of a patient’s condition, the question might be asked for the most probable diagnosis/the most appropriate treatment or examination required/mechanism of a certain condition, etc.

The majority of the dataset questions are non-factoid and open-ended in nature and seek detailed information about the health condition. Questions in MedMCQA are fairly long, with a mean length of 12.77 words, indicating the compositional nature of questions and different levels of complexity and details covered.

To understand the types of questions in MedMCQA, 25% of questions were sampled, and their properties were analyzed manually. It was observed that 68% of the questions started with an interrogative word, which generally tends to be open-ended. The dataset also contained many dichotomous questions, which often require explanations. The diversity of questions in the MedMCQA makes it a challenging dataset containing many aspects of medical knowledge. Another distinguishing factor of this dataset is that it has questions that were created for and by human domain experts.

2 Answer types

In the dataset, each question contains four options with an average length of 2.69 tokens. Out of which, 25% examples were sampled from the development set, and the answer types are presented in Fig. 5. As shown, MedMCQA covers a broad range of answer types, which matches the analysis on questions’ contribution. The answers were manually categorized, and it was observed that answers regarding drug/medicine’s name accounted for 22.49%. Medical procedure/Treatment type aiming to determine, measure, or diagnose a condition or parameter accounted for 18.74% of answers. In comparison, 11.24% of answers were related to the quantity of dose(in unit). It was observed that side effects, causes & affected body parts accounted for 12.74%, 10.49% & 9.75% of the dataset. The rest of the answer groups contained fewer instances of the time period, adverse events & other types.

3 Subject & Topic Analysis

Fig. 8(A) in the Appendix presents the distribution of medical topics per subject for the datasets. Almost 95% of the subjects contain above 50 topics, while 70% of subjects exceed 100 topics exhibiting a plethora of medical content. Topics range from Medicine (Endocrinology, Infection, Haematology, Respiratory, etc.), Surgery (General Surgery, Endocrinology, breast, and Vascular surgery, etc.) to Radiology & Biochemistry. This wide range of topics increases the dataset’s difficulty.

4 Reasoning Types

To provide a detailed & better understanding of the datasetś reasoning types, 25% of questions from MedMCQA were sampled randomly. The reasoning types required to answer were manually analyzed. The procedure was followed, and the annotation types presented in (Clark et al., 2018) were re-used to categorize them into the following reasoning types:

Question logic In this, the reasoning is tested by excluding the distractor.

Factual These are the questions that have facts as answers.

Explanation/definition The questions that require selection of definition or explanation or a term/phenomenon.

MultiHop Reasoning To answer these questions, the reasoning is required from multiple passages.

Analogy In these types of questions, the responder must select the most similar/analogous answer.

Teleology/purpose Requires understanding of the purpose of a phenomenon/a thing.

Comparison Questions that require reasoning by comparing multiple options.

Fill in the blanks The responder selects the most appropriate answer suitable to fill the blanks.

Natural language inference Determining whether a hypothesis is true, false (contradiction), or neutral given an assumption.

Mathematical Questions that require mathematical critical thinking and logical reasoning.

Treatment Questions that require selection of a correct treatment method for a given ailment / condition.

Diagnosis Questions that require selection of a correct cause of a given ailment / condition.

Fig. 6 shows statistics & examples of major reasoning types in the dataset.

Baseline Models

The primary motivation of the baseline experiments is to understand the adequacy of the current models in answering multiple-choice questions meant for human domain experts (post-graduate medical students) and to understand the level of domain specificity required in the models. Therefore, models and knowledge sources with varying levels of specificity are selected. We consider four existing models in our baseline experiments.

They are based on different pre-trained language models using Transformers architecture (Vaswani et al., 2017) , including BERT (Devlin et al., 2019) , SciBERT (Beltagy et al., 2019), BioBERT (Lee et al., 2020) and PubmedBERT(Gu et al., 2022). We fine-tuned these models on our training dataset in a multiclass classification fashion. We consider models of base size. BERT is evaluated for its out-domain pretraining, SciBERT and BioBERT for their mixed domain and in-domain continual training, and PubmedBERT for its in-domain pretraining. These models are explained in detail in the following section,

SciBERT (Beltagy et al., 2019) is a pretrained language model based on BERT. The model has been pre-trained from scratch on 1.14M papers on the semantic scholar. Even though SciBERT has been pre-trained from scratch, it has a mix of computer science (18%) and biomedical domain (82%), making it a mix-domain pretrained model. The uncased version of the model that uses a vocabulary called scivocab is used, which is a domain-specific vocabulary of size 30K

2 BioBERT

BioBERT (Lee et al., 2020) is the first biomedical domain-specific pretrained language model based on BERT. The model is initialized with standard BERT weights (pretrained from Wikipedia and BookCorpus), and continual pretraining is performed with PubMed abstracts and full texts. The model uses the same vocabulary as the standard BERT model. The base variant of the 1.1 version of the model is used in the experiments.

3 PubMedBERT

PubMedBERT (Gu et al., 2022) is a recent domain-specific pre-trained language model that is first to pretrain only on in-domain texts (PubMed abstracts

and full texts). The base version of the model trained with both abstracts and full texts is used in the experiments. This model is used to evaluate the performance of a fully in-domain pre-trained model on the dataset.

4 Retriever models

With the recent success of neural retrievers, dense passage retrieval (Karpukhin et al., 2020), and PubMedBERT(Gu et al., 2022) were utilized to evaluate Wikipedia and PubMed as knowledge bases, respectively. Dense passage retriever follows a siamese/bi-encoder architecture; One encoder encodes the documents and another to encode the query, originally trained with Maximum inner product search objective. The pretrained DPR model and Wikipedia index from Transformer’s library (Wolf et al., 2020) were used in the experiments.

Experiments

To complement the motivation stated in section 5, The reader models were chosen with varying domain specificity levels. The contribution of external knowledge sources (Wikipedia and PubMed) was evaluated by providing these sources as contexts. Furthermore, an ablation study was also performed on context by training and evaluating all the models without context. This was done to understand the contribution of external context and the usefulness of the internal knowledge stored in these domain-specific models. The baseline experiments are broadly classified as follows,

Out-Domain: Pre-trained models trained on out-domain corpora like Wikipedia and Book corpus were used in this experiment type.

Mix domain (continual): Pre-trained models trained on out-domain initially and later adapted to in-domain or trained from scratch on both out-domain and in-domain corpora were used in this experiment.

In-Domain: Pre-trained models trained from scratch on in-domain corpora like PubMed abstracts and full texts were used in this experiment type.

All these experiments were repeated with and without external knowledge context.

Before encoding the passages, the passages were truncated to 250 token lengths to fit the memory.

2 Retriever

For the experiments that involve context, a retriever+reader pipeline approach was opted (as introduced in (Chen et al., 2017)). The out-of-the-box retriever models were used (explained in the section 6.2) from Huggingface’s Transformers library (Wolf et al., 2020) to encode the passages and questions. The passage with the highest cosine similarity was retrieved and used as a context for training the reader models.

3 Reader finetuning

The finetuning approach was followed as in (Devlin et al., 2019) to finetune the reader models. The highest scoring contexts for each question are retrieved from the retriever. These contexts are combined by [SEP] token with the concatenation of question and answer pair. This creates four input sequences per question.

[CLS] Context [SEP] Question [SEP] Option [SEP]

A linear layer with softmax is applied over the output of the [CLS] token of the encoder. This is to select the most appropriate option for a question and context pair.

For the experiments that do not use context, question and answer pair concatenation is encoded, and a linear layer with softmax is applied over the output of the [CLS] token of the encoder to select the most appropriate option for a question.

The models were finetuned on two Tesla T4 GPUs for 5 epochs with a learning rate of 2e-4 and a batch size of 16. The model checkpoint with the highest validation score in the 5 epochs was selected and used to evaluate the Test Set.

Error Analysis

The error analysis details on a sample set of mispredictions by the best baseline model (PubMedBERT) is given in this section. The analysis was done manually for about 100 mispredictions that were sampled.This could be used for further research to improve the models/methods on the dataset.

Multi-hop reasoning: It was observed that the model often mispredicted the questions related to the cause of an event (diagnosis) and the right course of action (treatment) in a given medical situation. Such questions typically require information on multiple symptoms, ailments, and treatments to select the most appropriate choice. This multiplicity of information is not likely to be present in one passage, possibly the reason for the mispredictions.

Incorrect context passages: It is observed that inadequate contexts from the retriever are also major contributors to the mispredictions.

It is found that the models mispredicted the questions requiring arithmetic reasoning. This is in line with the observations in (Dua et al., 2019) on BERT-based models.

Result & Discussion

In this section, the results from the evaluation of the methods discussed in section 6 are presented.

It is observed that PubMedBERT performs better than other models in all the categories. This aligns with the results from (Gu et al., 2022) where PubMedBERT surpasses all other biomedical models in the majority of BLURB tasks. Examples of correct and incorrect predictions of the model is presented in Table Predictions from the best model

PubMedBERT is followed by SciBERT (mix domain pretraining) and BioBERT (continual pretraining) in accuracy. From this result, it can be inferred that the model’s performance decreases with a decrease in domain specificity of the models and external knowledge sources.

It is observed that there is an insignificant improvement in the model’s performance when Wikipedia is used as context compared to without context results, and the model variants trained on PubMed, which have a 4-7% improvement in the performance. This can be attributed to the domain specificity of the external knowledge source required by the dataset. The majority of the reasoning types (Diagnosis, treatment, etc.) mentioned in 4.4 require domain expertise as these questions are intended for post-graduate medical students.

The subject wise accuracies of the top PubMedBERT model is presented in Table 3

Conclusion

In this work, MedMCQA, a new large-scale, Multi-Choice Question Answering (MCQA) dataset, is presented, which requires a deeper domain and language understanding as it tests the 10+ reasoning abilities of a model across a wide range of medical subjects & topics. It is demonstrated that the dataset is challenging for the current state-of-the-art methods and domain-specific models, with the best baseline achieving only 47% accuracy. It is expected that this dataset would facilitate future research in this direction.