EHRSQL: A Practical Text-to-SQL Benchmark for Electronic Health Records

Introduction

Electronic health records (EHRs) are relational databases that store a patient’s entire medical history in the hospital. From hospital admission to patient treatment and discharge, all medical events that happened in the hospital are recorded and stored in the EHR. The records cover a wide range of clinical knowledge, from individual-level information to group-level insight, in various forms, including tables, text, and images . As a vast and comprehensive knowledge base, hospital staff, including physicians, nurses, and administrators, constantly interact with EHRs to store and retrieve patient information to make better clinical decisions .

Meanwhile, most hospital staff interact with the EHR database using pre-defined rule conversion systems. To look for information beyond the rules, one needs to undergo special training to modify and extend the system . As a result, it is a massive bottleneck for the users to fully utilize the information stored in the EHR. An alternative way to tackle this problem is to build a system that can automatically translate questions directly into the corresponding SQL queries. Then, the users can simply type or verbally ask their questions to the system, and it will return the answers without going through any complicated process. Such systems will unleash the potential of EHR data and tremendously speed up any task involving data retrieval from the database.

Existing datasets that tackle question answering (QA) over structured EHR data are MIMICSQL and emrKBQA . MIMICSQL is the first dataset for healthcare QA on MIMIC-III , where the questions are automatically generated with pre-defined templates. emrKBQA is another dataset on MIMIC-III derived from emrQA , a clinical reading comprehension dataset. However, through a poll at a university hospital, we discover that the existing datasets are far from fulfilling the actual needs in the hospital workplace in several aspects.

We present EHRSQLThe dataset is distributed under the CC BY-SA 4.0 license., a new large-scale text-to-SQL dataset linked to two open-source EHR databases—MIMIC-III and eICU . To the best of our knowledge, our work is the first EHR QA dataset that reflects the diverse needs of hospital staff while introducing practical issues in deploying healthcare QA systems. The unique challenges posed by our dataset are threefold:

Wide range of questions: We conducted a poll at a university hospital to collect questions frequently asked on structured EHR data. The total number of respondents was 222 people with varying years of experience in their professions (see Figure 2). After filtering and templatizing the responses, the resulting templates cover a wide range of questions, including retrieving patient records (e.g., vital signs measured and hospital cost) and conducting complex group-level operations (e.g., retrieving the top N medications prescribed after being diagnosed with a disease and calculating the N-year survival rate). See Table 1 for more examples.

Time-sensitive questions: Based on the poll, we learned that real-world questions in the hospital workplace are rich in time expressions, as time is one of the most crucial aspects of healthcare. To reflect this in the dataset, we systematically categorized time into multiple expression types (e.g., absolute, relative, and mixed), units (e.g., hospital visit, month, and day), and interval types (e.g., since, until, and in). Then, we combined the categorized time with the question templates to simulate time-sensitive questions asked in the hospital. More details are discussed in Section 3.1.2.

Trustworthy QA systems: Developing trustworthy systems is crucial for the adoption of AI in healthcare. Likewise, QA systems need to return only accurate answers and refuse to answer questions beyond their capabilities. To test this, we include unanswerable questions in the dataset, utilizing the remaining utterances from the poll result. These utterances are unanswerable due to incompatibility with the database schema or requiring external domain knowledge.

Related Works

Pampari et al. proposed emrQA for question answering on clinical notes based on physicians’ frequently asked questions. Later, Raghavan et al. proposed emrKBQAemrKBQA is yet to be publicly released at the time of this work., which adapted emrQA to structured patient records in MIMIC-III. Due to the nature of the original dataset targeting outpatients, the scope of the questions in emrKBQA is limited to mostly asking about patients’ test results . Recently, Wang et al. proposed MIMICSQL, which first tackles EHR QA with the text-to-SQL generation task. To construct the dataset, they automatically generated questions based on pre-defined rules and filtered them through crowd-sourcing. However, MIMICSQL contains a limited scope of questions and simple SQL queries restricted to five tables, which are far from the actual use case in the hospital . EHRSQL, on the other hand, originates from the actual poll result and covers a variety of real-world questions (spanning 13.5 tables on average) frequently asked on structured EHR data (see Table 1).

Semantic parsing has been one of the most active areas of research in natural language processing over the past few decades . It involves converting natural language utterances to logical form representations, which are often executable programs such as Python scripts and SQL queries. WikiSQL and Spider are leading datasets that assess semantic parsing models on unseen databases. WikiSQL provides a wide range of database domains extracted from HTML tables in Wikipedia, but it only contains simple SQL queries and single tables. Spider aims to tackle complex queries, including joining and set operations, on multiple tables in multiple database domains. Recently, more realistic datasets have been proposed to bridge the gap between academic and practical settings in semantic parsing. KaggleDBQA utilizes real-world databases from Kaggle and constructs domain-specific questions without relying on the database schema. SEDE deals with complex SQL queries that Stack Exchange users ask in real life. Our motivation aligns with theirs in that additional challenges arise in real-world healthcare QA systems.

In the early days of question answering, unanswerable questions were generated via distant supervision , rule-based editing , or adversarial creation by crowd workers . Yet, in semantic parsing, most works assume that all input questions are valid and can be answered; however, this is not true in practice . Retrieving answers to all the input questions is not always desirable for the model to ensure system reliability. To address this, Zhang et al. utilized other text-to-SQL and chit-chat datasets to construct unanswerable questions and posed the problem as a classification task to detect unanswerable questions. Our work, on the other hand, addresses both system reliability and semantic parsing simultaneously and the unanswerable questions are naturally collected through the poll. In task-oriented dialog systems, the same task has been tackled in the name of Out-Of-Domain (OOD) detection, and several recent works approach this problem without explicitly showing OOD samples to the model during training for practical reasons. Following their setup, our unanswerable questions are only included in the validation and test sets.

Dataset Construction

The motivation behind our work is to construct a dataset that reflects the actual needs of hospital staff and tackles several practical issues that can arise in real-world healthcare QA systems (see Table 1). To this end, we collaborated with the Konyang University Hospitalhttps://www.kyuh.ac.kr/eng/ and conducted a poll to collect real-world questions as if one is asking an AI speaker what they frequently look for in structured information in the EHR. In addition, to clarify what machines can and cannot answer, we provided typical negative examples of what should not be asked, such as requiring external knowledge, ambiguous or qualitative statements, or asking for reasons behind clinical decisions. As a result, we gathered a total of 1,742 utterances, and the number of valid respondents was 222 (see Figure 2 for respondent demographics).

1 Question and SQL Generation

After the poll, we filtered out the utterances that did not meet the criteria, including ambiguous statements, those that require external knowledge, or those that go beyond the database schema. We then categorized the utterances into three patient-based scopes: a single patient, a group of patients, and no patient. For each scope, duplicate utterances, but differently phrased, were merged into a single question template. We also manually added question templates that were simple variants of existing templates. For example, we modified the type of medical events (e.g., lab test, prescription, etc.) in question templates, such as from “Count the number of times that a patient had a lab test” to “Count the number of times that a patient received a prescription.” To cover more questions on longitudinal statistics, we linked two different medical events together whenever possible. For example, the question “What are the top N prescriptions after a diagnosis,” which was originally collected, is extended to “What are the top N lab tests after a diagnosis.” Finally, the filtered-out utterances were also templatized and considered unanswerable. The unanswerable questions included side effects, next check-up schedules, primary care doctor, patient’s personal information, whether a patient has signed a consent form, diagnosis received in other departments, and more.

The resulting question templates are natural utterances with slots that are later filled with pre-defined values or database records (see Table 2). Slots containing “time_filter” are processed in the time template sampling stage, described in Section 3.1.2. We eliminate any ambiguity in the question templates that could cause the model to infer table or column names incorrectly. For example, “When did this patient get [slot] today?” is removed because it does not tell us whether we are interested in drugs, lab tests, or something else unless the slot is filled. Such linguistic ambiguity is later introduced in the paraphrase generation stage described in Section 3.3. As a result, we curated 230 question templates (174 for answerable and 56 for unanswerable) based on the MIMIC-III and eICU schemas. For answerable questions, their corresponding SQL queries were labeled for each database, which is further discussed in Section 3.1.3.

1.2 Time template

Since the healthcare domain is inextricably linked to time, many utterances we collected were rich in time expressions. To reflect this in the dataset, we developed three time filter types and assigned different time factors (expression types, units, and interval types) to each filter type to compose a single time template. First, we define a “global” time filter ([time_filter_global]) that constrains the full range of time we are interested in. Second, within the global time filter, we can also use a “within” time filter ([time_filter_within]) to indicate the time range between two or more medical events. Third, we can point to an exact time ([time_filter_exact]), such as “last” measurement or “at 2105-12-31 09:00:00” if we know the exact time or the order of an event.

Each time filter type has three factors and an extra option: 1) Expression type is whether the filter is phrased in an absolute, relative, or mixed time expression (e.g., last year, in 2022, etc.); 2) Unit is the granularity of time, such as a standardized unit (e.g., year, month, day, etc.) or an arbitrary unit of events (e.g., hospital visit, ICU visit); 3) Interval type specifies the type of time interval (e.g., since, until, in, etc.). Depending on the use case, Option allows to choose one exact event (e.g., first, last) among the filtered events. Table 3 shows sample time templates and their natural language (NL) time expressions. It is important to note that each time template has its corresponding NL time expression (e.g., since {month}/{day}/{year}) and SQL time pattern (e.g., WHERE strftime('%Y-%m-%d',[time_column]) >= '{year}-{month}-{day}'), while the question templates are originally created in natural language. The NL time expressions are combined with the question templates to form complete questions. The SQL time patterns are combined with the labeled SQL queries to form complete SQL queries (see Section 3.1.4 for details). The full list of time templates (NL and SQL pairs) is reported in Supplementary B.2.

1.3 SQL annotation

The SQL annotation process was conducted manually by four graduate students over the course of five months, with repeated revisions. Since the questions were collected independently of the database schema, SQL labeling required numerous assumptions (e.g., which medical event is mapped to which column in the database, the choice of rank functions in SQL, the occasion of using DISTINCT, etc.). To sync template and value sampling, the same slots introduced in Section 3.1.1 and 3.1.2 (e.g., {drug_name}, [n_rank], [time_filter_global]) were also used in SQL labeling as placeholders. The SQL queries for question and time templates were labeled by one person, followed by a reviewer for each database. Unlike most SQL datasets, the students were asked to avoid using JOIN, which is a go-to operation when retrieving information across multiple tables. In MIMIC-III and eICU, several tables contain more than 100 million rows (e.g., 330 million rows in MIMIC-III chartevents) as the records are stored in a “log” based manner. In fact, it is extremely inefficient to merge all the tables in such large databases without understanding what specific columns are needed to answer a question. As a result, the students were asked to utilize the hierarchical structure of the EHR schema, as illustrated in Figure 2, and labeled each query in a nested manner whenever possible. SQL annotation details and the comparison between JOIN and nesting-based queries are discussed in Supplementary C.

1.4 Template combination and data generation

Data generation starts by selecting a question template, followed by three sampling stages that add semantic variety: operation value sampling, time template sampling, and condition value sampling (See Figure 3). Stage 1 involves sampling operation values from pre-defined values that are independent of the database schema or records (e.g., maximum, five-year, two or more times, etc.). Then, time templates are sampled in Stage 2 based on the time filter types that each question template can have. Stage 2 in Figure 3 illustrates that the time templates ([time_filter_global] and [time_filter_within]) have already been sampled and converted into NL time expressions. More details are discussed in Supplementary B.3. Finally, condition value sampling (e.g., {diagnosis_name}, {year}) occurs in Stage 3, and the slot filling process is complete. This final stage is where the actual database records are sampled, and all generated SQL queries are checked to have at least one valid answer. Once the query is successfully executed, the pair of SQL and the corresponding question is added to the data pool, which is later used for data splitting.

2 Database Pre-processing

We modified the original MIMIC-III and eICU databases to include a cost table in order to reflect hospital administration and insurance-related questions. When constructing the table, we referred to the OMOP Common Data Modelhttp://ohdsi.github.io/CommonDataModel/cdm54.html to create new column names, which include: patient_id, hospital_admission_id, event_type (cost_domain_id), event_id (cost_event_id), chargetime, and cost. For cost sampling, we take a two-step sampling procedure to simulate cost values. First, we sampled discrete-valued mean costs for four different medical event types (diagnosis, procedure, prescription, and lab events) from a Poisson distribution with a mean of 10. Then, we sampled continuous-valued costs from Gaussian distributions with their corresponding means.

The time span of MIMIC-III is over a hundred years due to the de-identification process (originally seven years), while eICU’s time span remains intact (two years). To simulate a more realistic time span asked in the hospital and match the time across the databases, we shifted the admission time of every patient’s records ranging from 2100 to 2105. In addition, to incorporate the concept of current and the use of relative time expressions, we set the current time to be ‘‘2105-12-31 23:59:00’’ and removed any record past the current time. In this way, patients with missing hospital discharge time are considered current patients. The poll result revealed that many time expressions used in the hospital are relative time expressions (e.g., today, yesterday, last month, etc.), and therefore this process was necessary to reflect time-sensitive questions. More details are reported in Supplementary D.2.

MIMIC-III and eICU databases are de-identified datasets, and a user needs to request credentialed access to PhysioNethttps://physionet.org/ to obtain them. Both datasets, however, do contain real patient records, and the questions derived from them could potentially reveal patient-specific information. For example, the question “What medication was given to patient ID 1234 after being diagnosed with diabetes?” implies that this patient was diagnosed with diabetes. Combined with some unforeseen external knowledge, this might lead to recovering the patient’s identity. To add another layer of de-identification, we randomly shuffled values across all patients in the database before sampling condition values. In this way, the semantic structure of the question remains the same, but sampled condition values are untraceable. More details of the de-identification process are reported in Supplementary D.3.

3 Paraphrase Generation

To add linguistic variety to the questions, we generated template paraphrases from the question templates with manual paraphrasing and the help of machine learning models. Before paraphrasing, we ensure that the templates do not contain any domain-specific expressions, creating the paraphrasing process departing from the healthcare domain to leverage general-domain tools. The overall procedure is as follows:

Human paraphrasing is first conducted to add more high-quality templates, averaging 21 paraphrases per question template.

Based on the human paraphrases, more paraphrases are generated using machine learning models, such as T5 paraphrasers and multilingual translation models for back-translation .

The paraphrases that are too different from the original meaning are filtered using a duplicate question detection model, specifically RoBERTa-large trained on the Quora duplicate question detection dataset.

The paraphrases are ranked by the perplexity score from GPT-Neo 1.3B per question template and disregarded if they are too similar to the other paraphrases with lower perplexity. The Levenshtein distance is used to detect lexical similarity.

The final paraphrases are reviewed by crowd workers to rate the quality of the paraphrases. In our case, we collaborated with a crowd-sourcing company named Selectstarhttps://selectstar.ai/en. Three groups of annotators were assigned to this task and marked a pass or fail for each paraphrase sample. The samples with unanimous pass marks were considered the final machine-paraphrased templates, leaving 47 paraphrases per template on average.

During the paraphrasing process, slots are replaced with generic values (e.g., {patient_id} $\rightarrow$ the patient) and returned to the original slots after the process is complete. Later, those slots are filled with the actual condition values in the value sampling process to construct the final question-to-SQL pairs. The overview of the paraphrase pipeline is illustrated in Supplementary E.

4 EHRSQL and Other Datasets

Table 4 summarizes the statistics of EHRSQL and other text-to-SQL datasets in the literature. Compared to general domain datasets (first three rows), EHRSQL has a large number of text-to-SQL pairs (# Example), tables per database (# Table/DB), and rows per table (# Row/Table). KaggleDBQA and SEDE are designed to bridge the gap between academic datasets and practical usability by using real databases and naturally-occurring utterances. However, we have gone one step further where the question authors (the poll respondents) were not presented with the database schema (?Schema), which adds more reality to the dataset . Finally, EHRSQL contains unanswerable questions (UnANS) that were collected together from the poll, which may play a critical role in assessing the reliability of the QA model. To the best of our knowledge, EHRSQL is the first attempt to collect and combine answerable and unanswerable questions in the context of text-to-SQL.

From a healthcare perspective, EHRSQL covers a variety of questions frequently asked in the hospital. The scope of these questions ranges from non-patient information (e.g., the cost of a procedure) to individual-level information (e.g., retrieving patient demographics and the lab values) and group-level information (e.g., counting the number of patients in the hospital and the top N frequently prescribed drugs of patients over 60s), further combined with various time expressions (% Time Used/Q). The SQL queries are linked to two EHR databases: MIMIC-III and eICU, and this is the first attempt to label SQL queries on eICU, allowing questions about multi-center critical care to be answered.

Benchmarks

In this section, we define a new text-to-SQL task that assesses models under more realistic healthcare QA settings than prior works—namely, trustworthy semantic parsing. As illustrated in Figure 4, the trustworthy model needs to 1) generate SQL queries that reflect a wide range of needs in the hospital workplace, 2) understand various time expressions to answer time-sensitive questions in healthcare, and 3) have the capacity to distinguish whether a given question is answerable or unanswerable based on the prediction confidence. Among various ways to calculate the confidence score, we argue that a desirable way to calculate it is not based on the data the model is trained on, but on the model prediction itself. As illustrated in Figure 4, once a confidence score exceeds some threshold of choice (assuming that a greater score means less confidence), the generated SQL should not be sent to the database.

To make the task feasible, we keep the condition values (e.g., prescription name) the same in this work, except for genders and vital signs, as this is another major challenge in semantic parsing . When splitting the dataset into train, validation, and test sets, we ensure that all the question templates are present in each split. To create a more realistic QA setting, we include unanswerable questions only in the validation and test sets (consisting of 33% of each split), following the OOD detection setting . Among 24,411 question pairs in the dataset, the train, valid, and test splits contain 9.3K, 1.1K, and 1.8K pairs, respectively, for each database. We release the train and valid splits, totaling 21K samples, and the other 3K samples are saved for a hidden test set. Details of data splitting are reported in Supplementary F.

2 Evaluation

The goal of the trustworthy semantic parsing task is to correctly return the answers to answerable questions while disregarding unanswerable questions. This process can be evaluated in two aspects. The model first needs to have a strategy that distinguishes whether a given question is answerable or unanswerable. Given that the goal is to correctly recognize as many answerable questions as possible, the model’s performance can be measured in precision and recall. In this case, the precision (Pans) calculates the number of correctly recognized answerable questions among all questions predicted to be answerable, while recall (Rans) calculates the number of correctly recognized answerable questions among all answerable questions. We use F1ans to combine these two scores.

Next, we must evaluate how well the model generates correct SQL queries, given that some questions are considered answerable and ready to predict. The word predict here means the actions of generating a query and sending it to the database, where extra care is needed because the answer from the database may be directly used for clinical decision-making. Combining the concept of F1ans and execution accuracy in semantic parsing, F1exe counts only when the returned answer is correct. In other words, F1exe is a combined score of Pexe and Rexe, where Pexe is the ratio of the number of correctly answered questions to all questions predicted to be answerable and Rexe is the ratio of the number of correctly answered questions to all answerable questions. As a result, the denominators of the precisions and recalls are the same, but the numerators are more strictly counted in metrics with ${exe}$. This gap between ${ans}$ and ${exe}$ reveals the model’s inability to generate correct SQL queries, given that the questions are both recognized as answerable and indeed answerable.

3 Model Development

Most state-of-the-art (SOTA) semantic parsing models utilize grammar-based decoders and are only compatible with the grammars (or their subsets) defined in the Spider dataset. As the queries in EHRSQL are created in response to actual needs, most of them are incompatible with the Spider parser mainly due to time-related operators like strftime, datetime and NULL. Therefore, we chose general-purpose sequence-to-sequence models, T5-base and T5-base with schema serialization , as baseline models for our task. This choice aligns with a recent finding that transfer learning from pre-trained language models surpasses healthcare-specific text-to-SQL models . Training details are reported in Supplementary G.

To give a model the ability to refuse to answer questions, we adopt a simple uncertainty estimation method inspired by . If the maximum entropy during the decoding process exceeds a pre-defined threshold, we consider this a refusal. The threshold values are determined by two heuristic approaches, and they are compared with the result obtained without refusal.

Clustering-based: Run K-means clustering with $k=2$ on all maximum entropy values of validation samples, assuming that high-entropy samples are from unanswerable questions. The decision boundary between the two clusters is then used as the threshold.

Percentile-based: Set the threshold to the 67th percentile of the maximum entropy values of the validation samples, based on the fact that 33% of the questions in the validation set are unanswerable.

To illustrate the domain gap between general-domain and healthcare datasets, we create a subset of EHRSQL that could be parsed with the Spider parser. Then, we compare how well the SOTA cross-domain semantic parsing models generalize to healthcare text-to-SQL datasets. We choose MIMICSQL, which contains simple SQL queries that can all be parsed with the Spider grammar, and EHRSQL for the healthcare datasets. Among many SOTA models in the Spider leaderboard, we use Generation-Augmented Pre-training (GAP) to test its zero-shot domain transfer performance.

4 Results and Findings

The baseline results are reported in Table 5. Among the three different threshold approaches, the percentile-based threshold performs best on both the validation and test sets. This result is an expected outcome because the ratios of the answerable and unanswerable questions in the validation and test sets are kept the same in our setting. As for the clustering-based method, an additional training or clustering technique is needed for the models to better discriminate the entropy values. The naive entropy values from the current models are not linearly separable across predictions (see Figure 7 in Supplementary H.4 for maximum entropy distributions). T5+Schema shows comparable performance to T5 in both databases. This result agrees with the recent finding in that models trained in a single database setting do not effectively leverage schema information. Additional qualitative results are provided in Supplementary H, including SQL generation results by question complexity, time expressions, falsely executed results, and refused results.

Table 6 shows the performance of zero-shot cross-domain transfer with the GAP model. Unlike the queries in MIMICSQL, which are all parsable with the Spider grammar, EHRSQL has much more complex SQL operators and structures, with only about 7% (107/1,515) of the answerable questions in the validation set being parsable. As for model performance, GAP achieves 16.4% in the full MIMICSQL validation set and 5.8% in the subset of the validation set in EHRSQL. This result motivates again the need for a new practical text-to-SQL dataset in healthcare and QA models that can handle multiple real-world challenges in the hospital.

Conclusion and Future Direction

In this paper, we present EHRSQL, a new practical text-to-SQL dataset for question answering over structured information in EHRs. Through a poll conducted at a university hospital, we collected questions that are frequently asked on structured EHR data across various professions in the hospital. The questions reflect the actual needs in the hospital and different time expressions used in daily work, which are particularly crucial in healthcare. Additionally, we also collected unanswerable questions, which were the questions submitted by the respondents but turned out to be beyond the EHR schema or ambiguous. Finally, we manually labeled SQL queries for two open-source EHR databases—MIMIC-III and eICU—and cast these challenges as one task—trustworthy semantic parsing—where QA models should only answer the question when their predictions are confident but not otherwise.

Though we have carefully designed the dataset, there are several limitations. First, despite more than two hundred people participating in the poll, the source of the questions is from one Korean university hospital, which may not reflect every unique situation in different hospitals worldwide. Secondly, the question templates are paraphrased using general-domain paraphrasers; therefore, paraphrases with the heavy use of medical jargon could make them more realistic. Finally, SQL labels for two EHR databases might not be a sufficient number of databases to train and test a model for unseen EHR databases.

We expect numerous research directions with our dataset. The seed questions can be a valuable resource for expanding the scope of table-based healthcare QA tasks, such as creating interactive QA or developing it into multimodal QA datasets on EHRs. With the idea of trustworthy semantic parsing, a new class of end-to-end uncertainty-aware semantic parsing models can be proposed. As existing text-to-SQL models assume all inputs are answerable, this can be a unique venue for semantic parsing to bridge the gap between research and industrial needs.

Acknowledgments and Disclosure of Funding

We would like to thank five anonymous reviewers for their time and insightful comments. We also thank Woochan Hwang for assisting us in clarifying our questions during the data labeling process. This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No.2019-0-00075, Artificial Intelligence Graduate School Program(KAIST)), National Research Foundation of Korea (NRF) grant (NRF-2020H1D3A2A03100945) and Data Voucher grant (2021-DV-I-P-00114), funded by the Korea government (MSIT).

References

Supplementary Material

Appendix A Datasheet for Datasets

The following section is answers to questions listed in datasheets for datasets.

For what purpose was the dataset created? EHRSQL is created to serve as a benchmark for trustworthy question answering systems on structured data in electronic health records (EHRs).

Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)? The authors of this paper.

Who funded the creation of the dataset? If there is an associated grant, please provide the name of the grantor and the grant name and number. This work was supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant (No.2019-0-00075, Artificial Intelligence Graduate School Program(KAIST)), National Research Foundation of Korea (NRF) grant (NRF-2020H1D3A2A03100945) and Data Voucher grant (2021-DV-I-P-00114), funded by the Korea government (MSIT).

A.2 Composition

What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? EHRSQL contains natural questions and their corresponding SQL queries (text).

How many instances are there in total (of each type, if appropriate)? There are about 24.4K instances (22.5K answerable; 1.9K unanswerable).

Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? We conducted a poll at a university hospital and collected a wide range of questions frequently asked on the structured EHR data. To reflect as many questions as possible, we templatized them and ensured that the final dataset contained all the question templates we created.

What data does each instance consist of? The dataset contains question-SQL pairs if the question is answerable. Unanswerable questions do not have SQL labels.

Is there a label or target associated with each instance? Labels are SQL queries.

Is any information missing from individual instances? If so, please provide a description, explaining why this information is missing (e.g., because it was unavailable). This does not include intentionally removed information, but might include, e.g., redacted text. N/A.

Are relationships between individual instances made explicit (e.g., users’ movie ratings, social network links)? N/A.

Are there recommended data splits (e.g., training, development/validation, testing)? See Section F.

Are there any errors, sources of noise, or redundancies in the dataset? Question templates are created to have slots that are later filled with pre-defined values and records from the database. As a result, final questions can sound unnatural or be grammatically incorrect depending on the sampled values (e.g., verb tense, articles, etc.).

Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? The labeled SQL queries rely on two open source databases: MIMIC-III (version 1.4)https://physionet.org/content/mimiciii/1.4/ and eICU (version 2.0)https://physionet.org/content/eicu-crd/2.0/, which are accessible on PhysioNethttps://physionet.org/.

Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor– patient confidentiality, data that includes the content of individuals’ non-public communications)? N/A.

Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? N/A.

Does the dataset identify any subpopulations (e.g., by age, gender)? EHRSQL is based on patients in MIMIC-III and eICU. MIMIC-III includes over forty thousand patients who stayed in critical care units of the Beth Israel Deaconess Medical Center between 2001 and 2012. eICU contains patients who were discharged between 2014 and 2015 in multiple critical care units in the United States.

Is it possible to identify individuals (i.e., one or more natural persons), either directly or indirectly (i.e., in combination with other data) from the dataset? Even though MIMIC-III and eICU are already de-identified datasets, we further corrupted patient-specific information to avoid any chance of recovering a patient’s identity. See Section LABEL:ssec:de-idenfication for details.

Does the dataset contain data that might be considered sensitive in any way (e.g., data that reveals race or ethnic origins, sexual orientations, religious beliefs, political opinions or union memberships, or locations; financial or health data; biometric or genetic data; forms of government identification, such as social security numbers; criminal history)? The dataset is already de-identified.

A.3 Collection Process

How was the data associated with each instance acquired? We collaborated with the Konyang University Hospitalhttps://www.kyuh.ac.kr/eng/ and conducted a poll to collect real-world questions that are frequently asked on the structured EHR data.

What mechanisms or procedures were used to collect the data (e.g., hardware apparatuses or sensors, manual human curation, software programs, software APIs)? We used the website SurveyMonkeywww.surveymonkey.com to create a poll and collect the responses. After the poll, we used Excel, Google Sheets, and Python to process and label the collected data.

If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)? When it involves sampling (e.g., data splitting and patient de-identification), we sampled with a fixed random seed.

Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)? There were three parts that required human involvement in the data collection process: poll for the question collection, SQL labeling, and quality checking for machine-paraphrased text. For the poll, we provided a $10 worth of coffee gift card to all poll respondents. For SQL labeling, the authors in the paper manually labeled SQL queries based on the database schemas. We did not hire any crowd worker for this task because the databases contain patient-specific information, and the SQL labeling process required numerous assumptions (e.g., choice of SQL operations, schema linking, etc.). Lastly, we hired crowd workers to check the quality of machine paraphrased text, who were paid approximately$18 per hour.

Over what timeframe was the data collected? The poll was conducted in February of 2021, but the results do not depend much on the date of date collection.

Were any ethical review processes conducted (e.g., by an institutional review board)? N/A.

Did you collect the data from the individuals in question directly, or obtain it via third parties or other sources (e.g., websites)? We directly collected the data through a poll.

Were the individuals in question notified about the data collection? Yes. The poll respondents were notified about the use of data. The actual website we used for the poll is herehttps://www.surveymonkey.com/r/Preview/?sm=hv3JkWYLdzXq2G8m_2Bh8yXI8Q_2FHVOzmZHcwFs7D5WhDYPQwgBHaa7OZXASgLWXsBw. The poll were conducted in Korean.

Did the individuals in question consent to the collection and use of their data? The purpose of the poll was announced to hospital staff, and only the staff who were interested in the poll participated.

If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future or for certain uses? N/A.

Has an analysis of the potential impact of the dataset and its use on data subjects (e.g., a data protection impact analysis) been conducted? The dataset does not have individual-specific information.

A.4 Preprocessing/cleaning/labeling

Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? N/A.

Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data (e.g., to support unanticipated future uses)? N/A.

Is the software that was used to preprocess/clean/label the data available? Preprocessing, cleaning, and labeling are done via Excel, Google Sheets, and Python.

A.5 Uses

Has the dataset been used for any tasks already? No.

Is there a repository that links to any or all papers or systems that use the dataset? No.

What (other) tasks could the dataset be used for? In addition to solving trustworthy semantic parsing, the seed questions themselves can be a good starting point for any healthcare table-based question answering tasks.

Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses? N/A.

Are there tasks for which the dataset should not be used? N/A.

A.6 Distribution

Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? No.

How will the dataset will be distributed (e.g., tarball on website, API, GitHub)? The dataset is released at https://github.com/glee4810/EHRSQL.

When will the dataset be distributed? Now.

Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? The dataset is released under MIT License.

Have any third parties imposed IP-based or other restrictions on the data associated with the instances? No.

Do any export controls or other regulatory restrictions apply to the dataset or to individual instances? No.

A.7 Maintenance

Who will be supporting/hosting/maintaining the dataset? The authors of this paper.

How can the owner/curator/manager of the dataset be contacted (e.g., email address)? Contact the first author (gyubok.lee@kaist.ac.kr) or other authors.

Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)? If any correction is needed, we plan to upload a new version.

If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were the individuals in question told that their data would be retained for a fixed period of time and then deleted)? N/A

Will older versions of the dataset continue to be supported/hosted/maintained? We plan to maintain the newest version only.

If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so? Contact the authors of the paper.

Appendix B Full List of Templates

The full list of question templates is reported in Table LABEL:tab:question_template_full. The total number of answerable question templates is 174, but a few can be unanswerable depending on the database (e.g., a question about a procedure done in other hospitals is not answerable in MIMIC-III). For unanswerable questions, the template generation process explained in Section 3.1.1 is not strictly applied. As a result, unanswerable questions can contain ambiguous and too detailed questions (e.g., Tell me what medicine to use to relieve a headache in hypertensive patients). We do not provide a full list of unanswerable questions as they are not subject to training and can be anything complementary to the answerable ones.

[ caption = Full list of answerable question templates., label = tab:question_template_full, ] colspec = X[c,m]X[8,l,m]X[1.2,c,m], colsep = 0.5pt, rowhead = 1, hlines, rows=font=, row1 = c, Patient scope & Question template Assumption None What is the intake method of {drug_name}? None What is the cost of a procedure named {procedure_name}? None What is the cost of a {lab_name} lab test? None What is the cost of a drug named {drug_name}? None What is the cost of diagnosing {diagnosis_name}? None What does {abbreviation} stand for? Single What is the gender of patient {patient_id}? Single What is the date of birth of patient {patient_id}? Single What was the [time_filter_exact1] length of hospital stay of patient {patient_id}? Only current patients Single What is the change in the weight of patient {patient_id} from the [time_filter_exact2] value measured [time_filter_global1]? Single What is the change in the weight of patient {patient_id} from the [time_filter_exact2] value measured [time_filter_global2] compared to the [time_filter_exact1] value measured [time_filter_global1]? Single What is the change in the value of {lab_name} of patient {patient_id} from the [time_filter_exact2] value measured [time_filter_global2] compared to the [time_filter_exact1] value measured [time_filter_global1]? Single What is the change in the {vital_name} of patient {patient_id} from the [time_filter_exact2] value measured [time_filter_global2] compared to the [time_filter_exact1] value measured [time_filter_global1]? Single Is the value of {lab_name} of patient {patient_id} [time_filter_exact2] measured [time_filter_global2] [comparison] than the [time_filter_exact1] value measured [time_filter_global1]? Single Is the {vital_name} of patient {patient_id} [time_filter_exact2] measured [time_filter_global2] [comparison] than the [time_filter_exact1] value measured [time_filter_global1]? Single What is_verb the age of patient {patient_id} [time_filter_global1]? Single What is_verb the name of insurance of {patient_id} [time_filter_global1]? Single What is_verb the marital status of patient {patient_id} [time_filter_global1]? Single What percentile is the value of {lab_value} in a {lab_name} lab test among patients of the same age as patient {patient_id} [time_filter_global1]? Single How many [unit_count] have passed since patient {patient_id} was admitted to the hospital currently? Only current patient Single How many [unit_count] have passed since patient {patient_id} was admitted to the ICU currently? Only current ICU patient Single How many [unit_count] have passed since the [time_filter_exact1] time patient {patient_id} stayed in careunit {careunit} on the current hospital visit? Only current patient Single How many [unit_count] have passed since the [time_filter_exact1] time patient {patient_id} stayed in ward {ward_id} on the current hospital visit? Only current patient Single How many [unit_count] have passed since the [time_filter_exact1] time patient {patient_id} received a procedure on the current hospital visit? Only current patient Single How many [unit_count] have passed since the [time_filter_exact1] time patient {patient_id} received a {procedure_name} procedure on the current hospital visit? Only current patient Single How many [unit_count] have passed since the [time_filter_exact1] time patient {patient_id} was diagnosed with {diagnosis_name} on the current hospital visit? Only current patient Single How many [unit_count] have passed since the [time_filter_exact1] time patient {patient_id} was prescribed {drug_name} on the current hospital visit? Only current patient Single How many [unit_count] have passed since the [time_filter_exact1] time patient {patient_id} received a {lab_name} lab test on the current hospital visit? Only current patient Single How many [unit_count] have passed since the [time_filter_exact1] time patient {patient_id} had a {intake_name} intake on the current ICU visit? Only current ICU patient Single What was the [time_filter_exact1] hospital admission type of patient {patient_id} [time_filter_global1]? Single What was the [time_filter_exact1] ward of patient {patient_id} [time_filter_global1]? Single What was the [time_filter_exact1] careunit of patient {patient_id} [time_filter_global1]? Single What was the [time_filter_exact1] measured height of patient {patient_id} [time_filter_global1]? Single What was the [time_filter_exact1] measured weight of patient {patient_id} [time_filter_global1]? Single What was the name of the diagnosis that patient {patient_id} [time_filter_exact1] received [time_filter_global1]? Single What was the name of the procedure that patient {patient_id} [time_filter_exact1] received [time_filter_global1]? Single What was the name of the drug that patient {patient_id} was [time_filter_exact1] prescribed via {drug_route} route [time_filter_global1]? Single What was the name of the drug that patient {patient_id} was [time_filter_exact1] prescribed [time_filter_global1]? Single What was the name of the drug that patient {patient_id} was prescribed [time_filter_within] after having been diagnosed with {diagnosis_name} [time_filter_global1]? Single What was the name of the drug that patient {patient_id} was prescribed [time_filter_within] after having received a {procedure_name} procedure [time_filter_global1]? Single What was the dose of {drug_name} that patient {patient_id} was [time_filter_exact1] prescribed [time_filter_global1]? Single What was the total amount of dose of {drug_name} that patient {patient_id} were prescribed [time_filter_global1]? Single What was the name of the drug that patient {patient_id} were prescribed [n_times] [time_filter_global1]? Single What is the new prescription of patient {patient_id} [time_filter_global2] compared to the prescription [time_filter_global1]? global filters do not overlap Single What was the [time_filter_exact1] measured value of a {lab_name} lab test of patient {patient_id}[time_filter_global1]? Single What was the name of the lab test that patient {patient_id} [time_filter_exact1] received [time_filter_global1]? Single what was the [agg_function] {lab_name} value of patient {patient_id} [time_filter_global1]? Single What was the name of the allergy that patient {patient_id} had [time_filter_global1]? Single What was the name of the substance that patient {patient_id} was allergic to [time_filter_global1]? Single What was the organism name found in the [time_filter_exact1] {culture_name} microbiology test of patient {patient_id} [time_filter_global1]? Single What was the name of the specimen that patient {patient_id} was [time_filter_exact1] tested [time_filter_global1]? Single What was the name of the intake that patient {patient_id} [time_filter_exact1] had [time_filter_global1]? Single What was the total volume of {intake_name} intake that patient {patient_id} received [time_filter_global1]? Single What was the total volume of intake that patient {patient_id} received [time_filter_global1]? Single What was the name of the output that patient {patient_id} [time_filter_exact1] had [time_filter_global1]? Single What was the total volume of {output_name} output that patient {patient_id} had [time_filter_global1]? Single What was the total volume of output that patient {patient_id} had [time_filter_global1]? Single What is the difference between the total volume of intake and output of patient {patient_id} [time_filter_global1]? Single What was the [time_filter_exact1] measured {vital_name} of patient {patient_id}[time_filter_global1]? Single What was the {agg_function} {vital_name} of patient {patient_id}[time_filter_global1]? Single What is_verb the total hospital cost of patient {patient_id} [time_filter_global1]? Single When was the [time_filter_extract1] hospital admission time of patient {patient_id} [time_filter_global1]? Single When was the [time_filter_extract1] hospital admission time that patient {patient_id} was admitted via {admission_route}[time_filter_global1]? Single When was the [time_filter_extract1] hospital discharge time of patient {patient_id}[time_filter_global1]? Single When was the [time_filter_extract1] length of ICU stay of patient {patient_id}? No current ICU patient Single When was the [time_filter_extract1] time that patient {patient_id} was diagnosed with {diagnosis_name} [time_filter_global1]? Single When was the [time_filter_extract1] procedure time of patient {patient_id} [time_filter_global1]? Single When was the [time_filter_extract1] time that patient {patient_id} received a {procedure_name} procedure [time_filter_global1]? Single When was the [time_filter_extract1] prescription time of patient {patient_id} [time_filter_global1]? Single When was the [time_filter_extract1] time that patient {patient_id} was prescribed {drug_name} [time_filter_global1]? Single When was the [time_filter_extract1] time that patient {patient_id} was prescribed {drug_name1} and {drug_name2} [time_filter_within] [time_filter_global1]? Single When was the [time_filter_extract1] time that patient {patient_id} was prescribed a medication via {drug_route} route [time_filter_global1]? Single When was the [time_filter_extract1] time that patient {patient_id} was prescribed a medication via {drug_route} route [time_filter_global1]? Single When was the [time_filter_extract1] lab test of patient {patient_id} [time_filter_global1]? Single When was the [time_filter_extract1] time that patient {patient_id} received a {lab_test} lab test [time_filter_global1]? Single When was the [time_filter_extract1] time that patient {patient_id} had the [sort] value of {lab_name} [time_filter_global1]? Single When was the [time_filter_extract1] microbiology test of patient {patient_id} [time_filter_global1]? Single When was patient {patient_id}’s [time_filter_extract1] {culture_name} microbiology test [time_filter_global1]? Single When was the [time_filter_extract1] time that patient {patient_id} had a {intake_name} intake [time_filter_global1]? Single When was the [time_filter_extract1] intake time of patient {patient_id} [time_filter_global1]? Single When was the [time_filter_extract1] time that patient {patient_id} had a {output_name} output [time_filter_global1]? Single When was the [time_filter_extract1] time that patient {patient_id} had a {vital_name} measured [time_filter_global1]? Single When was the [time_filter_extract1] time that the {vital_name} of patient {patient_id} was [comparison] than {vital_value} [time_filter_global1]? Single When was the [time_filter_extract1] time that patient {patient_id} had the [sort] {vital_name} [time_filter_global1]? Single Has_verb patient {patient_id} received a {procedure_name} procedure in other than the current hospital [time_filter_global1]? Only sample current patient Single Has_verb patient {patient_id} been admitted to the hospital [time_filter_global1]? Single Has_verb patient {patient_id} been to an emergency room [time_filter_global1]? Single Has_verb patient {patient_id} received any procedure [time_filter_global1]? Single Has_verb patient {patient_id} received a {procedure_name} procedure [time_filter_global1]? Single What was the name of the procedure that patient {patient_id} received [n_times] [time_filter_global1]? Single Has_verb patient {patient_id} received any diagnosis [time_filter_global1]? Single Has_verb patient {patient_id} been diagnosed with {diagnosis_name} [time_filter_global1]? Single Has_verb patient {patient_id} been prescribed {drug_name1},{drug_name2}, or {drug_name3} [time_filter_global1]? Single Has_verb patient {patient_id} been prescribed any medication [time_filter_global1]? Single Has_verb patient {patient_id} been prescribed {drug_name} [time_filter_global1]? Single Has_verb patient {patient_id} received any lab test [time_filter_global1]? Single Has_verb patient {patient_id} received a {lab_name} lab test [time_filter_global1]? Single Has_verb patient {patient_id} had any allergy [time_filter_global1]? Single Has_verb patient {patient_id} had any microbiology test result [time_filter_global1]? Single Has_verb patient {patient_id} had any {culture_name} microbiology test result [time_filter_global1]? Single Has_verb there been any organism found in the [time_filter_extract1]{culture_name} microbiology test of patient {patient_id} [time_filter_global1]? Single Has_verb patient {patient_id} had any {intake_name} intake [time_filter_global1]? Single Has_verb patient {patient_id} had any {output_name} output [time_filter_global1]? Single Has_verb the {vital_name} of patient {patient_id} been ever [comparison] than {vital_value} [time_filter_global1]? Single Has_verb the {vital_name} of patient {patient_id} been normal [time_filter_global1]? Single List the hospital admission time of patient {patient_id} [time_filter_global1]. Single List the [unit_average] [agg_function] {lab_name} lab value of patient {patient_id} [time_filter_global1]. Single List the [unit_average] [agg_function] weight of patient {patient_id} [time_filter_global1]. Single List the [unit_average] [agg_function] volume of {intake_name} intake that patient {patient_id} received [time_filter_global1]. Single List the [unit_average] [agg_function] volume of {output_name} output that patient {patient_id} had [time_filter_global1]. Single List the [unit_average] [agg_function] {vital_name} of patient {patient_id} [time_filter_global1]. Single Count the number of hospital visits of patient {patient_id} [time_filter_global1]. Single Count the number of ICU visits of patient {patient_id} [time_filter_global1]. Single Count the number of times that patient {patient_id} received a {procedure_name} procedure [time_filter_global1]. Single Count the number of drugs patient {patient_id} was prescribed [time_filter_global1]. Single Count the number of times that patient {patient_id} were prescribed {drug_name} [time_filter_global1]. Single Count the number of times that patient {patient_id} received a {lab_name} lab test [time_filter_global1]. Single Count the number of times that patient {patient_id} had a {intake_name} intake [time_filter_global1]. Single Count the number of times that patient {patient_id} had a {output_name} output [time_filter_global1]. Group Count the number of current patients. Group Count the number of current patients aged [age_group]. Group What is the [n_survival_period] survival rate of patients diagnosed with {diagnosis_name}? Group What is the [n_survival_period] survival rate of patients who were prescribed {drug_name} after having been diagnosed with {diagnosis_name}? Group What are the top [n_rank] diagnoses that have the highest [n_survival_period] mortality rate? Group What is_verb the [agg_fuction] total hospital cost that involves a procedure named {procedure_name} [time_filter_global1]? Group What is_verb the [agg_fuction] total hospital cost that involves a {lab_name} lab test [time_filter_global1]? Group What is_verb the [agg_fuction] total hospital cost that involves a drug named {drug_name} [time_filter_global1]? Group What is_verb the [agg_fuction] total hospital cost that involves a diagnosis named {diagnosis_name} [time_filter_global1]? Group List the IDs of patients diagnosed with {diagnosis_name} [time_filter_global1]. Group What is_verb the [agg_fuction] [unit_average]number of patient records diagnosed with {diagnosis_name} [time_filter_global1]? Group Count the number of patients who were dead after having been diagnosed with {diagnosis_name} [time_filter_within] [time_filter_global1]. Group Count the number of patients who did not come back to the hospital [time_filter_within] after diagnosed with {diagnosis_name} [time_filter_global1]. Group Count the number of patients who were admitted to the hospital [time_filter_global1]. Group Count the number of patients who were discharged from the hospital [time_filter_global1]. Group Count the number of patients who stayed in ward {ward_id} [time_filter_global1]. Group Count the number of patients who stayed in careunit{careunit} [time_filter_global1]. Group Count the number of patients who were diagnosed with{diagnosis_name} [time_filter_within] after having received a {procedure_name} procedure [time_filter_global1]. Group Count the number of patients who were diagnosed with{diagnosis_name2} [time_filter_within] after having been diagnosed with {diagnosed_name1} [time_filter_global1]. Group Count the number of patients who were diagnosed with {diagnosis_name} [time_filter_global1]. Group Count the number of patients who received a {procedure_name} procedure [time_filter_global1]. Group Count the number of patients who received a {procedure_name} procedure [n_times] [time_filter_global1]. Group Count the number of patients who received a {procedure_name2} procedure [time_filter_within] after having received a {procedure_name1} procedure [time_filter_global1]. Group Count the number of patients who received a {procedure_name} procedure [time_filter_within] after having been diagnosed with {diagnosis_name1} [time_filter_global1]. Group Count the number of {procedure_name} procedure cases [time_filter_global1]. Group Count the number of patients who were prescribed {drug_name} [time_filter_global1]. Group Count the number of {drug_name} prescription cases [time_filter_global1]. Group Count the number of patients who were prescribed {drug_name} [time_filter_within] after having received a {procedure_name} procedure [time_filter_global1]. Group Count the number of patients who were prescribed {drug_name} [time_filter_within] after having been diagnosed with {diagnosis_name} [time_filter_global1]. Group Count the number of patients who received a {lab_name} lab test [time_filter_global1]. Group Count the number of patients who received a {culture_name} microbiology test [time_filter_global1]. Group Count the number of patients who had a {intake_name} intake [time_filter_global1]. Group What are_verb the top [n_rank] frequent diagnoses [time_filter_global1]? Group What are_verb the top [n_rank] frequent diagnoses of patients aged [age_group] [time_filter_global1]? Group What are_verb the top [n_rank] frequent diagnoses that patients were diagnosed [time_filter_within] after having received a {procedure_name} procedure [time_filter_global1]? Group What are_verb the top [n_rank] frequent diagnoses that patients were diagnosed [time_filter_within] after having been diagnosed with {diagnosis_name} [time_filter_global1]? Group What are_verb the top [n_rank] frequent procedures [time_filter_global1]? Group What are_verb the top [n_rank] frequent procedures of patients aged [age_group] [time_filter_global1]? Group What are_verb the top [n_rank] frequent procedures that patients received [time_filter_within] after having received a {procedure_name} procedure [time_filter_global1]? Group What are_verb the top [n_rank] frequent procedures that patients received [time_filter_within] after having been diagnosed with {diagnosis_name} [time_filter_global1]? Group What are_verb the top [n_rank] frequently prescribed drugs [time_filter_global1]? Group What are_verb the top [n_rank] frequently prescribed drugs of patients aged [age_group] [time_filter_global1]? Group What are_verb the top [n_rank] frequent prescribed drugs for patients who were also prescribed {drug_name} [time_filter_within] [time_filter_global1]? Group What are_verb the top [n_rank] frequent drugs that patients were prescribed [time_filter_within] after having been prescribed with {drug_name} [time_filter_global1]? Group What are_verb the top [n_rank] frequent drugs that patients were prescribed [time_filter_within] after having received a {procedure_name} procedure [time_filter_global1]? Group What are_verb the top [n_rank] frequent drugs that patients were prescribed [time_filter_within] after having been diagnosed with {diagnosis_name} [time_filter_global1]? Group What are_verb the top [n_rank] frequently prescribed drugs that patients aged [age_group] were prescribed [time_filter_within] after having been diagnosed with {diagnosis_name} [time_filter_global1]? Group What are_verb the top [n_rank]frequently prescribed drugs that {gender} patients aged [age_group] were prescribed [time_filter_within] after having been diagnosed with {diagnosis_name} [time_filter_global1]? Group What are_verb the top [n_rank] frequent lab test [time_filter_global1]? Group What are_verb the top [n_rank] frequent lab tests of patients aged [age_group] [time_filter_global1]? Group What are_verb the top [n_rank] frequent lab tests that patients had [time_filter_within] after having been diagnosed with {diagnosis_name} [time_filter_global1]? Group What are_verb the top [n_rank]frequent lab tests that patients had [time_filter_within] after having received a {procedure_name} procedure [time_filter_global1]? Group What are_verb the top [n_rank] frequent specimens tested [time_filter_global1]? Group What are_verb the top [n_rank] frequent specimens that patients were tested [time_filter_within] after having been diagnosed with {diagnosis_name} [time_filter_global1]? Group What are_verb the top [n_rank] frequent specimens that patients were tested [time_filter_within] after having received a {procedure_name} procedure [time_filter_global1]? Group What are_verb the top [n_rank] frequent intake events [time_filter_global1]? Group What are_verb the top [n_rank] frequent output events [time_filter_global1]?

Several question templates assume a specific range of patients (e.g., current patients or already discharged patients), as indicated in the “Assumption” column in Table LABEL:tab:question_template_full. For example, depending on the patients, the way to calculate the duration of hospital stay can be different. The SQL query for already discharged patients (i.e., What was the [time_filter_exact1] length of hospital stay of patient patient_id?) calculates the time between the hospital admission and discharge, while the same query for the current patients (i.e., How many [unit_count] have passed since patient patient_id was admitted to the hospital currently?) calculates the time between the hospital admission and current time. We intentionally separated the templates to make the model better understand the hidden assumptions behind user utterances.

Time slots with numbering (e.g., [time_filter_global1] and [time_filter_global2] ) are the variants of the time slot without numbering (e.g., [time_filter_global]). The purpose of the numbering is to indicate the temporal order of time filters. Specifically, a higher number indicates the same or a later time for [time_filter_global1] and [time_filter_global2], and [time_filter_exact2] must be later than [time_filter_exact1].

Some verbs that end with “_verb” in question templates indicate that the verb tense can change depending on the sampled time templates.

B.2 Time Templates

Table LABEL:tab:time_template_full shows the full list of time templates with natural language (NL) time expressions and SQL time patterns. Based on the time filter types present in question templates, time templates are sampled, and their corresponding NL time expressions and SQL time patterns are added to the question templates and SQL queries. Column slots in the SQL time patterns such as [time_column] and [hospital_dischargetime] are replaced with the actual column names following the database schema. The blanks for the NL time expressions and SQL time patterns in the table indicate that no time filter is applied.

[ caption = Full list of NL time expressions and their corresponding SQL time patterns., label = tab:time_template_full, ] colspec = cccccX[3,c,m]X[6,l,m], colsep = 0.75pt, rowhead = 1, hlines, rows=font=, rowsep=1.0pt Time filter type & Expression type Unit Interval type Option NL time expression SQL time pattern global - - - - global relative hospital in first on the first hospital visit WHERE [hospital_dischargetime] IS NOT NULL ORDER BY [hospital_admittime] ASC LIMIT 1 global relative hospital in last on the last hospital visit WHERE [hospital_dischargetime] IS NOT NULL ORDER BY [hospital_admittime] DESC LIMIT 1 global relative hospital in current on the current hospital visit WHERE [hospital_dischargetime] IS NULL global relative ICU in first on the first ICU visit WHERE [icu_dischargetime] IS NOT NULL ORDER BY [icu_admittime] ASC LIMIT 1 global relative ICU in last on the last ICU visit WHERE [icu_dischargetime] IS NOT NULL ORDER BY [icu_admittime] DESC LIMIT 1 global relative ICU in current on the current ICU visit WHERE [icu_dischargetime] IS NULL global relative year in last last year WHERE datetime([time_column],'start of year') = datetime(current_time,'start of year ','-1 year') global relative year until last until last year WHERE datetime([time_column],'start of year') <= datetime(current_time,'start of year ','-1 year') global relative year since last since last year WHERE datetime([time_column],'start of year') >= datetime(current_time,'start of year ','-1 year') global relative year in this this year WHERE datetime([time_column],'start of year') = datetime(current_time,'start of year ','-0 year') global relative year until - until {year} year ago WHERE datetime([time_column]) <= datetime(current_time,'-{year} year') global relative year since - since {year} year ago WHERE datetime([time_column]) >= datetime(current_time,'-{year} year') global relative month in last last month WHERE datetime([time_column],'start of month') = datetime(current_time,'start of month','-1 month') global relative month until last until last month WHERE datetime([time_column],'start of month') <= datetime(current_time,'start of month','-1 month') global relative month since last since last month WHERE datetime([time_column],'start of month') >= datetime(current_time,'start of month','-1 month') global relative month in this this month WHERE datetime([time_column],'start of month') = datetime(current_time,'start of month','-0 month') global relative month until - until {month} month ago WHERE datetime([time_column]) <= datetime(current_time,'-{month} month') global relative month since - since {month} month ago WHERE datetime([time_column]) >= datetime(current_time,'-{month} month') global relative day in last yesterday WHERE datetime([time_column],'start of day') = datetime(current_time,'start of day','-1 day') global relative day in last until yesterday WHERE datetime([time_column],'start of day') <= datetime(current_time,'start of day','-1 day') global relative day in last since yesterday WHERE datetime([time_column],'start of day') >= datetime(current_time,'start of day','-1 day') global relative day in this today WHERE datetime([time_column],'start of day') = datetime(current_time,'start of day','-0 day') global relative day until - until {day} day ago WHERE datetime([time_column]) <= datetime(current_time,'-{day} day') global relative day since - since {day} day ago WHERE datetime([time_column]) >= datetime(current_time,'-{day} day') global absolute year in - in {year} WHERE strftime('%Y',[time_column])= '{year}' global absolute year until - until {year} WHERE strftime('%Y',[time_column]) <= '{year}' global absolute year since - since {year} WHERE strftime('%Y',[time_column]) >= '{year}' global absolute month in - in {month}/{year} WHERE strftime('%Y-%m',[time_column])= '{year}-{month}' global absolute month until - until {month}/{year} WHERE strftime('%Y-%m',[time_column]) <= '{year}-{month}' global absolute month since - since {month}/{year} WHERE strftime('%Y-%m',[time_column]) >= '{year}-{month}' global absolute day in - on {month}/{day}/{year} WHERE strftime('%Y-%m-%d',[time_column]) = '{year}-{month}-{day}' global absolute day until - until {month}/{day}/{year} WHERE strftime('%Y-%m-%d',[time_column]) <= '{year}-{month}-{day}' global absolute day since - since {month}/{day}/{year} WHERE strftime('%Y-%m-%d',[time_column]) >= '{year}-{month}{day}' global mix month in last in {month}/last year WHERE datetime([time_column],'start of year') = datetime(current_time,'start of year','-1 year') AND strftime('%m',[time_column]) = '{month}' global mix month in this in {month}/this year WHERE datetime([time_column],'start of year') = datetime(current_time,'start of year','-0 year') AND strftime('%m',[time_column]) = '{month}' global mix day in last on {month}/{day}/last year WHERE datetime([time_column],'start of year') = datetime(current_time,'start of year','-1 year') AND strftime('%m-%d',[time_column]) = '{month}-{day}' global mix day in this on {month}/{day}/this year WHERE datetime([time_column],'start of year') = datetime(current_time,'start of year','-0 year') AND strftime('%m-%d',[time_column]) = '{month}-{day}' global mix day in last on last month/{day} WHERE datetime([time_column],'start of month') = datetime(current_time,'start of month','-1 month') AND strftime('%d',[time_column]) = '{day}' global mix day in this on this month/{day} WHERE datetime([time_column],'start of month') = datetime(current_time,'start of month','-0 month') AND strftime('%d',[time_column]) = '{day}' within - - - - within - hospital in - within the same hospital visit WHERE [hospital_admission_id1] = [hospital_admission_id2] within - ICU in - within the same icu visit WHERE [icu_admission_id1] = [icu_admission_id2] within - year in - within the same year WHERE datetime([time_column1],'start of year') = datetime([time_column2],'start of year' within - n_year in - within {year} year WHERE datetime([time_column2]) BETWEEN datetime([time_column1] AND datetime([time_column1],'+{year} year' within - month in - within the same month WHERE datetime([time_column1],'start of month') = datetime([time_column2],'start of month' within - n_month in - within {month} month WHERE datetime([time_column2]) BETWEEN datetime([time_column1] AND datetime([time_column1],'+{month} month' within - day in - within the same day WHERE datetime([time_column1],'start of day') = datetime([time_column2],'start of day' within - n_day in - within {day} day WHERE datetime([time_column2]) BETWEEN datetime([time_column1] AND datetime([time_column1],'+{day} day' within - exact in - at the same time WHERE datetime([time_column1]) = datetime([time_column2]) exact relative exact at - first ORDER BY [time_column] ASC LIMIT 1 exact relative exact at - second ORDER BY [time_column] ASC LIMIT 1 OFFSET 1 exact relative exact at - second to last ORDER BY [time_column] DESC LIMIT 1 OFFSET 1 exact relative exact at - last ORDER BY [time_column1] DESC LIMIT 1 exact absolute exact at - at {year}-{month}-{day} {hour} :{minute}:{second} WHERE datetime([time_column]) = '{year}-{month}-{day} {hour}:{minute} :{second}'

Time templates with the option “this” are not combined with the interval type of “since” or “until” because combining “this” with “since” is equivalent to combining “this” with “in” (i.e., since this year is equivalent to this year). Additionally, combining “this” with “until” is equivalent to no time constraint (i.e., until this year is equivalent to no time filter).

In relative time expressions, the concept of N units before the current time is ambiguous because one year ago and the last year can differ depending on the context. Therefore, we strictly define “N units ago” as a time point exactly N units before the current time (e.g., one year ago of the current time 2105-12-31 23:59:00 is 2104-12-31 23:59:00) and they can be combined with the “until” and “since” interval types. Figure 5 illustrates several time templates.

B.3 Template Combination

As question templates can be combined with multiple slots, we tagged each question to track what time templates and pre-defined values are combined to form the final question. Each tag (Q_tag, O_tag, and T_tag) represents Stages 0, 1, and 2 in Figure 3. Except for Q_tag, which indicates the question template, O_tag and T_tag have fixed numbers of placeholders that store each sampled template and value. O_tag stores nine different types of operation values in a tuple: ([age_group], [agg_function], [comparison], [n_rank], [n_survival_period], [n_times], [sort], [unit_average], [unit_count]), sampled in Stage 1. T_tag stores time templates in a tuple: ([time_filter_global1], [time_filter_global2], [time_filter_within], [time_filter_exact1], [time_filter_exact2]), sampled in Stage 2.

A full list of the operation values is reported in Table 7. Similar to time templates, the operation values have both NL expressions and SQL patterns.

Appendix C SQL Labeling Details

Since the questions were collected independently of the database schema, our SQL labeling process required us to make numerous assumptions. Below is a list of the assumptions we made to label the queries.

The age of a patient is calculated only once at each hospital admission time. Therefore, even if a patient stays more than a year without hospital discharge, the age remains the same.

To count the number of patients or hospital (or ICU) visits, DISTINCT is used in the SELECT clause.

The queries about the cost of or drug routes use DISTINCT.

When retrieving a lab value or vital sign, only the value is returned, not the unit of measurement.

DENSE_RANK is used for ranking questions, meaning multiple items with the same ranks can be returned together. For example, a query asking about the top three frequent diagnoses may retrieve more than three diagnosis names when items with the same rank exist in the answer. Additionally, the retrieved results can be fewer than the expected number N when the number of diagnoses under some conditions is smaller than the expected number.

When a question is related to both death and diagnosis, only the first diagnosis time is considered.

When calculating the N-year survival rate, if a death record exists between the first diagnosis time and N years later, it counts as death. But if there is no death record within N years or the death happens after N years, it counts as survived.

The current time and normal ranges of vital signs are post-processed after SQL generation so that they are independent of the modeling pipeline when the value changes.

The vital signs we consider in the dataset are body temperature, SaO2, heart rate, respiratory rate, and blood pressures (systemic systolic, diastolic, and mean).

Diagnosis and procedure times are not available in the original database. To address this, we manually set diagnosis time as hospital admission time and procedure time as hospital discharge time. Thus, questions asking about current patients’ procedure time are excluded in the MIMIC-III questions.

Among many items in the CHARTEVENTS table, we only use weight, height, and the seven vital sign values.

We use INPUTEVENTS_CV instead of INPUTEVENTS_MV for input events as it contains more records and one time column per record (charttime), which is more closely aligned with eICU’s intakeoutput table.

For input and output events, we only use values stored in milliliters (mL) in the case of numerical reasoning within or between input and output events.

C.1.2 Assumptions in eICU

Diagnosis and treatment tables have path-based names for each record. Instead of using them directly, the paths are further pre-processed to have shorter names.

For vital signs, we choose the vitalperiodic table as it contains more records.

Questions about drug dose are not considered in eICU as drug doses are stored in free-text (values and units are mixed).

C.2 Mapping Between Condition Value Slots and Column Names

The mapping between condition value slots and column names in both MIMIC-III and eICU is shown in Table 8.

C.3 Comparison Between JOIN-based and Nesting-based Queries

Unlike most other semantic parsing datasets, the SQL queries in EHRSQL are labeled in a nested manner. Table 6 shows a comparison between queries with the naive use of JOIN and nesting. In terms of query length, JOIN-based queries are much shorter, but it is hard to follow the semantics in the queries. However, even if the length of the queries is long, T5 models are able to fully generate a long sequence of SQL (see Table 10 for qualitative results). As for execution speed, the naive use of JOIN takes almost four times slower than a nesting-based query in some cases (0.04 vs. 0.15 secs), as shown in Figure 6.

Appendix D Database Pre-processing Details

Patients aged 11 to 89 are included in the dataset.

1,000 patients are sampled to cover a greater number of medical events (MIMICSQL and emrKBQA use 100 patients).

When the same type of value has multiple units of measurement, only the value with the most common unit is retained and other values are removed from the database.

D.2 Time-shifting Process

To include questions with relative time expressions, we manually time-shift each patient’s hospital records. Specifically, we sample a random time point (between 2100 and 2105) to set the time of the first hospital visit. Then, we time-shift the whole patient records to the sampled time point while keeping all the record intervals the same. Additionally, we constrain the number of current patients to 10% of the total number of patients in patient sampling since any questions can be asked with relative time expressions, such as yesterday and last month.

D.3 De-identification Process

Even though MIMIC-III and eICU are de-identified databases, they still contain patient-specific information. In our case, when one or more condition values are sampled along with a patient ID, there is a risk that the question and its paired SQL query might reveal patient-specific information. To avoid such risk, we randomly shuffle records of diagnosis, procedure, lab tests, prescription, chart events, input events, output events, microbiology tests, care units, and ward IDs across patients in the database, while keeping the time of the records the same.

Appendix E Template Paraphrasing Pipeline

The overall template paraphrasing pipeline is illustrated in Figure 7.

Appendix F Data Splitting

We constrain the training, validation, and test sets to contain all question templates, but the template paraphrases do not overlap between the splits.

Training set: We provide more text-to-SQL pairs to question templates with a greater number of slots. Specifically, question templates with fewer than three slots are assigned thirty to sixty pairs. Questions with three or more slots and fewer than five receive forty to eighty pairs; questions with five or more slots receive fifty to one hundred pairs per question template.

Validation set: The number of pairs per question template is sampled between four and five.

Test set: Data sampling rules are identical to those of the validation set, except that we assign higher weights on the question templates that are considered important, which are labeled in “high,” “medium,” “low,” and “n/a” (not available) by a physician. With a score of 3 for “high,” 2 for “medium,” 1 for the rest, the final number of pairs in the test set is multiplied by the importance score for each question template.

Unanswerable question: They are assigned to the validation and test sets so that each split makes up 33%.

Appendix G Training Details

We use pre-trained T5-base models from Hugging Facehttps://huggingface.co/ for both the no schema and schema versions. Without a schema, the model is trained to translate from natural questions to SQL queries. With a schema, we additionally append schema information to the questions. The training and evaluation configurations are in Table 9. All models are trained on NVIDIA GeForce RTX 3090s.

Appendix H Qualitative Results

Table 10 shows samples of generated SQL queries by the number of slots. A T5-base model trained on MIMIC-III can generate a very long sequence of SQL queries if they are seen in the training data. In most cases, the errors do not come from generating complex, long nested queries, but from failures in schema linking (identifying references of columns, tables, and condition values in natural utterances).

H.2 SQL Generation with Different Time Expressions

Table 11 shows generated SQL samples with different time templates. The column Seen vs. Unseen indicates whether the exact question template (Q_tag) and time template (T_tag) combination is seen during training. Interestingly, the model can correctly generate SQL queries for both seen and unseen combinations.

H.3 Falsely Executed and Refused Results

Table 12 and 13 show samples of falsely executed and refused questions, respectively. Falsely executed results are retrieved results of the model even though the input question is unanswerable (see Table 12). These errors are fatal mistakes that a healthcare QA system must avoid, as retrieving incorrect information may lead to wrong clinical decisions.

Table 13 shows refused results of the model. In some cases, the model might be able to generate the SQL query, but chooses not to execute it due to low confidence.

H.4 Entropy Distribution of the Model Outcome

Figure 8 shows the maximum entropy values generated from T5. As the ground-truth labels (ANS: answerable; UnANS: unanswerable) indicate, the distributions of entropy values between answerable and unanswerable questions are significantly different (< 0.001 with the Mann-Whitney U test). Similar patterns are observed in models trained on eICU.

Appendix I Author statement

The authors of this paper bear all responsibility in case of violation of rights, etc. associated with the EHRSQL dataset.