A Comprehensive Exploration on WikiSQL with Table-Aware Word Contextualization

Wonseok Hwang, Jinyeong Yim, Seunghyun Park, Minjoon Seo

Introduction

NL2SQL is a popular form of semantic parsing tasks that asks for translating a natural language (NL) utterance to a machine-executable SQL query. As one of the first large-scale (80k) human-verified semantic parsing datasets, WikiSQL (Zhong et al., 2017) has attracted much attention in the community and enabled a significant progress through task-specific end-to-end neural models (Xu et al., 2017). On the other side of the NLP community, we have also observed a rapid advancement in contextualized word representations (Peters et al., 2018; Devlin et al., 2018), which have proved to be extremely effective for most language tasks that deal with unstructured text data. However, it has not been clear yet whether the word contextualization is also similarly effective when structured data such as tables in WikiSQL are involved.

In this paper, we discuss our approach on WikiSQL that coherently brings previous NL2SQL liteature and large pretrained models together. Our model, SQLova, consists of two layers, encoding layer that obtains table-aware word contextualization and NL2SQL layer that generates the SQL query from the contextualized representations. We show that SQLova outperforms the previous best model achieving 83.6% logical form accuracy and 89.6% execution accuracy on WikiSQL test set, outperforming the previous best model by 8.2% and 2.5%, respectively. It is important to note that, while BERT plays a significant role, merely attaching a seq2seq model on the top of BERT leads to a poor performance, indicating the importance of properly and carefully utilizing BERT when dealing with structured data.

We furthermore argue that these scores are near the upper bound in WikiSQL, where we observe that most of the evaluation errors are caused by either wrong annotations by humans or the lack of given information. In fact, according to our crowdsourced statistics on an approximately 10% sampled set of WikiSQL dataset, our model’s score exceeds human performance at least by 1.3% in execution accuracy.

We propose a carefully designed architecture that brings the best of previous NL2SQL approaches and large pretrained language models together. Our model clearly outperforms the previous best model and the human performance in WikiSQL.

We provide a diverse and detailed analysis on the dataset and our model. These examinations will further help future research on NL2SQL data creation and model development.

The rest of the paper is organized as follows. We first describe our model in Section 3. Then we report the quantitative results of our model in comparison to previous baselines in Section 4. Lastly, we discuss qualitative analysis on both the dataset and our model in Section 5. The source code and human evaluation data is available from https://github.com/naver/sqlova.

Related Work

WikiSQL is a large semantic parsing dataset consisting of 80,654 natural language utterances and corresponding SQL annotations on 24,241 tables extracted from Wikipedia (Zhong et al., 2017). The task is to build the model that generates SQL query for given natural language question on single table and table headers without using contents of the table. Some examples, using the table from WikiSQL, are shown in Figure 1.

The large size of the dataset has enabled adopting deep neural techniques for the task and drew much attention in the community recently. Although early studies on neural semantic parsers have started without syntax specific constraints on output space (Dong and Lapata, 2016; Jia and Liang, 2016; Iyer et al., 2017), many state-of-the-art results on WikiSQL have achieved by constraining the output space with the SQL syntax. The initial model proposed by (Zhong et al., 2017) independently generates the two components of the target SQL query, select-clause and where-clause, which outperforms the vanilla sequence-to-sequence baseline model proposed by the same authors. SQLNet (Xu et al., 2017) further simplifies the generation task by introducing a sequence-to-set model in which only where condition value is generated by the sequence-to-sequence model. TypeSQL (Yu et al., 2018) also employs a sequence-to-set structure but with an additional “type" information of natural language tokens.

Coarse2Fine (Dong and Lapata, 2018) first generates rough intermediate output, and then refines the results by decoding full where-clauses. Also, the table-aware contextual representation of the question is generated with bi-LSTM with attention mechanism which increases logical form accuracy by 3.1%. Our approach differs in that many layers of self-attentions (Vaswani et al., 2017; Devlin et al., 2018) are employed with a single concatenated input of question and table headers for stronger contextualization of the question.

Pointer-SQL (Wang et al., 2017) proposes a sequence-to-sequence model that uses an attention-based copying mechanism and a value-based loss function. Annotated Seq2seq (Wang et al., 2018b) utilizes a sequence-to-sequence model after automatic annotation of input natural language. MQAN (McCann et al., 2018) suggests a multitask question answering network that jointly learns multiple natural language processing tasks using various attention mechanisms. Execution guided decoding is suggested in (Wang et al., 2018a), in which non-executable (partial) SQL queries candidates are removed from output candidates during decoding step. IncSQL (Shi et al., 2018) proposes a sequence-to-action parsing approach that uses incremental slot filling mechanism with feasible actions from a pre-defined inventory.

Model

Our model, SQLova, consists of two layers: encoding layer that obtains table- and context-aware question word representations (Section 3.1), and NL2SQL layer that generates the SQL query from the encoded representations (Section 3.2).

We extend BERT (Devlin et al., 2018) for encoding the natural language query together with the headers of the entire table. We use [SEP], a special token in BERT, to separate between the query and the headers. That is, each query input $T_{n,1}\dots T_{n,L}$ ( $L$ is the number of query words) is encoded as [CLS], $T_{n,1}$ , $\cdots$ $T_{n,L}$ , [SEP], $T_{h_{1},1}$ , $T_{h_{1},2}$ , $\cdots$ , [SEP], $\cdots$ , [SEP], $T_{h_{N_{h}},1}$ , $\cdots$ , $T_{h_{N_{h}},M_{N_{h}}}$ ,[SEP]

where $T_{h_{j},k}$ is the $k$ -th token of the $j$ -th table header, $M_{j}$ is the total number of tokens of the $j$ -th table headers, and $N_{h}$ is the total number of table headers. Another input to BERT is the segment id, which is either 0 or 1. We use 0 for the question tokens and 1 for the header tokens. Other configurations largely follow (Devlin et al., 2018). The output from the final two layers of BERT are concatenated and used in NL2SQL Layer (Section 3.2).

2 NL2SQL Layer

In this section, we describe the details of NL2SQL Layer (Figure 3) on top of the table-aware encoding layer.

In a typical sequence generation model, the output is not explicitly constrained by any syntax, which is highly suboptimal for formal language generation. Hence, following (Xu et al., 2017), NL2SQL Layer uses syntax-guided sketch, where the generation model consists of six modules, namely select-column, select-aggregation, where-number, where-column, where-operator, and where-value (Figure 3). Also, following (Xu et al., 2017), column-attention is frequently used to contextualize question.

In all sub-modules, the output of table-aware encoding layer (Section 3.1) is further contextualized by two layers of bidirectional LSTM layers with 100 dimension. We denote the LSTM output of the $n$ -th token of the question with $E_{n}$ . Header tokens are encoded separately and the output of final token of each header from LSTM layer is used. $D_{c}$ is used to denotes the encoding of header $c$ . The role of each sub-module is described below.

finds select column from given natural language utterance by contextualizing question through column-attention mechanism (Xu et al., 2017).

Here, $\mathcal{W}$ stands for affine transformation, $C_{n}$ is context vector of question for given column, $[\cdot;\cdot]$ denotes the concatenation of two vectors, and $p_{sc}(c)$ indicates the probability of generating column $c$ . To make the equation uncluttered, same $\mathcal{W}$ is used to denote any affine transformation in our paper although all of them denote different transformations.

finds aggregation operator agg for given column $c$ among six possible choices (NONE, MAX, MIN, COUNT, SUM, and AVG). Its probability is obtained by

where $C_{c}$ is the context vector of the question obtained by the same way in select-column.

finds the number of where condition by contextualizing column ( $C$ ) via self-attention and contextualizing question ( $C_{Q}$ ) conditioned on $C$ .

Here, $h$ and $c$ are initial “hidden" and “cell" inputs to LSTM encoder. The probability of observing $k$ number of where condition is found from $k$ -th element of vector $\text{softmax}(s_{wn})$ . This submodule is same with that of SQLNet (Xu et al., 2017) and shown here for comprehensive reading.

obtains the probability of observing column $c$ ( $p_{wc}(c)$ ) through column-attention,

where $C_{c}$ is the context vector of the question obtained by the same way as in select-column. The probability of generating each column is obtained separately from sigmoid function and top $k$ columns are selected. $k$ is found from where-number sub-module.

finds where operator op ( $\in\{=,>,<\}$ ) for given column $c$ through column-attention.

where $C_{c}$ is the context vector of the question obtained by the same way in select-column.

finds where condition by locating start- and end-tokens from question for given column $c$ and operator op.

Here, $V\texttt{op}$ stands for one-hot vector of op ( $\in\{=,>,<\}$ ). The probability of $n$ -th token of question being start-index for given $c$ -th column and op is obtained by feeding 1st element of $s_{wv}$ vector to softmax function whereas that of end-index is obtained by using 2nd element of $s_{wv}$ by the same way.

To sum up, our NL2SQL Layer is motivated by SQLNet (Xu et al., 2017) but have following key differences. Unlike SQLNet, NL2SQL Layer does not share parameters. Also, instead of using pointer network for inferring the where condition values, we train for inferring the start and the end positions of the utterance, following (Dong and Lapata, 2018). Furthermore, the inference of the start and the end tokens in where-value module depends on both selected where-column and where-operators while the inference relies on where-columns only in (Xu et al., 2017). Lastly, when combining two vectors corresponding to the question and the headers, concatenation instead of addition is used.

During the decoding (SQL query generation) stage, non-executable (partial) SQL queries can be excluded from the output candidates for more accurate results, following the strategy suggested by (Wang et al., 2018a; Yin and Neubig, 2018). In select clause, (select column, aggregation operator) pairs are excluded when the string-type columns are paired with numerical aggregation operators such as MAX, MIN, SUM, or AVG. The pair with highest joint probability is selected from remaining pairs. In where clause decoding, the executability of each (where column, operator, value) pair is tested by checking the answer returned by the partial SQL query select agg(cols) where colw op val. Here, cols is the predicted select column, agg is the predicted aggregation operator, colw is one of the where column candidates, op is where operator, and val stands for the where condition value. The queries with empty returns are also excluded from the candidates. The final output of where clause is determined by selecting the output maximizing the joint probability estimated from the output of where-number, where-column, where-operator, and where-value modules.

Experiments

During training, BERT-based table-aware encoding layer (BERT-Large-Uncasedhttps://github.com/google-research/bert) are loaded and fine-tuned with ADAM optimizer with the learning rate of $10^{-5}$ , whereas NL2SQL Layer is trained with the learning rate of $10^{-3}$ . In both cases, the decay rates of ADAM optimizer are set to $\beta_{1}=0.9,\beta_{2}=0.999$ . Batch size is set to 32. To find word vectors, natural language utterance is first tokenized by using Standford CoreNLP (Manning et al., 2014). Each token is further tokenized (into sub-word level) by WordPiece tokenizer (Devlin et al., 2018; Wu et al., 2016). The headers of the tables and SQL vocabulary are tokenized by WordPiece tokenizer directly. The PyTorch version of BERT codehttps://github.com/huggingface/pytorch-pretrained-BERT is used for word embedding and some part of the code in NL2SQL Layer is influenced by the original SQLNet source codehttps://github.com/xiaojunxu/SQLNet. All experiments were performed on WikiSQL ver. 1.1 https://github.com/salesforce/WikiSQL.

The logical form (LF) and the execution accuracy (X) on dev set (consisting of 8,421 queries) and test set (consisting of 15,878 queries) of WikiSQL of several models are shown in Table 1. The execution accuracy is measured by evaluating the answer returned by ‘executing’ the query on the SQL database. The order of where conditions is ignored in measuring logical form accuracy in our models. The top rows in Table 1 show models without execution guidance (EG), and the bottom rows show models augmented with EG. SQLova outperforms previous baselines by a large margin, achieving [+5.3% LF] and [+2.5% X] for non-EG and achieving [+8.2% LF] and [+2.5% X] for EG.

To understand the performance of SQLova in detail, the logical form accuracy of each sub-module was obtained and shown in Table 2. All sub-modules show $\small\mathrel{\mathchoice{\raise 0.0pt\hbox{\scalebox{1.0}{\raise 0.0pt\hbox{$ \displaystyle\gtrsim $}}}}{\raise 0.0pt\hbox{\scalebox{1.0}{\raise 0.0pt\hbox{$ \textstyle\gtrsim $}}}}{\raise 0.0pt\hbox{\scalebox{1.0}{\raise 0.0pt\hbox{$ \scriptstyle\gtrsim $}}}}{\raise 0.0pt\hbox{\scalebox{1.0}{\raise 0.0pt\hbox{$ \scriptscriptstyle\gtrsim $}}}}}$ 95% in accuracy except select-aggregation module whose low accuracy partially results from the error in the ground-truth of WikiSQL (Section-5).In addition to SQLova we also provide two BERT-based models which also outperforms previous baselines by large margin, in Appendix A.1.

Choosing not to answer for low-confidence predictions is another important measure of performance. We use the output probability of a generated SQL query from SQLova as the confidence score and predict that the question is unanswerable when the score is low. The result shows that SQLova effectively assigns a low probability to wrong predictions, yielding a high precision of 95%+ with a recall rate of 80%. The precision-recall curve and its area under curve are shown in Figure A2.

2 Ablation Study

To understand the importance of each part of SQLova, we evaluate ablations in Table 3. The results show that word contextualization (without fine-tuning) contributes to overall logical form accuracy by 4.1% (dev) and 3.9% (test) (compare third and fifth rows of the table) which is similar to the observation by (Dong and Lapata, 2018) where the 3.1% increases observed with table-aware LSTM encoder. Consistently, replacing BERT by ELMo (Peters et al., 2018) shows similar results (fourth row of the table). But unlike GloVe, where fine-tuning increases only a few percents in accuracy (Xu et al., 2017), fine-tuning of BERT increases the accuracies by 11.7% (dev) and 12.2% (test) (compare first and third rows in the table) which may be attributed to the use of many layers of self-attention (Vaswani et al., 2017). Use of BERT-Base decreases the accuracy by 1.3% on both dev and test set compared to BERT-Large cases. We also developed BERT-to-Sequence where the encoder part of vanilla sequence-to-sequence model with attention (Jia and Liang, 2016) is replaced by BERT. The model achieves 57.3% and 56.4% logical form accuracies in dev and test sets respectively (Table 1) highlighting the importance of using proper decoding layers. To further validate the conclusion, we replaced LSTM decoder in BERT-to-Sequence into Transformer (BERT-to-Transformer), the model achieved 70.5 logical form accuracy in dev set again achieving 11.1% lower score compared to SQLova. The detailed description of the model is presented in Appendix A.1.3.

Analysis

There are 1,533 mismatches in logical form between the ground-truth (GT) and the predictions from SQLova in WikiSQL dev set. Among the mismatches, 100 samples were randomly selected, analyzed, and classified into two categories: (1) 26 “unanswerable” cases of which it is not possible to generate correct SQL query for given information (question and table schema), and (2) 74 “answerable” cases.

Unanswerable cases were further categorized into the following four types.

Type I: the headers of tables do not contain the necessary information. For example, a question “What was the score between Marseille and Manchester United on the second leg of the Champions League Round of 16?” and its corresponding table headers $\{$ ‘Team’, ‘Contest and round’, ‘Opponent’, ‘1st leg score’, ‘2nd leg score’, ‘Aggregate score’ $\}$ in QID-1986 (Table LABEL:tab:100example) do not contain information about which header should be selected for condition values ‘Manchester United’ and ‘Marseille’.

Type II: There exist multiple valid SQL queries per question (QID-783, 2175, 4229 in Table LABEL:tab:100example). For example, the GT SQL query of QID-783 has count aggregation operator and any header can be used for select column.

Type III: the generation of nested SQL query is required. For example, correct SQL query for QID-332 (Table LABEL:tab:100example) is “SELECT count(incumbent) WHERE District=(SELECT District WHERE Incumbent=Alvin Bush)”.

Type IV: questions are ambiguous. For example, the answer to the question “What is the number of the player who went to Southern University?” in QID-156 (Table LABEL:tab:100example) can vary depending on the interpretation of “the number of the player”.

The categorization of 26 samples is summarized in Table 6 in Appendix A.4..

Further analysis over the remaining 74 answerable examples reveals that there are 49 GT errors in logical forms. 45 out of 49 examples contain GT errors in aggregation operators (e.g. QID-7062), two have GT errors in select columns (e.g. QID-841, 5611), and remaining two contain GT errors in where clause (e.g. QID-2925, 7725). Interestingly, among 49 examples, 41 logical forms are correctly predicted by SQLova, indicating that the actual performances of the models in Table 1 are underestimated. This also may imply that most of examples in WikiSQL have correct GT for training. The results are summarized in Table 5, and all 100 examples are presented in Table LABEL:tab:100example in Appendix A.5.

As the questions in WikiSQL are created by paraphrasing queries generated automatically from the templates without considering the table contents, the meanings of the questions could change, especially when the quantitative answer is required, possibly leading to GT errors. For example, QID-3370 in Table LABEL:tab:100example is related to an “year” and the GT SQL query includes unnecessary COUNT aggregation operators.

Overall, the error analysis above may imply that near- $90\%$ accuracy of SQLova could be near the upper bound in WikiSQL task the “answerable” and non-erroneous questions when the contents of tables are not available.

2 Measuring Human Performance

The human performance on WikiSQL dataset has not been measured so far despite its popularity. Here, we provide the approximate human performance by collecting answers from 246 different crowdworkers through Amazon Mechanical Turk over 1,551 randomly sampled examples from the WikiSQL test set (which has 15,878 examples in total). The crowdworkers were selected with following three constraints: (1) 95% or higher task acceptance rate; (2) 1000 or higher HITs; (3) residents of the United States.

During the evaluation, crowdworkers were asked either to find value(s) or to compute a value using the given questions and corresponding tables following the instruction provided (Figure 4). Note that the task requires general capability of understandings English text and finding values from a table without a need for the generation of SQL queries. This effectively mimics the measurement of execution accuracy in WikiSQL. We find that the accuracy of crowdworkers on the randomly sampled test data is 88.3%, as shown in Table 1 while the execution accuracy of SQLova over 1,551 samples are 86.8% (w/o EG) and 91.0% (w/ EG). When measuring human performance, errors in ground truth are manually corrected by experts (us).

We manually checked and analyzed all answers from the crowd. Errors made by crowdworkers are similar to that of the model such as a mismatch of select columns or where columns. One notable mistake by only humans (that our model does not make) is confusion on the ambiguity of natural language. For example, when a question is asking a column value with more than two conditions, crowdworkers show the tendency to consider a single condition only because multiple conditions were written with “and" which is often considered as the meaning of “or" in real life.

Conclusion

In this paper, we propose the first NL2SQL model to achieve a super-human accuracy in WikiSQL. We demonstrate the effectiveness of a careful architecture design that brings and combines previous approaches in NL2SQL and table-aware word contextualization with large pretrained language model (BERT) together. We propose a BERT-based table-aware encoder and a task-specific module on the top of the encoder, outperforming the previous best model by 8.2% and 2.5% in logical form and execution accuracy, respectively. We hope our detailed explanation and analysis of the model and the dataset provide an insight on how future research on NL2SQL models and datasets can be effectively approached.

We thank Clova AI members for their great support, especially Jung-Woo Ha for proof-reading the manuscript, Sungdong Kim and Dongjun Lee for providing help on using BERT, Guwan Kim for insightful comments. We also thank the Hugging Face Team for sharing the PyTorch implementation of BERT.

References

Appendix A Appendix

Here, we present another task specific layer Shallow-Layer having lower model complexity compared to NL2SQL Layer. Shallow-Layer does not contain trainable parameters but controls the flow of information during fine-tuning of BERT via loss function. Like NL2SQL Layer, Shallow-Layer uses syntax-guided sketch, where the generation model consists of six modules, namely select-column, select-aggregation, where-number, where-column, where-operator, and where-value (Figure A1A).

select-column module finds the column in select clause from given natural language utterance by modeling the probability of choosing $i$ -th header ( $p_{sc}(\texttt{col}_{i})$ ) as

where $H_{h,i}$ is the contextualized output vector of first token of $i$ -th header by table-aware BERT encoder, and $(H_{h,i})_{0}$ indicates zeroth element of the vector $H_{h,I}$ . In general, $(V)_{\mu}$ denotes $\mu$ -th element of vector $V$ in this paper. Also, the conditional probability for given question and table-schema $p(\cdot|\text{Q},\text{table-schema})$ is simply denoted as $p(\cdot)$ to make equation uncluttered.

select-aggregation module finds the aggregation operator for the given select column. The probability of generating aggregation operator agg for given select column $\texttt{col}_{i}$ is described by

where $\texttt{agg}_{1}$ , $\texttt{agg}_{2}$ , $\texttt{agg}_{3}$ , $\texttt{agg}_{4}$ , $\texttt{agg}_{5}$ , and $\texttt{agg}_{6}$ are none, max, min, count, sum, and avg respectively.

where-number module predicts the number of where conditions by modeling the probability of generating $\mu$ -number of conditions as

where $H_{\texttt{[CLS]}}$ is the output vector of [CLS] token from table-aware BERT encoder, and $\mathcal{W}$ stands for affine transformation. Throughout the paper, any affine transformation shall be denoted by $\mathcal{W}$ for the clarity.

where-column module calculates the probability of generating each columns in where clause. The probability of generating $\texttt{col}_{i}$ is given by

where-operator module finds most probable operators for given where column among three possible choices ( $>,=,<$ ). The probability of generating operator $\texttt{op}_{\mu}$ for given where column $\texttt{col}_{i}$ is modeled as

where $\texttt{op}_{8}$ , $\texttt{op}_{9}$ , and $\texttt{op}_{10}$ are >, =, and < respectively.

where-value module finds which tokens of a question correspond to condition values for given where columns by locating start- and end-tokens. The probability that $k$ -th question token is selected as a start token for given where column $\texttt{col}_{\mu}$ is modeled as

Similarly the probability of $k$ -th question token is selected as an end token is

100 is selected to avoid overlap during inference between start- and end-token models. The maximum number of table headers in single table is 44 in WikiSQL task.

A.1.2 Decoder-Layer

Decoder-Layer contains LSTM decoders adopted from pointer network (Vinyals et al., 2015; Zhong et al., 2017) (Fig. 3B) with following special features. Instead of generating entire header tokens, we only generate first token of each header and interpret them as entire header tokens during inference stage using Point-to-SQL module (Fig. 3B). Similarly, the model generates only the pointers to start- and end- where-value tokens omitting intermediate points. Decoding process can be expressed as following equations which use the attention mechanism.

$P_{t-1}$ stands for the one-hot vector (pointer) at time $t-1$ , $h_{t-1}$ and $c_{t-1}$ are hidden- and cell- vectors of LSTM decoder, $d$ is the hidden dimension of BERT, $H_{i}$ is the BERT output of $i$ -th token, and $p_{t}(i)$ is the probability observing $i$ -th token at time $t$ .

A.1.3 BERT-to-Sequence

BERT-to-Sequence consists of the table-aware BERT encoder and LSTM decoder which is essentially a sequence-to-sequence model (with attention) (Jia and Liang, 2016) except that the LSTM encoder part is replaced by BERT. The encoding process is same with NL2SQL Layer. The decoding process is described by following equations.

where $w_{t}$ stands for word token (among 30,522 token vocabulary used in BERT) predicted at time $t$ , $\text{emb}_{\text{BERT}}$ is a map that transform the token to embedding vector $E_{t}$ via word embedding module of BERT, $H_{i}$ is the output vector from BERT encoder of $i$ -th input token, and $p(w_{t+1})$ is the probability of generating token $w_{t+1}$ at time $t+1$ .

A.1.4 The performance of Shallow-Layer and Decoder-Layer

Compared to previous best results, Shallow-Layer shows +5.5% LF and +3.1% X, Decoder-Layer shows +4.4% LF and +1.8% X for non-EG case (Table. 4). For EG case, Shallow-Layer shows +6.4% LF and +0.4% X, Decoder-Layer shows +7.8% LF and +2.5% X (Table. 4).