RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers

Introduction

The ability to effectively query databases with natural language (NL) unlocks the power of large datasetsto the vast majority of users who are not proficient in query languages.As such, a large body of research has focused on the task of translating NL questions into SQL queriesthat existing database software can execute.

The development of large annotated datasets of questions and the corresponding SQL queries has catalyzedprogress in the field.In contrast to prior semantic parsing datasets (Finegan-Dollak et al., 2018), new tasks such asWikiSQL (Zhong et al., 2017) and Spider (Yu et al., 2018b) pose the real-life challenge ofgeneralization to unseen database schemas.Every query is conditioned on a multi-table database schema, and the databases do not overlap between the train and testsets.Schema generalization is challenging for three interconnected reasons.First, any text-to-SQL parsing model must encode the schema intorepresentations suitable for decoding a SQL query that might involve the given columns or tables.Second, these representations should encode all the information about the schema such as its column types, foreignkey relations, and primary keys used for database joins.Finally, the model must recognize NL used to refer to columns and tables, which might differ fromthe referential language seen in training.The latter challenge is known as schema linking – aligning entity references in the question to theintended schema columns or tables.While the question of schema encoding has been studied in recent literature (Bogin et al., 2019a), schema linkinghas been relatively less explored.Consider the example in Figure 1.It illustrates the challenge of ambiguity in linking:while “model” in the question refers to car_names.model rather than model_list.model,“cars” actually refers to both cars_data and car_names (but not car_makers) forthe purpose of table joining.To resolve the column/table references properly, the semantic parser must take into account both the known schemarelations (\egforeign keys) and the question context.Prior work (Bogin et al., 2019a) addressed the schema representation problem by encoding the directed graph of foreign keyrelations in the schema with a graph neural network (GNN).While effective, this approach has two important shortcomings.First, it does not contextualize schema encoding with the question, thus making reasoningabout schema linking difficult after both the column representations and question word representations are built.Second, it limits information propagation during schema encoding to the predefined graph of foreign key relations.The advent of self-attentional mechanisms in NLP (Vaswani et al., 2017) shows thatglobal reasoning is crucial to effective representations of relational structures.However, we would like any global reasoning to still take into account the aforementioned schema relations.In this work, we present a unified framework, called RAT-SQL,Relation-Aware Transformer.for encoding relational structure in the database schema and a given question.It uses relation-aware self-attention to combine global reasoning over the schema entities and question wordswith structured reasoning over predefined schema relations.We then apply RAT-SQL to the problems of schema encoding and schema linking.As a result, we obtain 57.2% exact match accuracy on the Spider test set.At the time of writing, this result is the state of the art among models unaugmented with pretrained BERTembeddings – and further reaches to the overall state of the art (65.6%) when RAT-SQL is augmented withBERT.In addition, we experimentally demonstrate that RAT-SQL enables the model to build more accurate internalrepresentations of the question’s true alignment with schema columns and tables.

Related Work

Semantic parsing of NL to SQL recently surged in popularity thanks to the creation oftwo new multi-table datasets with the challenge of schema generalization –WikiSQL (Zhong et al., 2017) and Spider (Yu et al., 2018b).Schema encoding is not as challenging in WikiSQL as in Spider because it lacks multi-table relations.Schema linking is relevant for both tasks but also more challenging in Spider due to the richer NLexpressiveness and less restricted SQL grammar observed in it.The state of the art semantic parser on WikiSQL (He et al., 2019) achieves a test set accuracy of 91.8%, significantlyhigher than the state of the art on Spider.The recent state-of-the-art models evaluated on Spider use various attentional architectures for question/schemaencoding and AST-based structural architectures for query decoding.IRNet (Guo et al., 2019) encodes the question and schema separately with LSTM and self-attention respectively, augmentingthem with custom type vectors for schema linking.They further use the AST-based decoder of Yin and Neubig (2017) to decode a query in an intermediaterepresentation (IR) that exhibits higher-level abstractions than SQL.Bogin et al. (2019a) encode the schema with a GNN and a similar grammar-based decoder.Both works emphasize schema encoding and schema linking, but design separate featurizationtechniques to augment word vectors (as opposed to relations between words and columns) to resolve it.In contrast, the RAT-SQL framework provides a unified way to encode arbitrary relational information among inputs.Concurrently with this work, Bogin et al. (2019b) published Global-GNN, a different approach to schema linking forSpider, which applies global reasoning between question words and schema columns/tables.Global reasoning is implemented by gating the GNN that encodes the schema using the question token representations.This differs from RAT-SQL in two important ways:(a) question word representations influence the schema representations but not vice versa, and(b) like in other GNN-based encoders, message propagation is limited to the schema-induced edges such asforeign key relations.In contrast, our relation-aware transformer mechanism allows encoding arbitrary relations between question words andschema elements explicitly, and these representations are computed jointly over all inputs using self-attention.We use the same formulation of relation-aware self-attention as Shaw et al. (2018).However, they only apply it to sequences of words in the context of machine translation, and as such, their relation types only encode the relative distance between two words.We extend their work and show that relation-aware self-attention can effectively encode more complex relationships within an unordered set of elements (in our case, columns and tables within a database schema as well as relations between the schema and the question).To the best of our knowledge, this is the first application of relation-aware self-attention to joint representation learning with both predefined and softly induced relations in the input structure.Hellendoorn et al. (2020) develop a similar model concurrently with this work, where they use relation-aware self-attention to encode data flow structure in source code embeddings.Sun et al. (2018) use a heterogeneous graph of KB facts and relevant documents for open-domain question answering.The nodes of their graph are analogous to the database schema nodes in RAT-SQL, but RAT-SQL also incorporates the question in the same formalism to enable joint representation learning between the question and the schema.

Relation-Aware Self-Attention

First, we introduce relation-aware self-attention, a model for embedding semi-structured input sequences in a waythat jointly encodes pre-existing relational structure in the input as well as induced “soft” relations betweensequence elements in the same embedding.Our solutions to schema embedding and linking naturally arise as features implemented in this framework.Consider a set of inputs $X=\left\{\vect{x_{i}}\right\}_{i=1}^{n}$ where $\vect{x_{i}}\in\Reals^{d_{x}}$.In general, we consider it an unordered set, although $\vect{x_{i}}$ may be imbued with positional embeddings to add anexplicit ordering relation.A self-attention encoder, or Transformer, introduced by Vaswani et al. (2017), is a stack ofself-attention layers where each layer (consisting of $H$ heads) transforms each $\vect{x_{i}}$ into$\vect{y_{i}}\in\Reals^{d_{x}}$ as follows:

RAT-SQL

We now describe the RAT-SQL framework and its application to the problems of schema encoding and linking.First, we formally define the text-to-SQL semantic parsing problem and its components.In the rest of the section, we present our implementation of schema linking in the RAT framework.

Given a natural language question $Q$ and a schema $\mathcal{S}=\langle\mathcal{C},\mathcal{T}\rangle$ for arelational database, our goal is to generate the corresponding SQL $P$.Here the question $Q=q_{1}\dots q_{|Q|}$ is a sequence of words,and the schema consists of columns $\mathcal{C}=\{c_{1},\dots,c_{|\mathcal{C}|}\}$and tables $\mathcal{T}=\left\{t_{1},\dots,t_{|\mathcal{T}|}\right\}$.Each column name $c_{i}$ contains words $c_{i,1},\dots,c_{i,|c_{i}|}$ andeach table name $t_{i}$ contains words $t_{i,1},\dots,t_{i,|t_{i}|}$.The desired program $P$ is represented as an abstract syntax tree $T$ in the context-free grammar of SQL.Some columns in the schema are primary keys, used for uniquely indexing the corresponding table,and some are foreign keys, used to reference a primary key column in a different table.In addition, each column has a type $\tau\in\{\texttt{number},\,\texttt{text}\}$.

Formally, we represent the database schema as a directedgraph $\mathcal{G}=\langle\mathcal{V},\mathcal{E}\rangle$.Its nodes $\mathcal{V}=\mathcal{C}\cup\mathcal{T}$ are the columns and tables of the schema, each labeled with thewords in its name (for columns, we prepend their type $\tau$ to the label).Its edges $\mathcal{E}$ are defined by the pre-existing database relations, described inTable 1.Figure 2 illustrates an example graph (with a subset of actual edges and labels).While $\mathcal{G}$ holds all the known information about the schema, it is insufficient for appropriately encoding apreviously unseen schema in the context of the question $Q$.We would like our representations of the schema $\mathcal{S}$ and the question $Q$ to be joint, in particularfor modeling the alignment between them.Thus, we also define the question-contextualized schema graph$\mathcal{G}_{Q}=\langle\mathcal{V}_{Q},\mathcal{E}_{Q}\rangle$where $\mathcal{V}_{Q}=\mathcal{V}\cup Q=\mathcal{C}\cup\mathcal{T}\cup Q$ includesnodes for the question words (each labeled with a corresponding word), and$\mathcal{E}_{Q}=\mathcal{E}\cup\mathcal{E}_{Q\leftrightarrow\mathcal{S}}$are the schema edges $\mathcal{E}$ extended with additional special relations between the question words and schemamembers, detailed in the rest of this section.For modeling text-to-SQL generation, we adopt the encoder-decoder framework.Given the input as a graph $\mathcal{G}_{Q}$, the encoder $f_{\text{enc}}$ embeds it into joint representations$\vect{c}_{i}$, $\vect{t}_{i}$, $\vect{q}_{i}$ for each column $c_{i}\in\mathcal{C}$, table $t_{i}\in\mathcal{T}$, and question word $q\in Q$ respectively.The decoder $f_{\text{dec}}$ then uses them to compute a distribution $\Pr(P\mid\mathcal{G}_{Q})$ overthe SQL programs.

2 Relation-Aware Input Encoding

Following the state-of-the-art NLP literature, our encoder first obtains the initialrepresentations $\vect{c}_{i}^{\text{init}}$, $\vect{t}_{i}^{\text{init}}$for every node of $\mathcal{G}$ by(a) retrieving a pre-trained Glove embedding (Pennington et al., 2014) for each word, and(b) processing the embeddings in each multi-word label with a bidirectional LSTM (BiLSTM) (Hochreiter and Schmidhuber, 1997).It also runs a separate BiLSTM over the question $Q$ to obtain initial word representations$\vect{q}_{i}^{\text{init}}$.The initial representations $\vect{c}_{i}^{\text{init}}$, $\vect{t}_{i}^{\text{init}}$, and $\vect{q}_{i}^{\text{init}}$are independent of each other and devoid of any relational information known to hold in $\mathcal{E}_{Q}$.To produce joint representations for the entire input graph $\mathcal{G}_{Q}$, we use the relation-aware self-attentionmechanism (Section 3).Its input $X$ is the set of all the node representations in $\mathcal{G}_{Q}$:

The encoder $f_{\text{enc}}$ applies a stack of $N$ relation-aware self-attention layers to $X$,with separate weight matrices in each layer.The final representations $\vect{c}_{i}$, $\vect{t}_{i}$, $\vect{q}_{i}$ produced by the $N^{\text{th}}$ layerconstitute the output of the whole encoder.Alternatively, we also consider pre-trained BERT (Devlin et al., 2019) embeddings to obtain the initial representations.Following Huang et al. (2019); Zhang et al. (2019), we feed $X$ to the BERT and use the last hidden states as the initial representations before proceeding with the RAT layers.In this case, the initial representations $\vect{c}_{i}^{\text{init}}$, $\vect{t}_{i}^{\text{init}}$, $\vect{q}_{i}^{\text{init}}$ are not strictly independent although still yet uninfluenced by $\mathcal{E}$.Importantly, as detailed in Section 3, every RAT layer uses self-attention between all elements of the inputgraph $\mathcal{G}_{Q}$ to compute new contextual representations of question words and schema members.However, this self-attention is biased toward some pre-defined relations using the edge vectors$\vect{r_{ij}^{K}},\vect{r_{ij}^{V}}$ in each layer.We define the set of used relation types in a way that directly addresses the challenges of schema embedding andlinking.Occurrences of these relations between the question and the schema constitute the edges $\mathcal{E}_{Q\leftrightarrow\mathcal{S}}$.Most of these relation types address schema linking (Section 4.3);we also add some auxiliary edges to aid schema encoding(see Appendix A).

3 Schema Linking

Schema linking relations in $\mathcal{E}_{Q\leftrightarrow\mathcal{S}}$ aid the model with aligning column/tablereferences in the question to the corresponding schema columns/tables.This alignment is implicitly defined by two kinds of information in the input: matching names and matchingvalues, which we detail in order below.

Name-based linking refers to exact or partial occurrences of the column/table names in the question,such as the occurrences of “cylinders” and “cars” in the question in Figure 1.Textual matches are the most explicit evidence of question-schema alignment and as such, one might expect them to bedirectly beneficial to the encoder.However, in all our experiments the representations produced by vanilla self-attention were insensitive totextual matches even though their initial representations were identical.Brunner et al. (2020) suggest that representations produced by Transformers mix the information fromdifferent positions and cease to be directly interpretable after 2+ layers, which might explain our observations.Thus, to remedy this phenomenon, we explicitly encode name-based linking using RAT relations.Specifically, for all n-grams of length 1 to 5 in the question,we determine (1) whether it exactly matches the name of a column/table (exact match);or (2) whether the n-gram is a subsequence of the name of a column/table (partial match).This procedurematches that of Guo et al. (2019), but we use the matching information differently in RAT.Then, for every $(i,j)$ where $x_{i}\in Q$, $x_{j}\in\mathcal{S}$ (or vice versa), we set$r_{ij}\in\mathcal{E}_{Q\leftrightarrow\mathcal{S}}$ toQuestion-Column-M, Question-Table-M, Column-Question-M orTable-Question-M depending on the type of $x_{i}$ and $x_{j}$.Here M is one of ExactMatch, PartialMatch, or NoMatch.

Value-Based Linking

Question-schema alignment also occurs when the question mentions any values that occur in the database andconsequently participate in the desired SQL, such as “4” in Figure 1.While this example makes the alignment explicit by mentioning the column name “cylinders”, many real-worldquestions do not.Thus, linking a value to the corresponding column requires background knowledge.The database itself is the most comprehensive and readily available source of knowledge about possible values, but alsothe most challenging to process in an end-to-end model because of the privacy and speed impact.However, the RAT framework allows us to outsource this processing to the database engine to augment $\mathcal{G}_{Q}$with potential value-based linking without exposing the model itself to the data.Specifically, we add a new Column-Value relation between any word $q_{i}$ and column name $c_{j}$ s.t.$q_{i}$ occurs as a value (or a full word within a value) of $c_{j}$.This simple approach drastically improves the performance of RAT-SQL (see Section 5).It also directly addresses the aforementioned DB challenges:(a) the model is never exposed to database content that does not occur in the question,(b) word matches are retrieved quickly via DB indices & textual search.

Memory-Schema Alignment Matrix

Intuitively, the alignment matrices in Eq. 3 should resemble the real discrete alignments, therefore should respect certain constraints like sparsity.When the encoder is sufficiently parameterized, sparsity tends to arise with learning, but we can also encourage it with an explicit objective.Appendix B presents this objective and discusses our experiments with sparse alignment in RAT-SQL.

4 Decoder

The decoder $f_{\text{dec}}$ of RAT-SQL follows the tree-structured architecture of Yin and Neubig (2017).It generates the SQL $P$ as an abstract syntax tree in depth-first traversal order,by using an LSTM to output a sequence of decoder actions that either (i) expand the last generated node into a grammar rule, called ApplyRule; or when completing a leaf node, (ii) choose acolumn/table from the schema, called SelectColumn and SelectTable.Formally,$\Pr(P\mid\mathcal{Y})=\prod_{t}\Pr(a_{t}\mid a_{<t},\,\mathcal{Y})$where $\mathcal{Y}=f_{\text{enc}}(\mathcal{G}_{Q})$ is the final encoding of the question and schema,and $a_{<t}$ are all the previous actions.In a tree-structured decoder, the LSTM state is updated as$\vect{m}_{t},\vect{h}_{t}=f_{\text{LSTM}}\left([\vect{a}_{t-1}\conc\vect{z}_{t}\conc\vect{h}_{p_{t}}\conc\vect{a}_{p_{t}}\conc\vect{n}_{f_{t}}],\ \vect{m}_{t-1},\vect{h}_{t-1}\right)$where $\vect{m}_{t}$ is the LSTM cell state, $\vect{h}_{t}$ is the LSTM output at step $t$, $\vect{a}_{t-1}$ is theembedding of the previous action, $p_{t}$ is the step corresponding to expanding the parent AST node of the current node,and $\vect{n}_{f_{t}}$ is the embedding of the current node type.Finally, $\vect{z}_{t}$ is the context representation, computed using multi-head attention (with 8 heads) on$\vect{h}_{t-1}$ over $\mathcal{Y}$.For ApplyRule$[R]$, we compute $\Pr(a_{t}=\textsc{ApplyRule}[R]\mid a_{<t},y)=\mathsf{softmax}_{R}\left(g(\vect{h}_{t})\right)$where $g(\cdot)$ is a 2-layer MLP with a $\mathsf{tanh}$ non-linearity.For SelectColumn, we compute

and similarly for SelectTable.We refer the reader to Yin and Neubig (2017) for details.

Experiments

We implemented RAT-SQL in PyTorch (Paszke et al., 2017).During preprocessing, the input of questions, column names and table names are tokenized and lemmatized with the StandfordNLP toolkit (Manning et al., 2014).Within the encoder, we use GloVe (Pennington et al., 2014) word embeddings, held fixed in training except for the 50 most common words in the training set.For RAT-SQL BERT, we use the WordPiece tokenization.All word embeddings have dimension $300$.The bidirectional LSTMs have hidden size 128 per direction, and use the recurrent dropout method ofGal and Ghahramani (2016) with rate $0.2$.We stack 8 relation-aware self-attention layers on top of the bidirectional LSTMs.Within them, we set $d_{x}=d_{z}=256$, $H=8$, and use dropout with rate $0.1$.The position-wise feed-forward network has inner layer dimension 1024.Inside the decoder, we use rule embeddings of size $128$, node type embeddings of size $64$, and a hidden size of $512$inside the LSTM with dropout of $0.21$.We used the Adam optimizer (Kingma and Ba, 2015) with the default hyperparameters.During the first $warmup\_steps=max\_steps/20$ steps of training, the learning rate linearly increasesfrom 0 to $\numprint{7.4e-4}$.Afterwards, it is annealed to 0 with formula$10^{-3}(1-\frac{step-warmup\_steps}{max\_steps-warmup\_steps})^{-0.5}$.We use a batch size of 20 and train for up to 40,000 steps.For RAT-SQL + BERT, we use a separate learning rate of $\numprint{3e-6}$ to fine-tune BERT,a batch size of 24 and train for up to 90,000 steps.

We tuned the batch size (20, 50, 80), number of RAT layers (4, 6, 8), dropout (uniformly sampled from $[0.1,0.3]$),hidden size of decoder RNN (256, 512), max learning rate (log-uniformly sampled from $[\numprint{5e-4},\,\numprint{2e-3}]$).We randomly sampled 100 configurations and optimized on the dev set.RAT-SQL + BERT reuses most hyperparameters of RAT-SQL, only tuning the BERT learning rate (1×10-4, 3×10-4, 5×10-4), number of RAT layers (6, 8, 10), numberof training steps (4×104, 6×104, 9×104).

1 Datasets and Metrics

We use the Spider dataset (Yu et al., 2018b) for most of our experiments, and also conduct preliminary experiments onWikiSQL (Zhong et al., 2017)to confirm generalization to other datasets.As described by Yu et al., Spider contains 8,659 examples (questions and SQLqueries, with the accompanying schemas), including 1,659 examples lifted from the Restaurants(Popescu et al., 2003; Tang and Mooney, 2000), GeoQuery (Zelle and Mooney, 1996), Scholar(Iyer et al., 2017), Academic (Li and Jagadish, 2014),Yelp and IMDB (Yaghmazadeh et al., 2017) datasets.As Yu et al. (2018b) make the test set accessible only through an evaluation server, we perform most evaluations (other than the final accuracy measurement) using the development set.It contains 1,034 examples, with databases and schemas distinct from those in the training set.We report results using the same metrics as Yu et al. (2018a):exact match accuracy on all examples, as well as divided by difficulty levels.As in previous work on Spider, these metrics do not measure the model’s performance on generating values in the SQL.

2 Spider Results

In Table 2 we show accuracy on the (hidden) Spider test set for RAT-SQL and compare to all other approaches at or near state-of-the-art (according to the official leaderboard).RAT-SQL outperforms all other methods that are not augmented with BERT embeddings by a large margin of 8.7%.Surprisingly, it even beats other BERT-augmented models.When RAT-SQL is further augmented with BERT, it achieves the new state-of-the-art performance.Compared with other BERT-argumented models, our RAT-SQL + BERT has smaller generalization gap between development and test set.We also provide a breakdown of the accuracy by difficulty in Table 3.As expected, performance drops with increasing difficulty.The overall generalization gap between development and test of RAT-SQL was strongly affected by the significant drop inaccuracy (9%) on the extra hard questions.When RAT-SQL is augmented with BERT, the generalization gaps ofmost difficulties are reduced.

Table 4 shows an ablation study over different RAT-based relations.The ablations are run on RAT-SQL without value-based linking to avoid interference with information from the database.Schema linking and graph relations make statistically significant improvements (p<0.001).The full model accuracy here slightly differs from Table 2 becausethe latter shows the best model from a hyper-parameter sweep (used for test evaluation) and the former givesthe mean over five runs where we only change the random seeds.

3 WikiSQL Results

We also conducted preliminary experiments on WikiSQL (Zhong et al., 2017) to testgeneralization of RAT-SQL to new datasets.Although WikiSQL lacks multi-table schemas (and thus, its challenge of schema encoding is not as prominent), it stillpresents the challenges of schema linking and generalization to new schemas.For simplicity of experiments, we did not implement either BERT augmentation or execution-guided decoding (EG) (Wang et al., 2018),both of which are common in state-of-the-art WikiSQL models.We thus only compare to the models that also lack these two enhancements.While not reaching state of the art, RAT-SQL still achieves competitive performance on WikiSQL as shown inTable 5.Most of the gap between its accuracy and state of the art is due to the simplified implementation of valuedecoding, which is required for WikiSQL evaluation but not in Spider.Our value decoding for these experiments is a simple token-based pointer mechanism, which often fails to retrievemulti-token value constants accurately.A robust value decoding mechanism in RAT-SQL is an important extension that we plan to address outsidethe scope of this work.

4 Discussions

Recall from Section 4 that we explicitly model the alignment matrix between question words and tablecolumns, used during decoding for column and table selection.The existence of the alignment matrix provides a mechanism for the model to align words to columns.An accurate alignment representation has other benefits such as identifying question words to copy to emit a constant value in SQL.In Figure 5 we show the alignment generated by our model on the example from Figure 1.The full alignment also maps from column and table names, but those end up simply aligning to themselves or the tablethey belong to, so we omit them for brevity.For the three words that reference columns (“cylinders”, “model”, “horsepower”), the alignment matrixcorrectly identifies their corresponding columns.The alignments of other words are strongly affected by these three keywords, resulting in a sparse span-to-column like alignment, \eg“largest horsepower” to horsepower.The tables cars_data and cars_namesare implicitly mentioned by the word “cars”.The alignment matrix successfully infers to use these two tables instead of car_makers using the evidence that they contain the three mentioned columns.

The Need for Schema Linking

One natural question is how often does the decoder fail to select the correct column, even with the schema encoding and linking improvements we have made.To answer this, we conducted an oracle experiment (see Table 6).For “oracle sketch”, at every grammar nonterminal the decoder is forced to choose the correct production so the final SQLsketch exactly matches that of the ground truth.The rest of the decoding proceeds conditioned on that choice.Likewise, “oracle columns” forces the decoder to emit the correct column/table at terminal nodes.With both oracles, we see an accuracy of 99.4% which just verifies that our grammar is sufficient to answer nearlyevery question in the data set.With just “oracle sketch”, the accuracy is only 73.0%, which means 72.4% of the questions that RAT-SQL gets wrongand could get right have incorrect column or table selection.Similarly, with just “oracle columns”, the accuracy is 69.8%, which means that 81.0% of the questions that RAT-SQL getswrong have incorrect structure.In other words, most questions have both column and structure wrong, so both problems require important future work.

Error Analysis

An analysis of mispredicted SQL queries in the Spider dev set showed three main causes of evaluation errors.(I) 18% of the mispredicted queries are in fact equivalent implementations of the NL intent with a different SQL syntax (\egORDER BY $C$ LIMIT 1 vs. SELECT MIN($C$)).Measuring execution accuracy rather than exact match would detect them as valid.(II) 39% of errors involve a wrong, missing, or extraneous column in the SELECT clause.This is a limitation of our schema linking mechanism, which, while substantially improving column resolution, still struggles with some ambiguous references.Some of them are unavoidable as Spider questions do not always specify which columns should be returned by the desired SQL.Finally, (III) 29% of errors are missing a WHERE clause, which is a common error class in text-to-SQL models as reported by prior works.One common example is domain-specific phrasing such as “older than 21”, which requires background knowledge to map it to age > 21 rather than age < 21.Such errors disappear after in-domain fine-tuning.

Conclusion

Despite active research in text-to-SQL parsing, many contemporary models struggle to learn goodrepresentations for a given database schema as well as to properly link column/table references in the question.These problems are related: to encode & use columns/tables from the schema, the model must reason about their role inthe context of the question.In this work, we present a unified framework for addressing the schema encoding and linking challenges.Thanks to relation-aware self-attention, it jointly learns schema and question representations based on theiralignment with each other and schema relations.Empirically, the RAT framework allows us to gain significant state of the art improvement on text-to-SQL parsing.Qualitatively, it provides a way to combine predefined hard schema relations and inferred softself-attended relations in the same encoder architecture.This representation learning will be beneficial in tasks beyond text-to-SQL, as long as theinput has some predefined structure.

Acknowledgments

We thank Jianfeng Gao, Vladlen Koltun, Chris Meek, and Vignesh Shiv for the discussions that helped shape this work.We thank Bo Pang, Tao Yu for their help with the evaluation.We also thank anonymous reviewers for their invaluable feedback.

References

Appendix A Auxiliary Relations for Schema Encoding

In addition to the schema graph edges $\mathcal{E}$ (Section 4.2)and schema linking edges (Section 4.3),the edges in $\mathcal{E}_{Q}$ also include some auxiliary relation typesto aid the relation-aware self-attention.Specifically, for each $x_{i},x_{j}\in\mathcal{V}_{Q}$:

If $i=j$, then Column-Identity or Table-Identity.

$x_{i}\in Q$, $x_{j}\in Q$:Question-Dist-$d$, where

Otherwise, one of Column-Column, Column-Table, Table-Column, orTable-Table.

Appendix B Alignment Loss

The memory-schema alignment matrix is expected to resemble the real discrete alignments, thereforeshould respect certain constraints like sparsity.For example, the question word “model” in Figure 1 should be aligned withcar_names.model rather than model_list.model or model_list.model_id.To further bias the soft alignment towards the real discrete structures,we add an auxiliary loss to encourage sparsity of the alignment matrix.Specifically, for a column/table that is mentioned in the SQL query,we treat the model’s current belief of the best alignment as the ground truth.Then we use a cross-entropy loss, referred as alignment loss,to strengthen the model’s belief:

where $Rel(\mathcal{C})$ and $Rel(\mathcal{T})$ denote the set of relevant columns and tables that appear in the SQL.In earlier experiments, we found that the alignment loss did improve the model (statistically significantly, from 53.0% to 55.4%).However, it does not make a statistically significant difference in our final model in terms of overall exact match.We hypothesize that hyperparameter tuning that caused us to increase encoding depth eliminated the need for explicit supervision of alignment.With few layers in the Transformer, the alignment matrix provided additional degrees of freedom, which became unnecessary once the Transformer was sufficiently deep to build a rich joint representation of the question and the schema.

Appendix C Consistency of RAT-SQL

In Spider dataset, most SQL queries correspond to more than one question, making it possible to evaluate the consistency of RAT-SQL given paraphrases. We use two metrics to evaluate the consistency: 1) Exact Match – whether RAT-SQL produces the exact same predictions given paraphrases, 2) Correctness – whether RAT-SQL achieves the same correctness given paraphrases. The analysis is conducted on the development set.

The results are shown in Table 7. We found that when augmented with BERT, RAT-SQL becomes more consistent in terms of both metrics, indicating the pre-trained representations of BERT are beneficial for handling paraphrases.