LGESQL: Line Graph Enhanced Text-to-SQL Model with Mixed Local and Non-Local Relations

Introduction

The text-to-SQL task Zhong et al. (2017); Xu et al. (2017) aims to convert a natural language question into a SQL query, given the corresponding database schema. It has been widely studied in both academic and industrial communities to build natural language interfaces to databases (NLIDB, Androutsopoulos et al., 1995).

One daunting problem is how to jointly encode the question words and database schema items (including tables and columns), as well as various relations among these heterogeneous inputs. Typically, previous literature utilizes a node-centric graph neural network (GNN, Scarselli et al., 2008) to aggregate information from neighboring nodes. GNNSQL Bogin et al. (2019a) adopts a relational graph convolution network (RGCN, Schlichtkrull et al., 2018) to take into account different edge types between schema items, such as T-Has-C relationship For abbreviation, Q represents Question node, while T and C represent Table and Column nodes., primary key and foreign key constraints. However, these edge features are directly retrieved from a fixed-size parameter matrix and may suffer from the drawback: unaware of contextualized information, especially the structural topology of edges. Meta-path is defined as a composite relation linking two objects, which can be used to capture multi-hop semantics. For example, in Figure 1(a), relation Q-ExactMatch-C and C-BelongsTo-T can form a 2-hop meta-path indicating that some table $t$ has one column exactly mentioned in the question.

Although RATSQL Wang et al. (2020a) introduces some useful meta-paths such as C-SameTable-C, it treats all relations, either $1$-hop or multi-hop, in the same manner (relative position embedding, Shaw et al., 2018) in a complete graph. Without distinguishing local and non-local neighbors, see Figure 1(b), each node will attend to all the other nodes equally, which may lead to the notorious over-smoothing problem Chen et al. (2020a). Besides, meta-paths are currently constructed by domain experts or explored by breadth-first search Kong et al. (2012). Unfortunately, the number of possible meta-paths increases exponentially with the path length, and selecting the most important subset among them is an NP-complete problem Lao and Cohen (2010).

To address the above limitations, we propose a Line Graph Enhanced Text-to-SQL model (LGESQL), which explicitly considers the topological structure of edges. According to the definition of a line graph Gross and Yellen (2005), we firstly construct an edge-centric graph from the original node-centric graph. These two graphs capture the structural topology of nodes and edges, respectively. Iteratively, each node in either graph gathers information from its neighborhood and incorporates edge features from the dual graph to update its representation. As for the node-centric graph, we combine both local and non-local edge features into the computation. Local edge features denote $1$-hop relations and are dynamically provided by node embeddings in the line graph, while non-local edge features are directly extracted from a parameter matrix. This distinction encourages the model to pay more attention to local edge features while maintaining information from multi-hop neighbors. Additionally, we propose an auxiliary task called graph pruning. It introduces an inductive bias that the heterogeneous graph encoder of text-to-SQL should be intelligent to extract the golden schema items related to the question from the entire database schema graph.

Experimental results on benchmark Spider Yu et al. (2018b) demonstrate that our LGESQL model promotes the exact set match accuracy to $62.8\%$ (with GloVe, Pennington et al. 2014) and $72.0\%$ (with pretrained language model Electra, Clark et al. 2020). Our main contributions are summarized as follows:

We propose to model the $1$-hop edge features with a line graph in text-to-SQL. Both non-local and local features are integrated during the iteration process of node embeddings.

We design an auxiliary task called graph pruning, which aims to determine whether each node in the database schema graph is relevant to the given question.

Empirical results on dataset Spider demonstrate that our model is effective, and we achieve state-of-the-art performances both without and with pre-trained language models.

Preliminaries

Given a natural language question $Q=(q_{1},q_{2},\cdots,q_{|Q|})$ with length $|Q|$ and the corresponding database schema $S=T\cup C$, the target is to generate a SQL query $y$. The database schema $S$ contains multiple tables $T=\{t_{1},t_{2},\cdots\}$ and columns $C=\{c_{1}^{t_{1}},c_{2}^{t_{1}},\cdots,c_{1}^{t_{2}},c_{2}^{t_{2}},\cdots\}$. Each table $t_{i}$ is described by its name and is further composed of several words $(t_{i1},t_{i2},\cdots)$. Similarly, we use word phrase $(c_{j1}^{t_{i}},c_{j2}^{t_{i}},\cdots)$ to represent column $c_{j}^{t_{i}}\in t_{i}$. Besides, each column $c_{j}^{t_{i}}$ also has a type field $c_{j0}^{t_{i}}$ to constrain its cell values (e.g. Text and Number).

The entire input node-centric heterogeneous graph $G^{n}=(V^{n},R^{n})$ consists of all three types of nodes mentioned above, that is $V^{n}=Q\cup T\cup C$ with the number of nodes $|V^{n}|=|Q|+|T|+|C|$, where $|T|$ and $|C|$ are the number of tables and columns respectively.

As shown in Figure 1(a), a meta-path represents a path $\tau_{1}\overset{r_{1}}{\rightarrow}\tau_{2}\overset{r_{2}}{\rightarrow}\cdots\overset{r_{l}}{\rightarrow}\tau_{l+1}$, where the target vertex type of previous relation $r_{i-1}$ equals to the source vertex type $\tau_{i}$ of the current relation $r_{i}$. It describes a composite relation $r=r_{1}\circ r_{2}\cdots\circ r_{l}$ between nodes with type $\tau_{1}$ and $\tau_{l+1}$. In this work, $\tau_{i}\in\{\textsc{Question,Table,Column}\}$. Throughout our discussion, we use the term local to denote relations with path length $1$, while non-local relations refer to meta-paths longer than $1$. The relational adjacency matrix $R^{n}$ contains both local and non-local relations, see Appendix A for enumeration.

Each vertex $v_{i}^{e},i=1,2,\cdots,|V^{e}|$ in the line graph $G^{e}=(V^{e},R^{e})$ can be uniquely mapped to a directed edge $r_{st}^{n}\in R^{n}$, or $v_{s}^{n}\rightarrow v_{t}^{n}$, in the original node-centric graph $G^{n}=(V^{n},R^{n})$. Function $f$ maps the source and target node index tuple $(s,t)$ into the “edge” index $i=f(s,t)$ in $G^{e}$. The reverse mapping is $f^{\textrm{-}1}$. In the line graph $G^{e}$, a directed edge $r^{e}_{ij}\in R^{e}$ exists from node $v^{e}_{i}$ to $v^{e}_{j}$, iff the target node of edge $r^{n}_{f^{\textrm{-}1}(i)}$ and the source node of edge $r^{n}_{f^{\textrm{-}1}(j)}$ in $G^{n}$ are exactly the same node. Actually, $r^{e}_{ij}$ captures the information flow in meta-path $r^{n}_{f^{\textrm{-}1}(i)}\circ r^{n}_{f^{\textrm{-}1}(j)}$. We prevent back-tracking cases where two reverse edges will not be connected in $G^{e}$, illustrated in Figure 2.

We only utilize local relations in $R^{n}$ as the node set $V^{e}$ to avoid creating too many nodes in the line graph $G^{e}$. Symmetrically, each edge in $R^{e}$ can be uniquely identified by the node in $V^{n}$. For example, in the upper right part of Figure 2, the edge between nodes “e1” and “e2” in the line graph can be represented by the middle node with double solid borderlines in the original graph.

Method

Firstly, we flatten all question words and schema items into a sequence, where columns belong to the same table are clustered together Following Suhr et al. (2020), we randomly shuffle the order of tables and columns in different mini-batches to discourage over-fitting.: [CLS]$q_{1}q_{2}\cdots q_{|Q|}$[SEP]$t_{10}t_{1}c_{10}^{t_{1}}c_{1}^{t_{1}}c_{20}^{t_{1}}c_{2}^{t_{1}}\cdots$ $t_{20}t_{2}c_{10}^{t_{2}}c_{1}^{t_{2}}c_{20}^{t_{2}}c_{2}^{t_{2}}\cdots$[SEP]. The type information $t_{i0}$ or $c_{j0}^{t_{i}}$ is inserted before each schema item. Since each word $w$ is tokenized into sub-words, we append a subword attentive pooling layer after PLM to obtain word-level representations. Concretely, given the output sequence of subword features $\mathbf{w}^{s}_{1},\mathbf{w}^{s}_{2},\cdots,\mathbf{w}^{s}_{|w|}$ for each subword $w^{s}_{i}$ in $w$, the word-level representation $\mathbf{w}$ is Vectors throughout this paper are all row vectors.

where $\mathbf{v}_{s}$ and $\mathbf{W}_{s}$ are trainable parameters. After obtaining the word vectors, we also feed them into three BiLSTMs according to the node types and get the graph inputs $\mathbf{X}^{0}$ for all nodes.

2 Line Graph Enhanced Hidden Module

It contains a stack of $L$ dual relational graph attention network (Dual RGAT) layers. In each layer $l$, two RGATs Wang et al. (2020b) capture the structure of the original graph and line graph, respectively. Node embeddings in one graph play the role of edge features in another graph. For example, the edge features used in graph $G^{n}$ are provided by the node embeddings in graph $G^{e}$.

where $\mathbf{Z}_{nlc}$ is the aforementioned non-local edge features in the original graph $G^{n}$.

Given the node-centric graph $G^{n}$, the output representation $\mathbf{x}^{l+1}_{i}$ of the $l$-th layer is computed by

If $r_{ji}^{n}$ is a local relation, $\psi(r_{ji}^{n})$ returns the node embedding $\mathbf{z}^{l}_{f(j,i)}$ from the line graphFunction $f$ maps the tuple of source and target node indices in $G^{n}$ into the corresponding node index in $G^{e}$.. Otherwise, $\psi(r_{ji}^{n})$ directly retrieves the vector from the non-local embedding matrix $\mathbf{Z}_{nlc}$, see Figure 4. The neighborhood function $\mathcal{N}^{n}_{i}$ for node $v^{n}_{i}$ returns the entire node set $V^{n}$ and is shared across different heads.

An alternative is to split the muli-head attention module into two parts. In half of the heads, the neighborhood function $\mathcal{N}^{n}_{i}$ of node $v^{n}_{i}$ only contains nodes that are reachable within $1$-hop. In this case, $\psi(r_{ji}^{n})$ returns the layer-wise updated feature $\mathbf{z}^{l}_{f(j,i)}$ from $\mathbf{Z}^{l}$. In the other heads, each node has access to both local and non-local neighbors, and $\psi(\cdot)$ always returns static entries in the embedding matrix $\mathbf{Z}_{nlc}\cup\mathbf{Z}^{0}$, see Figure 5 for illustration.

In either scheme, the RGAT module treats local and non-local relations differently and relatively manipulates the local edge features more carefully.

2.2 RGAT for the Line Graph

Symmetrically, given edge-centric graph $G^{e}$, the updated node representation $\mathbf{z}^{l+1}_{i}$ from $\mathbf{z}^{l}_{i}$ is calculated similarly with little modifications:

The output matrices of the final layer $L$ are the desired outputs of the encoder: $\mathbf{X}=\mathbf{X}^{L},\mathbf{Z}=\mathbf{Z}^{L}$.

3 Graph Output Module

We adopt the grammar-based syntactic neural decoder Yin and Neubig (2017) to generate the abstract syntax tree (AST) of the target query $y$ in depth-first-search order. The output at each decoding timestep is either 1) an ApplyRule action that expands the current non-terminal node in the partially generated AST, or 2) SelectTable or SelectColumn action that chooses one schema item $\mathbf{x}_{s_{i}}$ from the encoded memory $\mathbf{X}_{s}=\mathbf{X}_{t}\cup\mathbf{X}_{c}$. Mathematically, $P(y|\mathbf{X})=\prod_{j}P(a_{j}|a_{<j},\mathbf{X})$, where $a_{j}$ is the action at the $j$-th timestep. For more implementation details, see Appendix B.

3.2 Graph Pruning

We hypothesize that a powerful encoder should distinguish irrelevant schema items from golden schema items used in the target query. In Figure 6, the question-oriented schema sub-graph (above the shadow region) can be easily extracted. The intent $c2$ and the constraint $c5$ are usually explicitly mentioned in the question, identified by dot-product attention mechanism or schema linking. The linking nodes such as $t1,c3,c4,t2$ can be inferred by the $1$-hop connections of the schema graph to form a connected component. To introduce this inductive bias, we design an auxiliary task that aims to classify each schema node $s_{i}\in S=T\cup C$ based on its relevance with the question and the sparse structure of the schema graph.

The ground truth label $y_{s_{i}}^{g}$ of a schema item is $1$ iff $s_{i}$ appears in the target SQL query. The training object can be formulated as

This auxiliary task is combined with the main text-to-SQL task in a multitasking way. Similar ideas Bogin et al. (2019b); Yu et al. (2020) and other association schemes are discussed in Appendix C.

Experiments

In this section, we evaluate our LGESQL model in different settings. Codes are public available https://github.com/rhythmcao/text2sql-lgesql.git..

Spider Yu et al. (2018b) is a large-scale cross-domain zero-shot text-to-SQL benchmark Leaderboard of the challenge: https://yale-lily.github.io//spider.. It contains $8659$ training examples across $146$ databases in total, and covers several domains from other datasets such as Restaurants Popescu et al. (2003), GeoQuery Zelle and Mooney (1996), Scholar Iyer et al. (2017), Academic Li and Jagadish (2014), Yelp and IMDB Yaghmazadeh et al. (2017) datasets. The detailed statistics are shown in Table 1. We follow the common practice to report the exact set match accuracy on the validation and test dataset. The test dataset contains $2147$ samples with $40$ unseen databases but is not public available. We submit our model to the organizer of the challenge for evaluation.

We preprocess the questions, table names, and column names with toolkit Stanza Qi et al. (2020) for tokenization and lemmatization. Our model is implemented with Pytorch Paszke et al. (2019), and the original and line graphs are constructed with library DGL Wang et al. (2019a). Within the encoder, we use GloVe Pennington et al. (2014) word embeddings with dimension $300$ or pretrained language models (PLMs), Bert Devlin et al. (2019) or Electra (Clark et al., 2020), to leverage contextual information. With GloVe, embeddings of the most frequent $50$ words in the training set are fixed during training while the remaining will be fine-tuned. The schema linking strategy is borrowed from RATSQL Wang et al. (2020a), which is also our baseline system. During evaluation, we adopt beam search decoding with beam size $5$.

In the encoder, the GNN hidden size $d$ is set to $256$ for GloVe and $512$ for PLMs. The number of GNN layers $L$ is $8$. In the decoder, the dimension of hidden state, action embedding and node type embedding are set to $512$, $128$ and $128$ respectively. The recurrent dropout rate Gal and Ghahramani (2016) is $0.2$ for decoder LSTM. The number of heads in multi-head attention is $8$ and the dropout rate of features is set to $0.2$ in both the encoder and decoder. Throughout the experiments, we use AdamW Loshchilov and Hutter (2019) optimizer with linear warmup scheduler. The warmup ratio of total training steps is $0.1$. For GloVe, the learning rate is $5e\textrm{-}4$ and the weight decay coefficient is $1e\textrm{-}4$; For PLMs, we use smaller leaning rate $2e\textrm{-}5$ (base) or $1e\textrm{-}5$ (large), and larger weight decay rate $0.1$. The optimization of the PLM encoder is carried out more carefully with layer-wise learning rate decay coefficient $0.8$. Batch size is $20$ and the maximum gradient norm is $5$. The number of training epochs is $100$ for Glove, and $200$ for PLMs respectively.

2 Main Results

The main results of the test set are provided in Table 2. Our proposed line graph enhanced text-to-SQL (LGESQL) model achieves state-of-the-art results in all configurations at the time of writing. With word vectors GloVe, the performance increases from $57.2\%$ to $62.8\%$, $5.6\%$ absolute improvements. With PLM bert-large-wwm, LGESQL also surpasses all previous methods, including the ensemble model, and attains $68.3\%$ accuracy. Recently, more advanced approaches all leverage the benefits of larger PLMs, more task adaptive data (text-table pairs), and tailored pre-training tasks. For example, Gap Shi et al. (2020) designs some task adaptive self-supervised tasks such as column prediction and column recovery to better address the downstream joint encoding problem. We utilize electra-large for its compatibility with our model and achieves $72.0\%$ accuracy.

Taking one step further, we compare more fine-grained performances of our model to the baseline system RATSQL Wang et al. (2020a) classified by the level of difficulty in Table 3. We observe that LGESQL surpasses RATSQL across all subdivisions in both the validation and test datasets regardless of the application of a PLM, especially at the Medium and Extra Hard levels. This validates the superiority of our model by exploiting the structural relations among edges in the line graph.

3 Ablation Studies

In this section, we investigate the contribution of each design choice. We report the average accuracy on the validation dataset with $5$ random seeds.

RGATSQL is our baseline system where the line graph is not utilized. It can be viewed as a variant of RATSQL with our tailored grammar-based decoder. From Table 4, we can discover that: 1) if non-local relations or meta-paths are removed (w/o NLC), the performance will decrease roughly by $2$ points in LGESQL, while $3$ points drop in RGATSQL. However, our LGESQL with merely local relations is still competitive. It consolidates our motivation that by exploiting the structure among edges, the line graph can capturing long-range relations to some extent. 2) graph pruning task contributes more in LGESQL ($+1.2\%$) than RGATSQL ($+0.7\%$) on account of the fact that local relations are more critical to structural inference. 3) Two strategies of combining local and non-local relations introduced in § 3.2.1 (w/ MSDE or MMC) are both beneficial to the eventual performances of LGESQL ($2.0\%$ and $2.1\%$ gains, respectively). It corroborates the assumption that local and non-local relations should be treated with distinction. However, the performance remains unchanged in RGATSQL, when merging a different view of the graph (w/ MMC) into multi-head attention. This may be caused by the over-smoothing problem of a complete graph.

3.2 Pre-trained Language Models

In this part, we analyze the effects of different pre-trained language models in Table 5. From the overall results, we can see that: 1) by involving the line graph into computation, LGESQL outperforms the baseline model RGATSQL with different PLMs, further demonstrating the effectiveness of explicitly modeling edge features. 2) large series PLMs consistently perform better than base models on account of their model capacity and generalization capability to unseen domains. 3) Task adaptive PLMs especially Electra are superior to vanilla Bert irrespective of the upper GNN architecture. We hypothesize the reason is that Electra is pre-trained with a tailored binary classification task, which aims to individually distinguish whether each input word is substituted given the context. Essentially, this self-supervised task is similar to our proposed graph pruning task, which focuses on enhancing the discriminative capability of the encoder.

4 Case Studies

In Figure 7, we compare the SQL queries generated by our LGESQL model with those created by the baseline model RGATSQL. We notice that LGESQL performs better than the baseline system, especially on examples that involve the JOIN operation of multiple tables. For instance, in the second case where the connection of three tables are included, RGATSQL fails to identify the existence of table flights. Thus, it is unable to predict the WHERE condition about the destination city and does repeat work. In the third case, our LGESQL still successfully constructs a connected schema sub-graph by linking table “template” to “documents”. Sadly, the RGATSQL model neglects the occurrence of “documents” again. However, in the last case, our LGESQL is stupid to introduce an unnecessary table “airports”. It ignores the situation that table “flights” has one column “source_airport” which already satisfies the requirement.

Related Work

To tackle the joint encoding problem of the question and database schema, Xu et al. (2017) proposes “column attention” strategy to gather information from columns for each question word. TypeSQL Yu et al. (2018a) incorporates prior knowledge of column types and schema linking as additional input features. Bogin et al. (2019a) and Chen et al. (2021) deal with the graph structure of database schema via GNN. EditSQL Zhang et al. (2019b) considers “co-attention” between question words and database schema nodes similar to the common practice in text matching Chen et al. (2017). BRIDGE Lin et al. (2020) further leverages the database content to augment the column representation. The most advanced method RATSQL Wang et al. (2020a), utilizes a complete relational graph attention neural network to handle various pre-defined relations. In this work, we further consider both local and non-local, dynamic and static edge features among different types of nodes with a line graph.

Apart from the structural topology, a heterogeneous graph Shi et al. (2016) also contains multiple types of nodes and edges. To address the heterogeneity of node attributes, Zhang et al. (2019a) designs a type-based content encoder and Fu et al. (2020) utilizes a type-specific linear transformation. For edges, relational graph convolution network (RGCN, Schlichtkrull et al., 2018) and relational graph attention network (RGAT, Wang et al., 2020b) have been proposed to parameterize different relations. HAN Wang et al. (2019b) converts the original heterogeneous graph into multiple homogeneous graphs and applies a hierarchical attention mechanism to the meta-path-based sub-graphs. Similar ideas have been adopted in dialogue state tracking Chen et al. (2020b, 2019a), dialogue policy learning Chen et al. (2018) and text matching Chen et al. (2020c); Lyu et al. (2021) to handle heterogeneous inputs. In another branch, Chen et al. (2019b), Zhu et al. (2019) and Zhao et al. (2020) construct the line graph of the original graph and explicitly model the computation over edge features. In this work, we borrow the idea of a line graph and update both node and edge features via iteration over dual graphs.

Conclusion

In this work, we utilize the line graph to update the edge features in the heterogeneous graph for the text-to-SQL task. Through the iteration over the structural connections in the line graph, local edges can incorporate multi-hop relational features and capture significant meta-paths. By further integrating non-local relations, the encoder can learn from multiple views and attend to remote nodes with shortcuts. In the future, we will investigate more useful meta-paths and explore more effective methods to deal with different meta-path-based neighbors.

Acknowledgments

We thank Tao Yu, Yusen Zhang and Bo Pang for their careful assistance with the evaluation. We also thank the anonymous reviewers for their thoughtful comments. This work has been supported by Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102), No.SKLMCPTS2020003 Project and Startup Fund for Youngman Research at SJTU (SFYR at SJTU).

References

Appendix A Local and Non-Local Relations

In this work, meta-paths with length $1$ are local relations, and other meta-paths are non-local relations. Specifically, Table 6 provides the list of all local relations according to the types of source and target nodes. Notice that we preserve the NoMatch relation because there is no overlapping between the entire question and any schema item in some cases. This relaxation will dramatically increase the number of edges in the line graph. To resolve it, we remove edges in the line graph that the source and target nodes both represent relation types of Match series. In other words, we prevent information propagating between these bipartite connections during the iteration of the line graph.

The checklist in Table 6 is only a subset of all relations defined in RATSQL Wang et al. (2020a). For the remaining relations, we treat them as non-local relations for a fair comparison to the baseline system RATSQL.

Appendix B Details of Text-to-SQL Decoder

The complete grammar used to translate the SQL into a series of actions is provided in Figure 8. Here are some criteria when we design the abstract syntax description language (ASDL, Wang et al., 1997) for the target SQL queries:

Keep the length of the action sequence short to prevent the long-term forgetting problem in the auto-regressive decoder. To achieve this goal, we remove the optional operator “?” defined in Wang et al. (1997) and extend the number of constructors by enumeration. For example, we expand all solutions of type sql_unit according to the existence of different clauses.

Hierarchically, group and re-use the same type in a top-down manner for parameter sharing. For example, we use the same type col_unit when choosing columns in different clauses and create the type val_unit such that both the SELECT clause and CONDITION clauses can refer to it.

When generating a list of items of the same type, instead of emitting a special action Reduce as the symbol of termination Yin and Neubig (2017), we enumerate all possible number of occurrences in the training set (see the constructors for type select and from in Figure 8). Then, we generate each item based on this quantitative limitation. Preliminary experimental results prove that thinking in advance is better than a lazy decision.

Our grammar can cover $98.7\%$ and $98.2\%$ cases in the training and validation dataset, respectively.

B.2 Decoder Architecture

where $\mathbf{v}_{0}$ is a trainable row vector and $\mathbf{W}_{0},\mathbf{W}_{1}$ are parameter matrices. Then, in the structured ON-LSTM decoder, the hidden states at each timestep $j$ is updated as

For ApplyRule action, the probability distribution is computed by a softmax classification layer:

For SelectTable action, we directly copy the table $t_{i}$ from the encoded memory $\mathbf{X}_{t}$.

To be consistent, we also apply the multi-head attention mechanism here with $H=8$ heads. The calculation of SelectColumn action is similar with different network parameters.

Appendix C Graph Pruning

Similar ideas have been proposed by Bogin et al. (2019b) and Yu et al. (2020). Our proposed task differs from their methods in two aspects:

Yu et al. (2020) devises several syntactic roles for schema items and performs multi-class classification instead of binary discrimination. Based on our assumption, the encoder is responsible for the discrimination capability while the decoder organizes different schema items and components into a complete semantic frame. Thus, we simplify the training target into binary labels.

Bogin et al. (2019b) utilizes another RGCN to calculate the relevance score for each schema item in Global-GNNSQL. This score is incorporated into the encoder RGCN as a soft input coefficient. Different from this cascaded method, graph pruning is employed in a multitasking manner. We have tried different approaches to combine this auxiliary module with the primary text-to-SQL model in our preliminary experiments, such as:

1) Similar to Bogin et al. (2019b), we utilize a separate graph encoder to conduct graph pruning firstly, and use another refined graph encoder (the same architecture, e.g., RGAT) to jointly encode the pruned schema graph and the question. These two encoders can share network parameters of only the embeddings or more upper GNN layers. If they share all $8$ layers, the entire encoder will degenerate from the pipelined mode into our multitasking fashion. Empirical results in Table 7 demonstrate that when these two encoders share more layers, the performance of the text-to-SQL model is better.

2) We can constrain the text-to-SQL decoder to only attend and retrieve schema items from the pruned encoded memory when calculating attention vectors and select columns or tables. In other words, the graph pruning module and the text-to-SQL decoder are connected in a cascaded way. Through pilot experiments, we observe the flagrant training-inference inconsistency problem. The text-to-SQL decoder is trained upon the golden schema items, but it depends on the predicted options from the graph pruning module during evaluation. Even if we endeavor various sampling-based methods (such as random sampling, sampling from current module predictions, or sampling from neighboring nodes of the golden schema graph) to inject some noise during training, the performance is merely competitive to that with multitasking. Therefore, based on Occam’s Razor Theorem, we only treat graph pruning as an auxiliary output module.