A Full End-to-End Semantic Role Labeler, Syntax-agnostic Over Syntax-aware?

Introduction

Semantic role labeling (SRL) is a shallow semantic parsing, which is dedicated to identifying the semantic arguments of a predicate and labeling them with their semantic roles. SRL is considered as one of the core tasks in the natural language processing (NLP), which has been successfully applied to various downstream tasks, such as information extraction [Bastianelli et al., 2013], question answering [Shen and Lapata, 2007, Berant et al., 2013], machine translation [Xiong et al., 2012, Shi et al., 2016].

Typically, SRL task can be put into two categories: constituent-based (i.e., phrase or span) SRL and dependency-based SRL. This paper will focus on the latter one popularized by CoNLL-2008 and 2009 shared tasks [Surdeanu et al., 2008, Hajič et al., 2009]. Most conventional SRL systems relied on sophisticated handcraft features or some declarative constraints [Pradhan et al., 2005, Zhao et al., 2009a], which suffers from poor efficiency and generalization ability. A recently tendency for SRL is adopting neural networks methods attributed to their significant success in a wide range of applications [Bai and Zhao, 2018, Zhang and Zhao, 2018]. However, most of those works still heavily resort to syntactic features. Since the syntactic parsing task is equally hard as SRL and comes with its own errors, it is better to get rid of such prerequisite as in other NLP tasks. Accordingly, ?) presented a neural model putting syntax aside for dependency-based SRL and obtain favorable results, which overturns the inherent belief that syntax is indispensable in SRL task [Punyakanok et al., 2008].

Besides, SRL task is generally formulated as multi-step classification subtasks in pipeline systems, consisting of predicate identification, predicate disambiguation, argument identification and argument classification. Most previous SRL approaches adopt a pipeline framework to handle these subtasks one after another. Until recently, some works [Zhou and Xu, 2015, He et al., 2017] introduce end-to-end models for span-based SRL, which motivates us to explore integrative model for dependency SRL.

In this work, we propose a syntactic-agnostic end-to-end system, dealing with predicate disambiguation and argument labeling in one model, unlike previous systems that treat the predicate disambiguation as a subtask and handle it separately. In detail, our model contains (1) a deep BiLSTM encoder, which is able to distinguish the predicates and arguments by mapping them into two different vector spaces, and (2) a biaffine attentional [Dozat and Manning, 2017] scorer, which unifiedly predicts the semantic role for argument and the sense for predicate.

We experimentally show that though our biaffine attentional model remains simple and does not rely on any syntactic feature, it achieves the best result on the benchmark for both Chinese and English even compared to syntax-aware systems. In summary, our major contributions are shown as follows:

We propose an accurate syntax-agnostic model for neural SRL, which outperforms the best reported syntax-aware model, breaking the long-held belief that syntax is a prerequisite for SRL.

Our model gives state-of-the-art results on the CoNLL-2008, CoNLL-2009 English and Chinese benchmarks, scoring 85.0% F1, 89.6% F1 and 84.4% F1, respectively.

Our work is the first attempt to apply end-to-end model for dependency-based SRL, which tackles the predicate disambiguation and the argument labeling subtasks in one shot.

Semantic Structure Decomposition

SRL includes two subtasks: predicate identification/disambiguation and argument identification/labeling. Since the CoNLL-2009 dataset provides the gold predicates, most previous neural SRL systems use a default model to perform predicate disambiguation and focus on argument identification/labeling. Despite nearly all SRL work adopted the pipeline model with two or more components, ?) and ?) presented an end-to-end solution for the entire SRL task with a word pair classifier. Following the same formulization, we propose the first neural SRL system that uniformly handles the tasks of predicate disambiguation and argument identification/labeling.

In semantic dependency parsing, we can always identify two types of words, semantic head (predicate) and semantic dependent (argument). To build the needed predicate-argument structure, the model only needs to predict the role of any word pair from the given sentence. For the purpose, an additional role label None and virtual root node $<$VR$>$ are introduced. The None label indicates that there is no semantic role relationship inside the word pair. We insert a virtual root $<$VR$>$ in the head of the sentence, and set it as the semantic head of all the predicates. By introducing the None label and the $<$VR$>$ node, we construct a semantic tree rooted at the $<$VR$>$ node with several virtual arcs labeled with None. Thus, the predicate disambiguation and argument identification/labeling tasks can be naturally regarded as the labeling process over all the word pairs. Figure 1 shows an example of the semantic graph augmented with a virtual root and virtual arc, and Table 1 lists all the corresponding word pair examples, in which two types of word pairs are included, $<$VR$>$ followed by predicate candidatesNote that there is a key difference between CoNLL 2008 and 2009 shared task for English, the latter has specified the predicate in the data so that here we have only one sample for $<$VR$>$-predicate pair to make predicate disambiguation. For the former, predicate candidates should be every words in the given sentence. More details are seen in Section 4.4. and a known predicate collocated with every words in the sentence as argument candidates. Note that since the nominal predicate sometimes takes itself as its argument, the predicate itself is also included in the argument candidate list.

Model

Our model contains two main components: (1) a deep BiLSTM encoder that takes each word embedding $\bm{e}$ of the given sentence as input and generates dense vectors for both words in the to-be-classified word pair respectively, (2) a biaffine attentional scorer which takes the hidden vectors for the given word pair as input and predict a label score vector. Figure 2 provides an overview of our model.

The word representation of our model is the concatenation of several vectors: a randomly initialized word embedding $\bm{e}^{(r)}$, a pre-trained word embedding $\bm{e}^{(p)}$, a randomly initialized part-of-speech (POS) tag embedding $\bm{e}^{(pos)}$, a randomly initialized lemma embedding $\bm{e}^{(l)}$. Besides, since previous work [He et al., 2018] demonstrated that the predicate-specific feature is helpful in promoting the role labeling process, we employ an indicator embedding $\bm{e}^{(i)}$ to indicate whether a word is a predicate when predicting and labeling the arguments for each given predicate. The final word representation is given by $\bm{e}=\bm{e}^{(r)}\oplus\bm{e}^{(p)}\oplus\bm{e}^{(l)}\oplus\bm{e}^{(pos)}\oplus\bm{e}^{(i)}$, where $\oplus$ is the concatenation operator.

As commonly used to model the sequential input in most NLP tasks [Wang et al., 2016, He et al., 2018], BiLSTM is adopted for our sentence encoder. By incorporating a stack of two distinct LSTMs, BiLSTM processes an input sequence in both forward and backward directions. In this way, the BiLSTM encoder provides the ability to incorporate the contextual information for each word.

Given a sequence of word representation $S=\{\bm{e}_{1},\bm{e}_{2},\cdots,\bm{e}_{N}\}$ as input, the $i$-th hidden state $\bm{g}_{i}$ is encoded as follows:

where $LSTM^{\mathcal{F}}$ denotes the forward LSTM transformation and $LSTM^{\mathcal{B}}$ denotes the backward LSTM transformation. $\bm{g}^{f}_{i}$ and $\bm{g}^{b}_{i}$ are the hidden state vectors of the forward LSTM and backward LSTM respectively.

2 Biaffine Attentional Role Scorer

Typically, to predict and label arguments for a given predicate, a role classifier is employed on top of the BiLSTM encoder. Some work like [Marcheggiani et al., 2017] shows that incorporating the predicate’s hidden state in their role classifier enhances the model performance, while we argue that a more natural way to incorporate the syntactic information carried by the predicate is to employ the attentional mechanism. Our model adopts the recently introduced biaffine attention [Dozat and Manning, 2017] to enhance our role scorer. Biaffine attention is a natural extension of bilinear attention [Luong et al., 2015] which is widely used in neural machine translation (NMT).

Usually, a BiLSTM decoder takes the concatenation $\bm{g}_{i}$ of the hidden state vectors as output for each hidden state. However, in the SRL context, the encoder is supposed to distinguish the currently considered predicate from its candidate arguments. To this end, we perform two distinct affine transformations with a nonlinear activation on the hidden state $\bm{g}_{i}$, mapping it to vectors with smaller dimensionality:

where $ReLU$ is the rectilinear activation function, $\bm{h}_{i}^{(pred)}$ is the hidden representation for the predicate and $\bm{h}_{i}^{(arg)}$ is the hidden representation for the candidate arguments.

By performing such transformations over the encoder output to feed the scorer, the latter may benefit from deeper feature extraction. First, ideally, instead of keeping both features learned by the two distinct LSTMs, the scorer is now enabled to learn features composed from both recurrent states together with reduced dimensionality. Second, it provides the ability to map the predicates and the arguments into two distinct vector spaces, which is essential for our tasks since some words can be labeled as predicate and argument simultaneously. Mapping a word into two different vectors can help the model disambiguate the role that it plays in different context.

In the standard NMT context, given a target recurrent output vector $h_{i}^{(t)}$ and a source recurrent output vector $h_{j}^{(s)}$, a bilinear transformation calculates a score $s_{ij}$ for the alignment:

However, considering that in a traditional classification task, the distribution of classes is often uneven, and that the output layer of the model normally includes a bias term designed to capture the prior probability $P(y_{i}=c)$ of each class, with the rest of the model focusing on learning the likelihood of each class given the data $P(y_{i}=c|x_{i})$, [Dozat and Manning, 2017] introduced the bias terms into the bilinear attention to address such uneven problem, resulting in a biaffine transformation. The biaffine transformation is a natural extension of the bilinear transformation and the affine transformation. In SRL task, the distribution of the role labels is similarly uneven and the problem comes worse after we introduce the additional $<$VR$>$ node and None label, directly applying the primitive form of bilinear attention would fail to capture the prior probability $P(y_{i}=c_{k})$ for each class. Thus, the biaffine attention introduced in our model would be extremely helpful for semantic role prediction.

It is worth noting that in our model, the scorer aims to assign a score for each specific semantic role. Besides learning the prior distribution for each label, we wish to further capture the preferences for the label that a specific predicate-argument pair can take. Thus, our biaffine attention contains two distinguish bias terms:

where $\bm{W}^{(role)}$, $\bm{U}^{(role)}$ and $\bm{b}^{(role)}$ are parameters that will be updated by some gradient descent methods in the learning process. There are several points that should be paid attention to in the above biaffine transformation. First, since our goal is to predict the label for each pair of $\bm{h}_{i}^{(arg)}$, $\bm{h}_{j}^{(pred)}$, the output of our biaffine transformation should be a vector of dimensionality $N_{r}$ instead of a real value, where $N_{r}$ is the number of all the candidate semantic labels. Thus, the bilinear transformation in Eq. (1) maps two input vectors into another vector. This can be accomplished by setting $\bm{W}^{(role)}$ as a $(d_{h}\times N_{r}\times d_{h})$ matrix, where $d_{h}$ is the dimensionality of the hidden state vector. Similarly, the output of the linear transformation in Eq. (2) is also a vector by setting $\bm{U}^{(role)}$ as a $(N_{r}\times 2d_{h})$ matrix. Second, Eq. (2) captures the preference of each role (or sense) label condition on taking the $j$-th word as predicate and the $i$-th word as argument. Third, the last term $\bm{b}^{(role)}$ captures the prior probability of each class $P(y_{i}=c_{k})$. Notice that Eq. (2) and (3) capture different kinds of bias for the latent distribution of the label set.

Given a sentence of length $L$ (including the $<$VR$>$ node), for one of its predicates $w_{j}$, the scorer outputs a score vector $\{\bm{s}_{1j},\bm{s}_{2j},\cdots,\bm{s}_{Lj}\}$. Then our model picks as its output the label with the highest score from each score vector: $y_{ij}=\mathop{\arg\max}_{1\leq k\leq N_{r}}(\bm{s}_{ij}[k])$, where $\bm{s}_{ij}[k]$ denotes the score of the $k$-th candidate semantic label.

Experiments

We evaluate our modelThe code is available at https://github.com/JiaxunCai/Dynet-Biaffine-SRL on English and Chinese CoNLL-2009 datasets with the standard split into training, test and development sets. The pre-trained embedding for English is trained on Wikipedia and Gigaword using the GloVe [Pennington et al., 2014], while those for Chinese is trained on Wikipedia. Our implementation uses the DyNethttps://github.com/clab/dynet library for building the dynamic computation graph of the network.

When not otherwise specified, our model uses: $100$-dimensional word, lemma, pre-trained and POS tag embeddings and $16$-dimensional predicate-specific indicator embedding; and a $20\%$ chance of dropping on the whole word representation; $3$-layer BiLSTMs with $400$-dimensional forward and backward LSTMs, using the form of recurrent dropout suggested by [Gal and Ghahramani, 2016] with an $80\%$ keep probability between time-steps and layers; two $300$-dimensional affine transformation with the ReLU non-linear activation on the output of BiLSTM, also with an $80\%$ keep probability.

The parameters in our model are optimized with Adam [Kingma and Ba, 2015], which keeps a moving average of the L2 norm of the gradient for each parameter throughout training and divides the gradient for each parameter by this moving average, ensuring that the magnitude of the gradients will on average be close to one. For the parameters of optimizer, we follow the settings in [Dozat and Manning, 2017], with $\beta_{1}=\beta_{2}=0.9$ and learning rate $0.002$, annealed continuously at a rate of $0.75$ every $5,000$ iterations, with batches of approximately $5,000$ tokens. The maximum number of epochs of training is set to $50$.

2 Results

Tables 2 and 4 report the comparison of performance between our model and previous dependency-based SRL model on both English and Chinese. Note that the predicate disambiguation subtask is unifiedly tackled with arguments labeling in our model with precisions of 95.0% and 95.6% respectively on English and Chinese test sets in our experimentsNote that we give comparable predicate disambiguation results with ?), with 95.01% and 95.58% F1 on development and test sets, respectively.. The proposed model accordingly outperforms all the SRL systems so far on both languages, even including those syntax-aware and ensemble ones. The improvement grows even larger when comparing only with the single syntax-agnostic models.

For English, our syntax-agnostic model even slightly outperforms the best reported syntax-aware model [He et al., 2018] with a margin of 0.1% F1. Compared to syntax-agnostic models, our model overwhelmingly outperforms (with an improvement of 0.9% F1) the previous work [He et al., 2018].

Although we used the same parameters as for English, our model substantially outperforms the state-of-art models on Chinese, demonstrating that our model is robust and less sensitive to the parameter selection. For Chinese, the proposed model outperforms the best previous model [He et al., 2018] with a considerable improvement of 1.5% F1, and surpasses the best single syntax-agnostic model [He et al., 2018] with a margin of 2.5% F1.

Table 4 compares the results on English out-of-the-domain (Brown) test set, from which our model still remains strong. The proposed model gives a comparable result with the highest score from syntax-aware model of [He et al., 2018], which affirms that our model does well learn and generalize the latent semantic preference of the data.

Results on both in-domain and out-of-the-domain test sets demonstrate the effectiveness and the robustness of the proposed model structure—the non-linear transformation after the BiLSTM serves to distinguish the predicate from argument while the biaffine attention tells what to attend for each candidate argument. In Section 4.3.2, we will get an insight into our model and explore how each individual component impacts the model performance.

3 Ablation Analysis

To learn how the input word representation choice impacts our model performance, we conduct an ablation study on the English test set whose results are shown in Table 5. Since we deal with the two subtasks in a single model, the choice of word representation will simultaneously influence the results of both of them. Besides the results of argument labeling, we also report the precision of predicate disambiguation.

The results demonstrate that the multiple dimensional indicator embedding proposed by (He et al., 2018) contributes the most to the final performance of our model. It is consistent with the conclusion in [Marcheggiani et al., 2017] which argue that encoding predicate information promotes the SRL model. It is interesting that the impact of POS tag embedding (about 0.3% F1) is less compared to the previous works, which possibly allows us to build an accuracy model even when the POS tag label is unavailable.

3.2 Into the Model

In this section, we get insight into the proposed model, exploring how the deep BiLSTM encoder and the biaffine attention affect the labeling results respectively. Specifically, we present two groups of results on the CoNLL-2009 English test set. 1) Shallow biaffine attentive (SBA) labeler. Instead of mapping the output of the BiLSTM into two distinct vector spaces, we apply a single non-linear affine transformation on the output. The single transformation just serves to reduce the dimensionality and does not differ the predicates from the arguments. 2) Deep bilinear attentive (DBA) labeler. We apply the primitive form of bilinear attention in the scorer by removing the two bias terms of the biaffine transformation. By this means, we learn to what extent can the bias terms fit the prior distribution of the data. Results of the two experiments are shown in Table 7.

The results show that the bias terms in biaffine attention play an important role in promoting the model performance. Removal of the bias terms dramatically declines the performance by 1.7% F1. Thus we can draw a conclusion that the bias term does well in fitting the prior distribution and global preference of the data. The bilinear attentional model behaves more poorly since it struggles to learn the likelihood of each class on an uneven data set without knowledge about the prior distribution. Though the deep encoder contributes less to the performance, it also brings an improvement of 0.5% F1. Note that the only difference of SBA-labeler of our standard model is whether the hidden representations of the arguments and the predicates lay in different vector spaces. Such a result confirms that distinguishing the predicates from the arguments in encoding process indeed enhances the model to some extent.

3.3 Syntax-aware and Syntax-agnostic

Noting that the work [Zhao and Kit, 2008] and [Zhao et al., 2013] are similar to ours in modeling the dependency-based SRL tasks as word pair classification, and that they successfully incorporate the syntactic information by applying argument candidate pruning, we further perform empirical study to explore whether employing such pruning method enhance or hinder our model. Specifically, we use the automatically predicted parse with moderate performance provided by CoNLL-2009 shared task, with the LAS score about 86%.

The pruning method is supposed to work since it can alleviate the imbalanced label distribution caused by introducing the None label. However, as shown in Table 7, the result is far from satisfying. The main reason might be the pruning algorithm is so strict that too many true arguments are falsely pruned. To address this problem, ?) introduced an extended $k$-order argument pruning algorithm. Figure 4 shows the curves of coverage and reduction rate against the pruning order $k$ on the English training set following [He et al., 2018]. Following this work, we further perform different orders of pruning and obtain the F1 scores curve shown in Figure 4. However, the $k$-order pruning does not boost the performance of our model. Table 8 presents the performance gap between syntax-agnostic and syntax-aware settings of the same models. Unlike the other two works, the introduction of syntax information fails to bring about bonus for our model. Nevertheless, it is worth noting that even when running without the syntax information, our mode still show a promising result compared to the other syntax-aware models.

4 CoNLL 2008: Augment the Model with Predicate Identification

Though CoNLL-2009 provided the gold predicate beforehand, the predicate identification subtask is still indispensable for a real world SRL task. Thus, we further augment our model with the predicate identification ability.

Specifically, we first attach all the words in the sentence to the virtual root $<$VR$>$ and label the word which is not a predicate with the None role label. It should be noting that, in CoNLL-2009 settings, we just attach the predicates to the virtual root, since we do not need to distinguish the predicate from other word. The training scheme still keeps the same as that in CoNLL-2009 settings, while in testing phase, an additional procedure is performed to find out all the predicates of a given sentence.

First, our model is fed the representations of the virtual root and each word of the input sentence, identifying and disambiguating all the predicates of the sentence. Second, it picks each predicate predicted by the model with each word of the sentence to identify and label the semantic role in between, which remains the same as the model does on CoNLL-2009. The second phase is repeated until all the predicates have got its arguments being identified and labeled. We evaluate our model on CoNLL-2008 benchmark using the same hyperparameters settings mentioned in Section 4.1 except that we remove the predicate-specific indicator feature.

The F1 scores on predicates identification and labeling of our model is 89.43%, which remain comparable with the most recent work [He et al., 2018] (90.53% F1). As shown in Table 9, though tackling all the subtasks of CoNLL-2008 SRL unifiedly in a full end-to-end manner, our model outperforms the best reported results with a large margin of about 1.7% semantic F1.

Related Work

Semantic role labeling was pioneered by ?). Most traditional SRL models heavily rely on complex feature engineering [Pradhan et al., 2005, Zhao et al., 2009a, Björkelund et al., 2009]. Among those early works, ?) combined features derived from different syntactic parses based on SVM classifier, while ?) exploited the abundant set of language-specific features that were carefully designed for SRL task.

In recent years, applying neural networks in SRL task has gained a lot of attention due to the impressive success of deep neural networks in various NLP tasks [Zhang et al., 2016, Cai et al., 2017, Qin et al., 2017, Cai and Zhao, 2017]. ?) initially introduced neural networks into the SRL task. They developed a feed-forward network that employed a convolutional network as sentence encoder and a conditional random field as a role classifier. ?) extended their model to further use syntactic information by including binary indicator features. ?) exploited a neural network to unifiedly embed arguments and semantic roles, similar to the work [Lei et al., 2015] which induced a compact feature representation applying tensor-based approach. ?) introduced the dependency path embedding to incorporate syntax and exhibited a notable success, while ?) employed the graph convolutional network to integrate syntactic information into their neural model.

Besides the above-mentioned works who relied on syntactic information, several works attempted to build SRL systems without or with little syntactic information. ?) came up with an end-to-end model for span-based SRL and obtained surprising performance putting syntax aside. ?) further extended their work with the highway network. Simultaneously, ?) proposed a syntax-agnostic model with effective word representation for dependency-based SRL.

However, almost all of previous works treated the predicate disambiguation as individual subtasks, apart from [Zhao and Kit, 2008, Zhao et al., 2009a, Zhao et al., 2009c, Zhao et al., 2013], who presented the first end-to-end system for dependency SRL. For the neural models of dependency SRL, we have presented the first end-to-end solution that handles both semantic labeling subtasks in one single model. At the same time, our model enjoys the advantage that does not rely any syntactic information.

This work is also closely related to the attentional mechanism. The traditional attention mechanism was proposed by ?) in the NMT literature. Following the work [Luong et al., 2015] that encouraged substituting the MLP in the attentional mechanism with a single bilinear transformation, ?) introduced the bias terms into the primitive form of bilinear attention and applied it for dependency parsing. They demonstrate that the bias terms help their model to capture the uneven prior distribution of the data, which is again verified by our practice on SRL in this paper.

Different from the latest strong syntax-agnostic models in [Marcheggiani and Titov, 2017] and [He et al., 2018] which both adopted sequence labeling formulization for the SRL task, this work adopts word pair classification scheme implemented by LSTM encoder and biaffine scorer. Compared to the previous state-of-the-art syntax-agnostic model in [He et al., 2018] whose performance boosting (more than 1% absolute gain) is mostly due to introducing the enhanced representation, namely, the CNN-BiLSTM character embedding from [Peters et al., 2018], our performance promotion mainly roots from model architecture improvement, which results in quite different syntax-aware enhanced impacts. Using the same latest syntax-aware $k$-order pruning, the syntax-agnostic backbone in [He et al., 2018] may receive about 1% performance gain, while our model is furthermore enhanced little. This comparison also suggests the possibility that maybe our model can be further improved by incorporating with the same character embedding as [He et al., 2018] doesSuch an attempt may be hindered by too luxurious computational resource requirement, as there comes extremely high graphic memory prerequisite when integrating both biaffine scorer and the ELMo character embedding..

Conclusion and Future Work

This paper presents a full end-to-end neural model for dependency-based SRL. It is the first time that a SRL model shows its ability to unifiedly handle the predicate disambiguation and the argument labeling subtasks. Our model is effective while remains simple. Experiments show that it achieves the best scores on CoNLL benchmark both for English and Chinese, outperforming the previous state-of-the-art models even with syntax-aware features. Our further investigation by incorporating the latest syntax-aware pruning algorithm shows that the proposed model is insensitive to the input syntactic information, demonstrating an interesting performance style for the SRL task. Of course, we cannot exclude the possibility that the proposed model can be furthermore improved by other syntactic information integration ways, which is left for the future work.

References