Simple BERT Models for Relation Extraction and Semantic Role Labeling

Introduction

Relation extraction and semantic role labeling (SRL) are two fundamental tasks in natural language understanding. The task of relation extraction is to discern whether a relation exists between two entities in a sentence. For example, in the sentence “Obama was born in Honolulu”, “Obama” is the subject entity and “Honolulu” is the object entity. The task of a relation extraction model is to identify the relation between the entities, which is per:city_of_birth (birth city for a person). For SRL, the task is to extract the predicate–argument structure of a sentence, determining “who did what to whom”, “when”, “where”, etc. Both capabilities are useful in several downstream tasks such as question answering Shen and Lapata (2007) and open information extraction Fader et al. (2011).

State-of-the-art neural models for both tasks typically rely on lexical and syntactic features, such as part-of-speech tags Marcheggiani et al. (2017), syntactic trees Roth and Lapata (2016); Zhang et al. (2018); Li et al. (2018), and global decoding constraints Li et al. (2019). In particular, Roth and Lapata (2016) argue that syntactic features are necessary to achieve competitive performance in dependency-based SRL. Zhang et al. (2018) also showed that dependency tree features can further improve relation extraction performance. Although syntactic features are no doubt helpful, a known challenge is that parsers are not available for every language, and even when available, they may not be sufficiently robust, especially for out-of-domain text, which may even hurt performance He et al. (2017).

Recently, the NLP community has seen excitement around neural models that make heavy use of pretraining based on language modeling Peters et al. (2018); Radford et al. (2018). The latest development is BERT Devlin et al. (2018), which has shown impressive gains in a wide variety of natural language tasks ranging from sentence classification to sequence labeling. A natural question follows: can we leverage these pretrained models to further push the state of the art in relation extraction and semantic role labeling, without relying on lexical or syntactic features? The answer is yes. We show that simple neural architectures built on top of BERT yields state-of-the-art performance on a variety of benchmark datasets for these two tasks. The remainder of this paper describes our models and experimental results for relation extraction and semantic role labeling in turn.

BERT for Relation Extraction

For relation extraction, the task is to predict the relation between two entities, given a sentence and two non-overlapping entity spans. In order to encode the sentence in an entity-aware manner, we propose the BERT-based model shown in Figure 1. First, we construct the input sequence [[cls] sentence [sep] subject [sep] object [sep]]. To prevent overfitting, we replace the entity mentions in the sentence with masks, comprised of argument type (subject or object) and entity type (such as location and person), e.g., Subj-Loc, denoting that the subject entity is a location.

The input is then tokenized by the WordPiece tokenizer Sennrich et al. (2016) and fed into the BERT encoder. After obtaining the contextual representation, we discard the sequence after the first [sep] for the following operations.

We use $\mathcal{H}=[h_{0},h_{1},...,h_{n},h_{n+1}]$ to denote the BERT contextual representation for [[cls] sentence [sep]]. Note that $n$ can be different from the length of the sentence because the tokenizer might split words into sub-tokens. The subject entity span is denoted $\mathcal{H}_{s}=[h_{s_{1}},h_{s_{1}+1},...,h_{s_{2}}]$ and similarly the object entity span is $\mathcal{H}_{o}=[h_{o_{1}},h_{o_{1}+1},...,h_{o_{2}}]$. Following Zhang et al. (2017), we define a position sequence relative to the subject entity span $[p_{0}^{s},...,p_{n+1}^{s}]$, where

2 Experiments

We evaluate our model on the TAC Relation Extraction Dataset (TACRED) Zhang et al. (2017), a standard benchmark dataset for relation extraction. In our experiments, the hidden sizes of the LSTM and MLP are 768 and 300, respectively, and the position embedding size is 20. The learning rate is $5\times 10^{-5}$. The BERT base-cased model is used in our experiments. Embeddings for the masks (e.g., Subj-Loc) are randomly initialized and fine-tuned during the training process, as well as the position embeddings.

Results on the TACRED test set are shown in Table 1. Our model outperforms the works of Zhang et al. (2018) and Wu et al. (2019), which use GCNs Kipf and Welling (2016) and variants to encode syntactic tree information as external features. Alt et al. (2019) leverage the pretrained language model GPT Radford et al. (2018) and achieves better recall than our system. In terms of $F_{1}$, our system obtains the best known score among individual models, but our score is still below that of the interpolation model of Zhang et al. (2018) because of lower recall.

BERT for Semantic Role Labeling

The standard formulation of semantic role labeling decomposes into four subtasks: predicate detection, predicate sense disambiguation, argument identification, and argument classification. There are two representations for argument annotation: span-based and dependency-based. Semantic banks such as PropBank usually represent arguments as syntactic constituents (spans), whereas the CoNLL 2008 and 2009 shared tasks propose dependency-based SRL, where the goal is to identify the syntactic heads of arguments rather than the entire span. Here, we follow Li et al. (2019) to unify these two annotation schemes into one framework, without any declarative constraints for decoding. For several SRL benchmarks, such as CoNLL 2005, 2009, and 2012, the predicate is given during both training and testing. Thus, in this paper, we only discuss predicate disambiguation and argument identification and classification.

Predicate sense disambiguation. The predicate disambiguation task is to identify the correct meaning of a predicate in a given context. As an example, for the sentence “Barack Obama went to Paris”, the predicate went has sense “motion” and has sense label 01.

We formulate this task as sequence labeling. The input sentence is fed into the WordPiece tokenizer, which splits some words into sub-tokens. The predicate token is tagged with the sense label. Following the original BERT paper, two labels are used for the remaining tokens: ‘O’ for the first (sub-)token of any word and ‘X’ for any remaining fragments. We feed the sequences into the BERT encoder to obtain the contextual representation $\mathcal{H}$. A “predicate indicator” embedding is then concatenated to the contextual representation to distinguish the predicate tokens from non-predicate ones. The final prediction is made using a one-hidden-layer MLP over the label set.

Argument identification and classification. This task is to detect the argument spans or argument syntactic heads and assign them the correct semantic role labels. In the above example, “Barack Obama” is the Arg1 of the predicate went, meaning the entity in motion.

Formally, our task is to predict a sequence $\boldsymbol{z}$ given a sentence–predicate pair ($\mathcal{X}$, $v$) as input, where the label set draws from the cross of the standard BIO tagging scheme and the arguments of the predicate (e.g., B-Arg1). The model architecture is illustrated in Figure 2, at the point in the inference process where it is outputting a tag for the token “Barack”. In order to encode the sentence in a predicate-aware manner, we design the input as [[cls] sentence [sep] predicate [sep]], allowing the representation of the predicate to interact with the entire sentence via appropriate attention mechanisms. The input sequence as described above is fed into the BERT encoder. The contextual representation of the sentence ([cls] sentence [sep]) from BERT is then concatenated to predicate indicator embeddings, followed by a one-layer BiLSTM to obtain hidden states $\mathcal{G}=[g_{1},g_{2},...,g_{n}]$. For the final prediction on each token $g_{i}$, the hidden state of predicate $g_{p}$ is concatenated to the hidden state of the token $g_{i}$, and then fed into a one-hidden-layer MLP classifier over the label set.

2 Experimental Setup

We conduct experiments on two SRL tasks: span-based and dependency-based. For span-based SRL, the CoNLL 2005 Carreras and Màrquez (2004) and 2012 Pradhan et al. (2013) datasets are used. For dependency-based SRL, the CoNLL 2009 Hajič et al. (2009) dataset is used. We follow standard splits for the training, development, and test sets.

In our experiments, the hidden sizes of the LSTM and MLP are 768 and 300, respectively, and the predicate indicator embedding size is 10. The learning rate is $5\times 10^{-5}$. BERT base-cased and large-cased models are used in our experiments. The position embeddings are randomly initialized and fine-tuned during the training process.

3 Dependency-Based SRL Results

Predicate sense disambiguation. The predicate sense disambiguation subtask applies only to the CoNLL 2009 benchmark. In this line of research on dependency-based SRL, previous papers seldom report the accuracy of predicate disambiguation separately (results are often mixed with argument identification and classification), causing difficulty in determining the source of gains. Here, we report predicate disambiguation accuracy in Table 2 for the development set, test set, and the out-of-domain test set (Brown). The state-of-the-art model He et al. (2018b) is based on a BiLSTM and linguistic features such as POS tag embeddings and lemma embeddings. Instead of using linguistic features, our simple MLP model achieves better accuracy with the help of powerful contextual embeddings. These predicate sense disambiguation results are used in the dependency-based SRL end-to-end evaluation.

Argument identification and classification. We provide SRL performance excluding predicate sense disambiguation to validate the source of improvements: results are shown in Table 3. Figures from some systems are missing because they only report end-to-end results.

Our end-to-end results are shown in Table 4. We see that the BERT-LSTM-large model (using the predicate sense disambiguation results from above) yields large $F_{1}$ score improvements over the existing state of the art Li et al. (2019), and beats existing ensemble models as well. This is achieved without using any linguistic features and declarative decoding constraints.

4 Span-Based SRL Results

Our span-based SRL results are shown in Table 5. We see that the BERT-LSTM-large model achieves the state-of-the-art $F_{1}$ score among single models and outperforms the Ouchi et al. (2018) ensemble model on the CoNLL 2005 in-domain and out-of-domain tests. However, it falls short on the CoNLL 2012 benchmark because the model of Ouchi et al. (2018) obtains very high precision. They are able to achieve this with a more complex decoding layer, with human-designed constraints such as the “Overlap Constraint” and “Number Constraint”.

Conclusions

Based on this preliminary study, we show that BERT can be adapted to relation extraction and semantic role labeling without syntactic features and human-designed constraints. While we concede that our model is quite simple, we argue this is a feature, as the power of BERT is able to simplify neural architectures tailored to specific tasks. Nevertheless, these results provide strong baselines and foundations for future research. Many natural follow-up questions emerge: Can syntactic features be re-introduced to further improve results? Can multitask learning be used to simultaneously benefit relation extraction and semantic role labeling? We are actively working on answering these and additional questions.

Acknowledgments

This research was supported by the Natural Sciences and Engineering Research Council (NSERC) of Canada.

References