Joint Constrained Learning for Event-Event Relation Extraction

Introduction

Human languages evolve to communicate about real-world events. Therefore, understanding events plays a critical role in natural language understanding (NLU). A key challenge to this mission lies in the fact that events are not just simple, standalone predicates. Rather, they are often described at different granularities and may form complex structures. Consider the example in Figure 1, where the description of a storm (e1) involves more fine-grained event mentions about people killed (e2), flights canceled (e3) and passengers affected (e4). Some of those mentions also follow strict temporal order (e3, e4 and e5). Our goal is to induce such an event complex that recognizes the membership of multi-granular events described in the text, as well as their temporal order. This is not only at the core of text understanding, but is also beneficial to various applications such as question answering Khashabi et al. (2018), narrative prediction Chaturvedi et al. (2017), timeline construction Do et al. (2012a) and summarization Daumé III and Marcu (2006).

Recently, significant research effort has been devoted to several event-event relation extraction tasks, such as event temporal relation (TempRel) extraction Ning et al. (2018a, 2019) and subevent relation extraction Liu et al. (2018); Aldawsari and Finlayson (2019). Addressing such challenging tasks requires a model to recognize the inherent connection between event mentions as well as their contexts in the documents. Accordingly, a few previous methods apply statistical learning methods to characterize the grounded events in the documents Glavaš et al. (2014); Ning et al. (2017b, 2018c). Such methods often require designing various features to characterize the structural, discourse and narrative aspects of the events, which are costly to produce and are often specific to a certain task or dataset. More recent works attempted to use data-driven methods based on neural relation extraction models Dligach et al. (2017); Ning et al. (2019); Han et al. (2019a, b) which refrain from feature engineering and offer competent performances.

While data-driven methods provide a general and tractable way for event-event relation extraction, their performance is restricted by the limited annotated resources available Glavaš et al. (2014); Ning et al. (2018b). For example, the largest temporal relation extraction dataset MATRES Ning et al. (2018b) only has 275 articles, which is far from enough for training a well-performing supervised model. The observation that relations and, in particular, event-event relations should be constrained by their logical properties Roth and Yih (2004); Chambers and Jurafsky (2008), led to employing global inference to comply with transitivity and symmetry consistency, specifically on TempRel Do et al. (2012b); Ning et al. (2017b); Han et al. (2019a). However, in an event complex, the logical constraints may globally apply to different task-specific relations, and form more complex conjunctive constraints. Consider the example in Figure 1: given that e2:died is Before e3:canceled and e3:canceled is a Parent event of e4:affecting, the learning process should enforce e2:died Before e4:affecting by considering the conjunctive constraints on both TempRel and subevent relations. While previous works focus on preserving logical consistency through (post-learning) inference or structured learning Ning et al. (2017a), there was no effective way to endow neural models with the sense of global logical consistency during training. This is key to bridging the learning processes of TempRel and subevent relations, which is a research focus of this paper.

The first contribution of this work is proposing a joint constrained learning model for multifaceted event-event relation extraction. The joint constrained learning framework seeks to regularize the model towards consistency with the logical constraints across both temporal and subevent relations, for which three types of consistency requirements are considered: annotation consistency, symmetry consistency and conjunction consistency. Such consistency requirements comprehensively define the interdependencies among those relations, essentially unifying the ordered nature of time and the topological nature of multi-granular subevents based on a set of declarative logic rules. Motivated by the logic-driven framework proposed by Li et al. (2019), the declarative logical constraints are converted into differentiable functions that can be incorporated into the learning objective for relation extraction tasks. Enforcing logical constraints across temporal and subevent relations is also a natural way to combine the supervision signals coming from two different datasets, one for each of the relation extraction tasks with a shared learning objective. Despite the scarce annotation for both tasks, the proposed method surpasses the SOTA TempRel extraction method on MATRES by relatively 3.27% in $F_{1}$; it also offers promising performance on the HiEve dataset for subevent relation extraction, relatively surpassing previous methods by at least 3.12% in $F_{1}$.

From the NLU perspective, the second contribution of this work lies in providing a general method for inducing an event complex that comprehensively represents the relational structure of several related event mentions. This is supported by the memberships vertically identified between multi-granular events, as well as the horizontal temporal reasoning within the event complex. As far as we know, this is different from all previous works that only formulated relations along a single axis. Our model further demonstrates the potent capability of inducing event complexes when evaluated on the RED dataset O’Gorman et al. (2016).

Related Work

Various approaches have been proposed to extract event TempRels. Early effort focused on characterizing event pairs based on various types of semantic and linguistic features, and utilizing statistical learning methods, such as logistic regression Mani et al. (2006); Verhagen and Pustejovsky (2008) and SVM Mirza and Tonelli (2014), to capture the relations. Those methods typically require extensive feature engineering, and do not comprehensively consider the contextual information and global constraints among event-event relations. Recently, data-driven methods have been developed for TempRel extraction, and have offered promising performance. Ning et al. (2019) addressed this problem using a system combining an LSTM document encoder and a Siamese multi-layer perceptron (MLP) encoder for temporal commonsense knowledge from TemProb Ning et al. (2018a). Han et al. (2019a) proposed a bidirectional LSTM (BiLSTM) with structured prediction to extract TempRels. Both of these works incorporated global inference to facilitate constraints on TempRels.

Besides TempRels, a couple of efforts have focused on event hierarchy construction, a.k.a. subevent relation extraction. This task seeks to extract the hierarchy where each parent event contains child events that are described in the same document. To cope with this task, both Araki et al. (2014) and Glavaš and Šnajder (2014) introduced a variety of features and employed logistic regression models for classifying event pairs into subevent relations (Parent-Child and Child-Parent, coreference (Coref), and no relation (NoRel). Aldawsari and Finlayson (2019) further extended the characterization with more features on the discourse and narrative aspects. Zhou et al. (2020a) presented a data-driven method by fine-tuning a time duration-aware BERT Devlin et al. (2019) on corpora of time mentions, and used the estimation of time duration to predict subevent relations.

Though previous efforts have been devoted to preserving logical consistency through inference or structured learning Roth and Yih (2004); Roth and tau Yih (2007); Chang et al. (2008), this is difficult to do in the context of neural networks. Moreover, while it is a common strategy to combine multiple training data in multi-task learning Lin et al. (2020), our work is distinguished by enhancing the learning process by pushing the model towards a coherent output that satisfies logical constraints across separate tasks.

Methods

In this section, we present the joint learning framework for event-event relation extraction. We start with the problem formulation (Section 3.1), followed by the techniques for event pair characterization (Section 3.2), constrained learning (Section 3.3) and inference (Section 3.4).

A document $D$ is represented as a sequence of tokens $D=[t_{1},\cdots,e_{1},\cdots,e_{2},\cdots,t_{n}]$. Some of the tokens belong to the set of annotated event triggers, i.e., $\mathcal{E}_{D}=\{e_{1},e_{2},\cdots,e_{k}\}$, whereas the rest are other lexemes. The goal is to induce event complexes from the document, which is through extracting the multi-faceted event-event relations. Particularly, we are interested in two subtasks of relation extraction, corresponding to the label set $\mathcal{R}=\mathcal{R}_{T}\cup\mathcal{R}_{H}$. $\mathcal{R}_{T}$ thereof denotes the set of temporal relations defined in the literature Ning et al. (2017b, 2018b, 2019); Han et al. (2019b), which contains Before, After, Equal, and Vague. To be consistent with previous studies Ning et al. (2018b, 2019), the temporal ordering relations between two events are decided by the order of their starting time, without constraining on their ending time. $\mathcal{R}_{H}$ thereof denotes the set of relation labels defined in the subevent relation extraction task Hovy et al. (2013); Glavaš et al. (2014), i.e., Parent-Child, Child-Parent, Coref and NoRel. Following the definitions by Hovy et al. (2013), an event $e_{1}$ is said to have a child event $e_{2}$ if $e_{1}$ is a collector event that contains a sequence of activities, where $e_{2}$ is one of these activities, and $e_{2}$ is spatially and temporally contained within $e_{1}$. Note that each pair of events can be annotated with one relation from each of $\mathcal{R}_{H}$ and $\mathcal{R}_{T}$ respectively, as the labels within each task-specific relation set are mutually exclusive.

Our learning framework first obtains the event pair representation that combines contextualized and syntactic features along with commonsense knowledge, and then use an MLP to get confidence scores for each relation in $\mathcal{R}$. The joint learning objective seeks to enforce the logical consistency of outputs for both TempRel and subevent relations. The overall architecture is shown in Figure 2.

2 Event Pair Representation

To characterize the event pairs in the document, we employ a neural encoder architecture which provides event representations from two groups of features. Specifically, the representation here incorporates the contextualized representations of the event triggers along with statistical commonsense knowledge from several knowledge bases. On top of the features that characterize an event pair $(e_{1},e_{2})$, we use an MLP with $|\mathcal{R}|$ outputs to estimate the confidence score for each relation $r$, denoted as $r_{(e_{1},e_{2})}$. Two separate softmax functions are then added to normalize the outputs for two task-specific label sets $\mathcal{R}_{T}$ and $\mathcal{R}_{H}$.

Given a document, we first use a pre-trained language model, RoBERTa Liu et al. (2019), to produce the contextualized embeddings for all tokens of the entire document. The token embeddings are further concatenated with the one-hot vectors of POS (part-of-speech) tags, and fed into a BiLSTM. The hidden state of the last BiLSTM layer that is stacked on top of each event trigger $e$ is therefore treated as the embedding representation of the event, denoted as $h_{e}$. For each event pair $(e_{1},e_{2})$, the contextualized features are obtained as the concatenation of $h_{e_{1}}$ and $h_{e_{2}}$, along with their element-wise Hadamard product and subtraction. This is shown to be a comprehensive way to model embedding interactions Zhou et al. (2020b).

2.2 Commonsense Knowledge

We also incorporate the following sources of commonsense knowledge to characterize event pairs. Specifically, we first extract relevant knowledge from ConceptNet Speer et al. (2017), which is a large-scale commonsense knowledge graph for commonsense concepts, entities, events and relations. A portion of the relations in ConceptNet that are relevant to our tasks include “HasSubevent”, “HasFirstSubevent” and “HasLastSubevent” relations. From ConceptNet we extract around 30k pairs of event concepts labeled with the aforementioned relations, along with 30k randomly corrupted negative samples. We also incorporate commonsense knowledge from TemProb Ning et al. (2018a). This provides prior knowledge of the temporal order that some events usually follow.

We use the event pairs from those knowledge bases to train two MLP encoders. Each takes the concatenated token embeddings of two event triggers as inputs, and is trained with contrastive loss to estimate the likelihood that if a relation holds. For subevent and temporal related commonsense knowledge, two MLPs are separately trained. After the encoders are well-trained, we fix their parameters and combine them as a black box that corresponds to “Common Sense Features” in Figure 2.

3 Joint Constrained Learning

Given the characterization of grounded event pairs from the document, we now define the learning objectives for relation prediction. The goal of learning is to let the model capture the data annotation, meanwhile regularizing the model towards consistency on logic constraints. Inspired by the logic-driven framework for consistency of neural models Li et al. (2019), we specify three types of consistency requirements, i.e. annotation consistency, symmetry consistency and conjunction consistency. We hereby define the requirements with declarative logic rules, and show how we transform them into differentiable loss functions.

For labeled cases, we expect the model to predict what annotations specify. That is to say, if an event pair is annotated with relation $r$, then the model should predict so:

To obtain the learning objective that preserves the annotation consistency, we use the product t-norm to get the learning objective of maximizing the probability of the true labels, by transforming to the negative log space to capture the inconsistency with the product t-norm. Accordingly, the annotation loss is equivalently defined as the cross entropy

in which $w_{r}$ is the label weight that seeks to balance the loss for training cases of each relation $r$.

Given any event pair $(e_{1},e_{2})$, the grounds for a model to predict a relation $\alpha(e_{1},e_{2})$ to hold between them should also implies the hold of the converse relation $\bar{\alpha}(e_{2},e_{1})$. The logical formula is accordingly written as

where the $\mathcal{R}_{S}$ is the set of relations enforcing the symmetry constraint. Particularly for the TempRel extraction task, $\mathcal{R}_{S}$ contains a pair of reciprocal relations Before and After, as well as two reflexive ones Equal and Vague. Similarly, the subevent relation extraction task adds reciprocal relations Parent-Child and Child-Parent as well as reflexive ones Coref and NoRel.

Using the product t-norm and transformation to the negative log space as before, we have the symmetry loss:

This set of constraints are applicable to any three related events $e_{1},e_{2}$ and $e_{3}$. If we group the events into three pairs, namely $(e_{1},e_{2}),(e_{2},e_{3})$ and $(e_{1},e_{3})$, the relation definitions mandate that not all of the possible assignments to these three pairs are allowed. More specifically, if two relations $\alpha{(e_{1},e_{2})}$ and $\beta{(e_{2},e_{3})}$ apply to the first two pairs of events, then the conjunction consistency may enforce the following two conjunctive rules.

In the first rule, the conjunction of the first two relations infers the hold of another relation $\gamma$ between the third event pair $(e_{1},e_{3})$, namely

$\text{De}(\alpha,\beta)$ thereof is a set composed of all relations from $\mathcal{R}$ that do not conflict with $\alpha$ and $\beta$, which is a subset of the deductive closure Stine (1976) of the conjunctive clause for these two relations. A special case that the above formula expresses is a (task-specific) transitivity constraint, where $\alpha=\beta=\gamma$ present the same transitive relation.

Another condition could also hold, where the former two relations always infer the negation of a certain relation $\delta$ on $(e_{1},e_{3})$, for which we have

Table 1 is an induction table that describes all the conjunctive rules for relations in $\mathcal{R}$. To illustrate the conjunction consistency requirement (see the orange cell in Table 1), assume that $(e_{1},e_{2})$ and $(e_{2},e_{3})$ are respectively annotated with Before and Parent-Child. Then the two conjunctive formulae defined above infer that we have the relation Before hold on $(e_{1},e_{3})$, whereas we should not have Child-Parent hold.

Similar to the other consistency requirements, the loss function dedicated to the conjunction consistency is derived as follows:

where the two terms of triple losses are defined as

It is noteworthy that modeling the conjunctive consistency is key to the combination of two different event-event relation extraction tasks, as this general consistency requirement can be enforced between both TempRels and subevent relations.

After expressing the logical consistency requirements with different terms of cross-entropy loss, we combine all of those into the following joint learning objective loss

The $\lambda$’s are non-negative coefficients to control the influence of each loss term. Note that since the consistency requirements are defined on both temporal and subevent relations, the model therefore seamlessly incorporates both event-event relation extraction tasks with a shared learning objective. In this case, the learning process seeks to unify the ordered nature of time and the topological nature of subevents, therefore supporting the model to comprehensively understand the event complex.

4 Inference

To support task-specific relation extraction, i.e. extracting either a TempRel or a subevent relation, our framework selects the relation $r$ with highest confident score $r_{(e_{1},e_{2})}$ from either of $\mathcal{R}_{T}$ and $\mathcal{R}_{H}$. When it comes to extracting event complexes with both types of relations, the prediction of subevent relations has higher priority. The reason lies in the fact that a relation in $\mathcal{R}_{H}$, except for NoRel, always implies a TempRel, yet there is not a single TempRel that necessitates a subevent relation.

We also incorporate ILP in the inference phase to further ensure the logical consistency in predicted results. Nevertheless, we show in experiments that a well-trained constrained learning model may not additionally require global inference (Section 4.5).

Experiments

In this section, we present the experiments on event-event relation extraction. Specifically, we conduct evaluation for TempRel and subevent relation extraction based on two benchmark datasets (Section 4.1-Section 4.4). To help understand the significance of each model component in the framework, we also give a detailed ablation study (Section 4.5). Finally, a case study on the RED dataset is described to demonstrate the capability of inducing event complexes (Section 4.6).

Since there is not a large-scale dataset that amply annotates for both TempRel and subevent relations, we evaluate the joint training and prediction of both categories of relations on two separate datasets. Specifically, we use MATRES Ning et al. (2018b) for TempRel extraction and HiEve Glavaš et al. (2014) for subevent relation extraction.

MATRES is a new benchmark dataset for TempRel extraction, which is developed from TempEval3 UzZaman et al. (2013). It annotates on top of 275 documents with TempRels Before, After, Equal, and Vague. Particularly, the annotation process of MATRES has defined four axes for the actions of events, i.e. main, intention, opinion, and hypothetical axes. The TempRels are considered for all event pairs on the same axis and within a context of two adjacent sentences. The labels are decided by comparing the starting points of the events. The multi-axis annotation helped MATRES to achieve a high IAA of 0.84 in Cohen’s Kappa.

The HiEve corpus is a news corpus that contains 100 articles. Within each article, annotations are given for both subevent and coreference relations. The HiEve adopted the IAA measurement proposed for TempRels by UzZaman and Allen (2011), resulting in 0.69 $F_{1}$.

In addition to these two datasets, we also present a case study on an updated version of the RED dataset O’Gorman et al. (2016). This dataset contains 35 news articles with annotations for event complexes that contain both membership relations and TempRels. Since small dataset is not sufficient for training, we use it only to demonstrate our method’s capability of inducing event complexes on data that are external to training.

We briefly summarize the data statistics for HiEve, MATRES, and RED dataset in Table 3.

2 Baselines and Evaluation Protocols

On MATRES, we compare with four baseline methods. Ning et al. (2018b) present a baseline method based on a set of linguistic features and an averaged perceptron classifier (Perceptron). Han et al. (2019b) introduce a BiLSTM model that incorporates MAP inference (BiLSTM+MAP). Ning et al. (2019) present the SOTA data-driven method incorporating ILP and commonsense knowledge from TemProb with LSTM (LSTM+CSE+ILP). We also compare with the CogCompTime system Ning et al. (2018c). On HiEveDespite carefully following the details described in Aldawsari and Finlayson (2019) and communicating with the authors, we were not able to reproduce their results. Therefore, we choose to compare with other methods., we compare with a structured logistic regression model (StructLR, Glavaš and Šnajder 2014) and a recent data-driven method based on fined-tuning a time duration-aware BERT on large time-related web corpora (TacoLM, Zhou et al. 2020a).

MATRES comes with splits of 183, 72 and 20 documents respectively used for training, development and testing. Following the settings in previous work Ning et al. (2019); Han et al. (2019b), we report the micro-average of precision, recall and F1 scores on test cases. On HiEve, we use the same evaluation setting as Glavaš and Šnajder (2014) and Zhou et al. (2020a), leaving 20% of the documents out for testing. The results in terms of $F_{1}$ of Parent-Child and Child-Parent and the micro-average of them are reported. Note that in the previous setting by Glavaš and Šnajder (2014), the relations are only considered for event pairs $(e_{1},e_{2})$ where $e_{1}$ appears before $e_{2}$ in the document. We also follow Glavaš and Šnajder (2014) to populate the annotations by computing the transitive closure of CoRef and subevent relations.

3 Experimental Setup

To encode the tokens of each document, we employ the officially released 768 dimensional RoBERTa Liu et al. (2019), which is concatenated with 18 dimensional one-hot vectors representing the tokens’ POS tags. On top of those embeddings, the hidden states of the trainable BiLSTM are 768 dimensional, and we only apply one layer of BiLSTM. Since the TempRel extraction and subevent relation extraction tasks are considered with two separate sets of labels, we use two separate softmax functions for normalizing the outputs for each label set from the single MLP. For all the MLPs we employ one hidden layer each, whose dimensionality is set to the average of the input and output space following convention Chen et al. (2018).

We use AMSGrad Reddi et al. (2018) to optimize the parameters, with the learning rate set to 0.001. Label weights in the annotation loss $L_{A}$ is set to balance among training cases for different relations. The coefficients $\lambda_{S}$ and $\lambda_{D}$ in the learning objective function are both fixed to $0.2$. Training is limited to 80 epochs, which is sufficient to converge.

4 Results

In Table 2 we report the TempRel extraction results on MATRES. Among the baseline methods, Ning et al. (2019) offer the best performance in terms of $F_{1}$ by incorporating an LSTM with global inference and commonsense knowledge. In contrast, the proposed joint constrained learning framework surpasses the best baseline method by a relative gain of 3.27% in $F_{1}$, and excels in terms of both precision and recall. While both methods ensure logical constraints in learning or inference phases, the improvement by the proposed method is largely due to the joint constraints combining both TempRel and subevent relations. Learning to capture subevent relations from an extrinsic resource simultanously offer auxiliary supervision signals to improve the comprehension on TempRel, even though the resources dedicated to the later is limited.

The results in Table 4 for subevent relation extraction exhibit similar observation. Due to scarcer annotated data, the pure data-driven baseline method (TacoLM) falls behind the statistical learning one (i.e. StructLR) with comprehensively designed features. However, our model successfully complements the insufficient supervision signals, partly by incorporating linguistic and commonsense knowledge. More importantly, while our model is able to infer TempRel decently, the global consistency ensured by cross-task constraints naturally makes up for the originally weak supervision signals for subevent relations. This fact leads to promising results, drastically surpassing TacoLM with a relative gain of 21.4% in micro-average $F_{1}$, and outperforming StructLR by $\sim$3% relatively.

In general, the experiments here show that the proposed joint constrained learning approach effectively combines the scarce supervision signals for both tasks. Understanding the event complex by unifying the ordered nature of time and the topological nature of multi-granular subevents, assists the comprehension on both TempRel and memberships among multi-granular events.

5 Ablation Study

To help understand the model components, we conduct an ablation study and report the results in Table 5. Starting from the vanilla single-task BiLSTM model with only RoBERTa features, changing to joint training both tasks with only annotation brings along 1.1-1.5% of absolute gain in $F_{1}$. Incorporating task-specific constraints to learning for relations only in $\mathcal{R}_{T}$ or $\mathcal{R}_{H}$ notably brings up the $F_{1}$ 2.0-8.9%, whereas the cross-task constraints bring along an improvement of 1.0-2.6% in $F_{1}$. This indicates that the global consistency ensured within and across TempRel and subevent relations is important for enhancing the comprehension for both categories of relations. The commonsense knowledge leads to another 0.8-1.2% of improvement. Lastly, global inference does not contribute much to the performance in our setting, which indicates that the rest model components are already sufficient to preserve global consistency through joint constrained learning.

To compare both ways of ensuring logical consistency, we also report a set of results in the last row of Table 5, where constrained learning is removed and only global inference is used to cope with consistency requirements in prediction. As expected, this leads to significant performance drop of 2.6-12.3% in $F_{1}$. This fact implies that ensuring the logical consistency in the learning phase is essential, in terms of both complementing task-specific training and enhancing the comprehension of event complex components.

6 Case Study on the RED Dataset

We use the RED dataset (2019 updated version) to further evaluate our model trained on MATRES and HiEve for inducing complete event complexes, as well as to show the model’s generalizability to an external validation set. Since the labels of RED are defined differently from those in the datasets we train the model on, Table 6 shows the details about how some RED lables are mapped to MATRES and HiEve labels. Other event-event relations in RED are mapped to Vague or NoRel according to their relation types, and the relations annotated between entities are discarded. To obtain the event complexes, as stated in Section 3.4, prediction of subevent relations is given higher priority than that of TempRels. In this way, our model achieves 0.72 $F_{1}$ on TempRel extraction and 0.54 $F_{1}$ on subevent relation extraction.

Here we give an example of an event complex extracted from the RED dataset in Figure 3, using our joint constrained learning method.

Conclusion

We propose a joint constrained learning framework for extracting event complexes from documents. The proposed framework bridges TempRel and subevent relation extraction tasks with a comprehensive set of logical constraints, which are enforced during learning by converting them into differentiable objective functions. On two benchmark datasets, the proposed method outperforms SOTA statistical learning methods and data-driven methods for each task, without using data that is jointly annotated with the two classes of relations. It also presents promising event complex extraction results on RED that is external to training. Thus, our work shows that the global consistency of the event complex significantly helps understanding both temporal order and event membership. For future work, we plan to extend the framework towards an end-to-end system with event extraction. We also seek to extend the conjunctive constraints along with event argument relations.

Acknowledgement

We appreciate the anonymous reviewers for their insightful comments. Also, we would like thank Jennifer Sheffield and other members from the UPenn Cognitive Computation Group for giving suggestions to improve the manuscript.

This research is supported by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA Contract No. 2019-19051600006 under the BETTER Program, and by contract FA8750-19-2-1004 with the US Defense Advanced Research Projects Agency (DARPA). The views expressed are those of the authors and do not reflect the official policy or position of the Department of Defense or the U.S. Government.

References