Imposing Label-Relational Inductive Bias for Extremely Fine-Grained Entity Typing

Introduction

Fine-grained entity typing is the task of identifying specific semantic types of entity mentions in given contexts. In contrast to general entity types (e.g., organization, event), fine-grained types (e.g., political party, natural disaster) are often more informative and can provide valuable prior knowledge for a wide range of NLP tasks, such as coreference resolution Durrett and Klein (2014), relation extraction Yaghoobzadeh et al. (2016) and question answering Lee et al. (2006); Yavuz et al. (2016).

In practical scenarios, a key challenge of entity typing is to correctly predict multiple ground-truth type labels from a large candidate set that covers a wide range of types in different granularities. In this sense, it is essential for models to effectively capture the inter-label correlations. For instance, if an entity is identified as a “criminal”, then the entity must also be a “person”, but it is less likely for this entity to be a “police officer” at the same time. When ignoring such correlations and considering each type separately, models are often inferior in performance and prone to inconsistent predictions. As shown in Table 1, an existing model that independently predicts different types fails to reject predictions that include apparent contradictions.

Existing entity typing research often address this aspect by explicitly utilizing a given type hierarchy to design hierarchy-aware loss functions Ren et al. (2016b); Xu and Barbosa (2018) or enhanced type label encodings Shimaoka et al. (2017) that enable parameter sharing between related types. These methods rely on the assumption that the underlying type structures are predefined in entity typing datasets. For benchmarks annotated with the knowledge base (KB) guided distant supervision, this assumption is often valid since all types are from KB ontologies and naturally follow tree-like structures. However, since knowledge bases are inherently incomplete Min et al. (2013), existing KBs only include a limited set of entity types. Thus, models trained on these datasets fail to generalize to lots of unseen types. In this work, we investigate entity typing in a more open scenario where the type set is not restricted by KB schema and includes over 10,000 free-form types Choi et al. (2018). As most of the types do not follow any predefined structures, methods that explicitly incorporate type hierarchies cannot be straightforwardly applied here.

To effectively capture the underlying label correlations without access to known type structures, we propose a novel label-relational inductive bias, represented by a graph propagation layer that operates in the latent label space. Specifically, this layer learns to incorporate a label affinity matrix derived from global type co-occurrence statistics and word-level type similarities. It can be seamlessly coupled with existing models and jointly updated with other model parameters. Empirically, on the Ultra-Fine dataset Choi et al. (2018), the graph layer alone can provide a significant $11.9\%$ relative F1 improvement over previous models. Additionally, we show that the results can be further improved ($11.9\%$ $\rightarrow$ $15.3\%$) with an attention-based mention-context matching module that better handles pronouns entity mentions. With a simple modification, we demonstrate that the proposed graph layer is also beneficial to the widely used OntoNotes dataset, despite the fact that samples in OntoNotes have lower label multiplicity (i.e., average number of ground-truth types for each sample) and thus require less label-dependency modeling than the Ultra-Fine dataset.

To summarize, our major contribution includes:

We impose an effective label-relational bias on entity typing models with an easy-to-implement graph propagation layer, which allows the model to implicitly capture type dependencies;

We augment our graph-enhanced model with an attention-based matching module, which constructs stronger interactions between the mention and context representations;

Empirically, our model is able to offer significant improvements over previous models on the Ultra-Fine dataset and also reduces the cases of inconsistent type predictions.

Related Work

The task of fine-grained entity typing was first thoroughly investigated in Ling and Weld (2012), which utilized Freebase-guided distant supervision (DS) Mintz et al. (2009) for entity typing and created one of the early large-scale datasets. Although DS provides an efficient way to annotate training data, later work Gillick et al. (2014) pointed out that entity type labels induced by DS ignore entities’ local context and may have limited usage in context-aware applications. Most of the following research has since focused on testing in context-dependent scenarios. While early methods Gillick et al. (2014); Yogatama et al. (2015) on this task rely on well-designed loss functions and a suite of hand-craft features that represent both context and entities, Shimaoka et al. (2016) proposed the first attentive neural model which outperformed feature-based methods with a simple cross-entropy loss.

Modeling Entity Type Correlations

To better capture the underlying label correlations, Shimaoka et al. (2017) employed a hierarchical label encoding method and AFET Ren et al. (2016a) used the predefined label hierarchy to identify noisy annotations and proposed a partial-label loss to reduce such noise. A recent work Xu and Barbosa (2018) proposed hierarchical loss normalization which alleviated the noise of too specific types. Our work differs from these works in that we do not rely on known label structures and aim to learn the underlying correlations from data. Rabinovich and Klein (2017) recently proposed a structure-prediction approach which used type correlation features. The inference on their learned factor graph is approximated by a greedy decoding algorithm, which outperformed unstructured methods on their own dataset. Instead of using an explicit graphical model, we enforce a relational bias on model parameters, which does not introduce extra burden on label decoding.

Task Definition

Specifically, the task we consider takes a raw sentence $C$ as well as an entity mention span $M$ inside $C$ as inputs, and aims to predict the correct type labels $T_{m}$ of $M$ from a candidate type set $\mathcal{T}$, which includes more than 10,000 free-form types. The entity span $M$ here can be named entities, nominals and also pronouns. The ground-truth type set $T_{m}$ here usually includes more than one types (approximately five types on average), making this task a multi-label classification problem.

Methodology

In this section, we first briefly introduce the neural architecture to encode raw text inputs. Then we describe the matching module we use to enhance the interaction between the mention span and the context sentence. Finally, we move to the label decoder, on which we impose the label-relational bias with a graph propagation layer that encodes type co-occurrence statistics and word-level similarities. Figure 1 provides a graphical overview of our model, with 1a) illustrating both the text encoders and the matching module, and 1b) showing an example of graph propagation.

2 Mention-Context Interaction

Since most previous datasets only consider named entities, a simple concatenation of the two features $[\mathcal{C};\mathcal{M}]$ followed by a linear output layer Shimaoka et al. (2016, 2017) usually works reasonably well when making predictions. This suggests that $\mathcal{M}$ itself provides important information for recognizing entity types. However, as in our target dataset, a large portion of entity mentions are actually pronouns, such as “he” or “it”, this kind of mentions alone provide only limited clues about general entity types (e.g., “he” is a “person”) but little information about fine-grained types. In this case, directly appending representation of pronouns does not provide extra useful information for making fine-grained predictions. Thus, instead of using the concatenation operator, we propose to construct a stronger interaction between the mention and context with an attention-based matching module, which has shown its effectiveness in recent natural language inference models Mou et al. (2016); Chen et al. (2017).

With the projected mention representation $m_{proj}$ and the retrieved context feature $r_{c}$, we define the following interaction operators:

where $\rho(\cdot)$ is a gaussian error linear unit Hendrycks and Gimpel (2016) and $r$ is the fused context-mention feature; $\sigma(\cdot)$ indicates a sigmoid function and $g$ is the resulting gating function, which controls how much information in mention span itself should be passed down. We expect the model to focus less on the mention representation when it is not informative. The concatenation $[r_{c};m_{proj};r_{c}-m_{proj}]$ here is supposed to capture different aspects of the interactions. To emphasize the context’s impact, we finally concatenate the extracted context feature ($\mathcal{C}$) with the output ($o$) of the matching module ($f=[o;\mathcal{C}]$) for prediction.

3 Imposing Label-Relational Inductive Bias

We can see that every row vector of $W_{o}$ is responsible for predicting the probability of one particular type. We will refer the row vectors as type vectors for the rest of this paper. As these type vectors are independent, the label correlations are only implicitly captured by sharing the model parameters that are used to extract $f$. We argue that the paradigm of parameter sharing is not enough to impose strong label dependencies and the values of type vectors should be better constrained.

A straightforward way to impose the desired constraints is to add extra regularization terms on $W_{o}$. We first tested several auxiliary loss functions based on the heuristics from GloVe Pennington et al. (2014), which operates on the type co-occurrence matrix. However, the auxiliary losses only offer trivial improvements in our experiments. Instead, we find that directly imposing a model-level inductive bias on the type vectors turns out to be a more principled solution. This is done by adding a graph propagation layer over randomly initialized $W_{o}$ and generating the updated type vectors $W^{{}^{\prime}}_{o}$, which is used for final prediction. Both $W_{o}$ and the graph convolution layer are learned together with other model parameters. We view this layer as the key component of our model and use the rest of this section to describe how we create the label graph and compute the propagation over the graph edges.

In KB-supervised datasets, the entity types are usually arranged in tree-like structures. Without any prior about type structures, we consider a more general graph-like structure. While the nodes in the graph straightforwardly represent entity types, the meaning of the edges is relatively vague, and the connections are also unknown. In order to create meaningful edges using training data as the only resource, we utilize the type co-occurrence matrix: if two type $t_{1}$ and $t_{2}$ both appear to be the true types of a particular entity mention, we will add an edge between them. In other words, we are using the co-occurrence statistics to approximate the pair-wise dependencies and the co-occurrence matrix now serves as the adjacent matrix. Intuitively, if $t_{2}$ co-appears with $t_{1}$ more often than another type $t_{3}$, the probabilities of $t_{1}$ and $t_{2}$ should have stronger dependencies and the corresponding type vectors should be more similar in the vector space. In this sense, we expect each type vector to effectively capture the local neighbor structure on the graph.

Correlation Encoding via Graph Convolution

To encode the neighbor information into each node’s representation, we follow the propagation rule defined in Graph Convolution Network (GCN) Kipf and Welling (2016). In particular, with the adjacent or co-occurrence matrix $A$, we define the following propagation rule on $W_{o}$:

works similarly well and is more efficient as it involves less matrix multiplications. If we look closely and take each node out, the propagation can be written as

From this formula, we can see that the propagation is essentially gathering features from the first-order neighbors. In this way, the prediction on type $t_{i}$ is dependent on its neighbor types.

Compared to original GCNs that often use multi-hop propagations (i.e., multiple graph layers connected by nonlinear functions) to capture higher-order neighbor structures. We only apply one-hop propagation and argue that high-order label dependency is not necessarily beneficial in our scenario and might introduce false bias. A simple illustration is shown in Figure 2. We can see that propagating 2-hop information introduces undesired inductive bias, since types that are more than 1-hop away (e.g., “Engineer” and “Politician”) usually do not have any dependencies. In fact, some of the 2-hop type pairs can be contradictory types (e.g., “police” and “prisoner”). This hypothesis is consistent with our experiment results: adding more than one graph layer leads to worse results. Additionally, we also omit GCN’s nonlinear activation which introduces unnecessary constraints on the scale of $W_{o}^{{}^{\prime}}$, with which we calculate the unscaled scores before calculating the probability via a sigmoid function.

4 Leveraging Label Word Embeddings

As the type labels are all written as text phrases, an interesting question is whether we can exploit the semantics provided by pre-trained word embeddings to improve entity typing. We explore this possibility by using the cosine similarity of word embeddings. We first calculate type embeddings by simply summing the embeddings of all tokens in the type name. Then we build a label affinity matrix $A_{word}$ by calculating pair-wise cosine similarities. With the assumption that word-level similarity measures some degree of label dependency, we propose to integrate $A_{word}$ into the graph convolution layer following

Experiments

Our experiments mainly focus on the Ultra-Fine entity typing dataset which has 10,331 labels and most of them are defined as free-form text phrases. The training set is annotated with heterogeneous supervisions based on KB, Wikipedia and head words in dependency trees, resulting in about 25.2MChoi et al. (2018) use the licensed Gigaword to build part of the dataset, while in our experiments we only use the open-sourced training set which has approximately 6M training samples. training samples. This dataset also includes around 6,000 crowdsourced samples. Each of these samples has five ground-truth labels on average. For a fair comparison, we use the original test split of the crowdsourced data for evaluation. To better understand the capability of our model, we also test our model on the commonly-used OntoNotes Gillick et al. (2014) benchmark. It is worth noting that this dataset is much smaller and has lower label multiplicity than the Ultra-Fine dataset, i.e., each sample only has around 1.5 labels on average. Figure 3 shows a comparison of these two datasets.

Baselines

For the Ultra-Fine dataset, we compare our model with AttentiveNER Shimaoka et al. (2016) and the multi-task model proposed with the Ultra-Fine dataset. Note that other models that require pre-defined type hierarchy are not applicable to this dataset. For experiments on OntoNotes, in addition to the two neural baselines for Ultra-Fine, we compare with several existing methods that explicitly utilize the pre-defined type structures in loss functions. Namely, these methods are AFET Ren et al. (2016a), LNR Ren et al. (2016b) and NFETC Xu and Barbosa (2018).

Evaluation Metrics

On Ultra-Fine, we first evaluate the mean reciprocal rank (MRR), macro precision(P), recall (R) and F1 following existing research. As P, R and F1 all depend on a chosen threshold on probabilities, we also consider a more transparent comparison using precision-recall curves. On OntoNotes, we use the standard metrics used by baseline models: accuracy, macro, and micro F1 scores.

Implementation Details

Most of the model hyperparameters, such as embedding dimensions, learning rate, batch size, dropout ratios on context and mention representations are consistent with existing models. Since the mention-context matching module brings more parameters, we apply a dropout layer over the extracted feature $f$ to avoid overfitting. We list all the hyperparameters in the appendix. Models for OntoNotes are trained with standard binary cross-entropy (BCE) losses defined on all candidate labels. When training on Ultra-Fine, we adopt the multi-task loss proposed in Choi et al. (2018) which divides the cross-entropy loss into three separate losses over different type granularities. The multi-task objective avoids penalizing false negative types and can achieve higher recalls.

2 Evaluation on the Ultra-Fine Dataset

We report the results on Ultra-Fine in Table 2. It is worth mentioning that our model, denoted as LabelGCN, is trained using the unlicensed training set which is smaller than the one used by compared baselines. Even though our model significantly outperforms the baselines, for a fair comparison, we first test our model using the same decision threshold (0.5) used by previous models. In terms of F1, our best model (LabelGCN) outperforms existing methods by a large margin. Compared to Choi et al. (2018), our model improves on both precision and recall significantly. Compared to the AttentiveNER trained with standard BCE loss, our model achieves much higher recall but performs worse in precision. This is due to the fact that when trained with BCE loss, the model usually retrieves only one label per sample and these types are mostly general typesAccording to the results of our own implementation of BCE-trained model which achieves similar performance as AttentiveNER. which are easier to predict. With higher recalls or more retrieved types, achieving high precision requires being accurate on fine-grained types, which are often harder to predict.

As the precision and recall scores both rely on the decision threshold, different models or different metrics can have different optimal thresholds. As shown by the “LabelGCN + thresh tuning” entry in Table 2, with threshold tuning, our model beats baselines in all metrics. We also see that recall is usually lagging behind precision on this dataset, indicating that F1 score is mainly affected by the recall and tuning towards recall can usually lead to higher F1 scores. For more transparent comparisons, we show the precision-recall curves in Figure 4. These data points are based on the validation performance given by 50 equal-interval thresholds between 0 and 1. We can see there is a clear margin between our model and the multi-task baseline method (LabelGCN vs Choi et al.).

3 Ablation Studies

To quantify the effect of different model components, we report the performance of model variants in Table 2 and Figure 4. We can clearly see that the graph convolution layer is the most essential component. The information provided by word embedding is useful and can further improve both precision and recall. Although Table 2 seems to indicate the interaction module decreases the precision, we can see from Figure 4 that with a proper threshold, the enhanced interaction actually improves both precision and recall. In term of this, we recommend future research to use PR curves for more accurate model analysis.

4 Fine-Grained Performance for Pronouns

As discussed in Section 4.2, the mention representation of pronouns provide limited information about fine-grained types. We investigate the effect of the enhanced mention-context interaction by analyzing the decomposed performance on pronouns and other kinds of entities. From the results in Table 3, we can see that the enhanced interaction offers consistent improvements over pronouns entities and also maintains the performance on other kinds of entities.

5 Qualitative Analysis

To gain insights on the improvements provided by our model, we manually analyze 100 error casesThe baseline model achieves the lowest precision on these 100 samples. of the baseline model (Choi et al. (2018) with threshold 0.5) and see if our model can generate high-quality predictions. We first observe that many errors actually results from incomplete annotations. This suggests models’ precision scores are often underestimated in this dataset. We discuss several typical error cases shown in Table 4 and list more samples in the appendix (Table 7).

A key observation is that while the baseline model tends to make inconsistent predictions (see examples 1, 2, 3), our model can avoid predicting such inconsistent type pairs. This indeed validates our model’s ability to encode label correlations. We also notice that our model is more sensitive to gender information indicated by pronouns, while the baseline model sometimes holds the gender-indicating predictions and predict other types, our model predicts the gender-indicating types more often (examples 3, 4, 5). We conjecture that our model learns this easy way to maintain precision.

For cases that both models fail, some of them actually require background knowledge (example 4) to make accurate predictions. Another typical case is that both models predict some other entities in the context (example 5). We think this potentially results from the data bias introduced by the head-word supervision.

6 Evaluation on OntoNotes

To better understand the requirements for applying our model, we further evaluate on the OntoNotes dataset. Here we do not apply the proposed mention-context matching module as this dataset does not include any pronoun entities. To obtain more reliable co-occurrence statistics, we use the augmented training data released by Choi et al. (2018). However, since the training set is still much smaller than that of the Ultra-Fine dataset, the derived co-occurrence statistics are relatively noisy and might introduce undesired bias. We thus add an additional residual connection to our graph convolution layer, which allows the model to selectively use co-occurrence statistics. This indeed gives us improvements over previous state-of-the-arts, as shown in Table 5. However, compared to Ultra-Fine, the margin of the improvement is smaller. In view of the key differences of these two datasets, we highlight two key requirements for our proposed model to offer substantial improvements. First, there should be a large-scale training set so that the derived co-occurrence statistics can reasonably reflect the true label correlations. Second, the samples themselves should also have higher label multiplicity. In fact, most of the samples in OntoNotes only have 1 or 2 labels. This property actually alleviates the need for models to capture label dependencies.

Conclusion

In this paper, we present an effective method to impose label-relational inductive bias on fine-grained entity typing models. Specifically, we utilize a graph convolution layer to incorporate type co-occurrence statistics and word-level type similarities. This layer implicitly captures the label correlations in the latent vector space. Along with an attention-based mention-context matching module, we achieve significant improvements over previous methods on a large-scale dataset. As our method does not require external knowledge about the label structures, we believe our method is general enough and has the potential to be applied to other multi-label tasks with plain-text labels.

Acknowledgement

This research was supported in part by DARPA Grant D18AP00044 funded under the DARPA YFA program. The authors are solely responsible for the contents of the paper, and the opinions expressed in this publication do not reflect those of the funding agencies.

References

Appendix A Appendix