Taxonomy Completion via Triplet Matching Network

Jieyu Zhang, Xiangchen Song, Ying Zeng, Jiaze Chen, Jiaming Shen, Yuning Mao, Lei Li

Introduction

Taxonomies, formulated as directed acyclic graphs or trees, have been widely used to organize knowledge in various domains, such as news domain (Vrandecic 2012; Mao et al. 2019), scientific domain (Lipscomb 2000; Sinha et al. 2015; Shen et al. 2018c) and online commerce (Karamanolakis, Ma, and Dong 2020; Mao et al. 2020). Equipped with these curated taxonomies, researchers are able to boost the performance of numerous downstream applications such as query understanding (Hua et al. 2017; Yang, Zhang, and Han 2020), content browsing (Yang 2012), personalized recommendation (Zhang et al. 2014; Huang et al. 2019), and web search (Wu et al. 2012; Liu et al. 2019).

As human knowledge is constantly growing and new concepts emerge everyday, it is needed to dynamically complete an existing taxonomy. Figure 1 shows an illustrative example where a taxonomy of “Electronic Device” is completed to include new devices (e.g., “Smart Phone”) and hardware (e.g., “SSD”). Most existing taxonomies are curated by domain experts. However, such manual curations are labor-intensive, time-consuming and rarely-complete, and therefore infeasible to handle the influx of new contents in online streaming setting. To this end, many recent studies (Shen et al. 2020; Manzoor et al. 2020; Yu et al. 2020) investigate the problem of taxonomy expansion which aims to automatically expand an existing taxonomy. Specifically, given a query concept, these methods first rank each concept in the existing taxonomy based on how likely it is the hypernym of the query concept measured by an one-to-one matching score between the two concepts. Then, the query concept is added into the existing taxonomy as the hyponym of the top-ranked concepts. Notice that such a formulation is built upon one strong assumption: all new concepts can only be added into existing taxonomy as hyponyms (i.e., leaf nodesNodes with zero out-degree in a directed acyclic graph.). However, we argue that such a “hyponym-only” assumption is inappropriate in real applications. For example, in Fig 1, the term “Smart Phone” is invented much later than term “CPU”, which means that when “Smart Phone” emerges, “CPU” already exists in taxonomy. In this case, it is inappropriate to add “Smart Phone” into taxonomy as leaf node because “CPU” is a hyponym of “Smart Phone”.

In this paper, instead, we define and investigate a new taxonomy completion task without the strong “hyponym-only” assumption. Formally, given an existing taxonomy and a set of new concepts, we aim to automatically complete the taxonomy to incorporate these new concepts by discovering the most likely $\langle$ hypernym, hyponym $\rangle$ pairs of each new concept. For instance, in Fig 1, one of the most likely candidate pairs for “Smart Phone” is $\langle$ “Electronic Device”, “CPU” $\rangle$ . This formulation leads to a novel one-to-pair matching problem different from the previous one-to-one setting in taxonomy expansion task that only seeks for a new concept’s most likely hypernyms while ignores its possible hyponyms. Note that the hypernym/hyponym concept within the candidate $\langle$ hypernym, hyponym $\rangle$ pair could be a “pseudo concept” in case there is no appropriate one for a given query concept. We can easily see that the taxonomy expansion task is a special case of taxonomy completion when the hyponym concepts are always “pseudo concept”.

Tackling the new taxonomy completion task is challenging because the induced one-to-pair matching problem results in the existence of a special type of negative candidate $\langle$ hypernym, hyponym $\rangle$ pairs we called partially-correct negative candidates. Before introducing partially-correct negative candidates, we first clarify that for a given query concept $n_{q}$ , a candidate pair of existing concepts $\langle n_{p},n_{c}\rangle$ is positive if $n_{p}$ ( $n_{c}$ ) is the true hypernym (hyponym) of $n_{q}$ and negtiave otherwise. Then, a candidate pair $\langle n_{p},n_{c}\rangle$ is partially-correct negative if either $n_{p}$ is true hypernym but $n_{c}$ is not true hyponym or vice versa. We illustrate the different types of candidate pairs in the table of Fig 1. Due to the high correlation of positive and partially-correct negative candidates, the model might struggle to distinguish one from another.

To solve the aforementioned challenge, we propose a novel Triplet Matching Network (TMN), which learns a scoring function to output the matching score of a $\langle$ query, hypernym, hyponym $\rangle$ triplet and leverages auxiliary signals to help distinguish positive pairs from partially-correct negative ones. Specifically, auxiliary signals are binary signals indicating whether one component within the pair is positive or not, in contrast to binary primal signals that reveal holistically whether a candidate position is positive or not. To make best use of the auxiliary signals to handle the existence of partially-correct negative, TMN consists of multiple auxiliary scorers that learn different auxiliary signals via corresponding auxiliary loss and one primal scorer that aggregates internal feature representations of auxiliary scorers to output the final matching score. The auxiliary and primal scorers are jointly trained in an auxiliary learning framework. In this way, we encourage the model to learn meaningful internal feature representations for the primal scorer to discriminate between positive, negative, and partially-correct negative candidates. In addition, we propose an innovative technique called channel-wise gating mechanism to regulate the representations of concepts. It produces a channel-wise gating vector based on the $\langle$ query, hypernym, hyponym $\rangle$ triplet, and then modifies the embeddings using this channel-wise gating vector to reduce the effect of irrelevant information stored in embeddings while retain the most task-specific information when calculating matching scores.

In the experiments, we benchmark the taxonomy completion task on four real-world taxonomies from different domains using modified version of multiple one-to-one matching models and state-of-the-art taxonomy expansion methods. Our experimental results show that TMN outperforms the baselines by a large margin on both taxonomy completion task and taxonomy expansion task. Finally, ablation study demonstrates the effectiveness of each component of TMN, and efficiency analysis shows the efficiency of TMN at inference stage.

Contributions. To summarize, our major contributions include: (1) a more realistic task called taxonomy completion which simultaneously finds hypernym and hyponym concepts of new concepts; (2) a novel and effective Triple Matching Network (TMN) to solve the one-to-pair matching problem induced from the taxonomy completion task by leveraging auxiliary signals and an innovative channel-wise gating mechanism; and (3) extensive experiments that verify both the effectiveness and efficiency of TMN framework on four real-world large-scale taxonomies from different domains.

Problem Formulation

Taxonomy. Follow (Shen et al. 2020), we define a taxonomy $\mathcal{T}=(\mathcal{N},\mathcal{E})$ as a directed acyclic graph where each node $n\in\mathcal{N}$ represents a concept (i.e., a word or a phrase) and each directed edge $\langle n_{p},n_{c}\rangle\in\mathcal{E}$ indicates a relation expressing that concept $n_{p}$ is the most specific concept that is more general than concept $n_{c}$ . Here, the relation types of edges are implicitly defined by existing taxonomy.

Candidate Position. A valid candidate position is a pair of concepts $\langle n_{p},n_{c}\rangle$ where $n_{c}$ is one of the descendants of $n_{p}$ in the existing taxonomy. This definition reduces the search space of candidate positions. Note that $n_{p}$ or $n_{c}$ could be a “pseudo concept” acting as a placeholder.

Positive Position. For a query concept $n_{q}$ , positive position $\langle n_{p},n_{c}\rangle$ is a candidate position wherein $n_{p}$ and $n_{c}$ is the true parent and child of $n_{q}$ , respectively.

Negative Position. For a query concept $n_{q}$ , negative position $\langle n_{p},n_{c}\rangle$ is a candidate position wherein $n_{p}$ or $n_{c}$ is not the true parent or child of $n_{q}$ , respectively.

Partially-correct Negative Position. For a query concept $n_{q}$ , partially-correct negative position $\langle n_{p},n_{c}\rangle$ is a negative position but $n_{p}$ or $n_{c}$ is the true parent or child of $n_{q}$ .

The TMN Framework

In this section, we first introduce our one-to-pair matching model which leverages auxiliary signals to augment primal matching task. Then, we present a novel channel-wise gating mechanism designed for regulating concept embedding to boost the model performance. Finally, we discuss how to generate self-supervision data from the existing taxonomy and use them to train the TMN model. The overall model architecture is presented in Fig 2.

Where $f$ is a parameterized scoring function of choice that outputs the relatedness score of $n_{q}$ and $\langle n_{p},n_{c}\rangle$ , and $[\cdot]$ represents the concatenation operation. This formulation simply degenerates one-to-pair matching into one-to-one matching by using concatenation of $\mathbf{x}_{p}$ and $\mathbf{x}_{c}$ as representation of candidate position. Here, we choose the neural tensor network (Socher et al. 2013) as our base model:

However, such a naïve instantiation only measures the coarse-grained relatedness of query $n_{q}$ and the whole candidate pair $\langle n_{p},n_{c}\rangle$ but fails to capture the fine-grained relatedness of $\langle n_{q},n_{p}\rangle$ and $\langle n_{q},n_{c}\rangle$ , preventing the model from learning to clearly distinguish positive candidates from partially-correct negatives candidates.

To address the limitation of the naive approach, we propose a novel expressive Triplet Matching Network (TMN). Specifically, we develop multiple auxiliary scorers to capture both coarse- and fine-grained relatedness in one-to-pair matching, and one primal scorer that inputs the internal feature representations of all auxiliary scorers and outputs final matching scores. For each auxiliary scorer, we adopt neural tensor network as in Eq. 2 as instantiation due to its expressiveness, and the corresponding $k$ -dimension internal feature representation $\mathbf{h}$ is as in Eq. 3. Assume we have $l$ auxiliary scorers, each with $k$ -dimension internal feature representation $\mathbf{h}_{j}$ and $j=1,\dots,l$ . Then, the primal scorer is a single-layer projection with non-linear activation function:

Where the auxiliary scorer $s_{1}$ and $s_{2}$ capture the fine-grained relatedness of $\langle n_{q},n_{p}\rangle$ and $\langle n_{q},n_{c}\rangle$ respectively, while $s_{3}$ is for coarse-grained relatedness between $n_{q}$ and $\langle n_{p},n_{c}\rangle$ .

Given above formulations, primal scorer $s_{primal}$ can be trained using primal signals indicating whether $\langle n_{p},n_{c}\rangle$ is positive candidate of $n_{q}$ or not, and auxiliary scorers can be trained via corresponding auxiliary signals. Particularly, $s_{1}$ will be trained to learn whether $n_{p}$ is positive parent of $n_{q}$ , $s_{2}$ is to learn whether $n_{c}$ is positive child of $n_{q}$ , and $s_{3}$ captures coarse-grained relatedness between $n_{q}$ and $\langle n_{p},n_{c}\rangle$ so its auxiliary signal is exactly the same as primal signal. Although $s_{3}$ and $s_{primal}$ share the same supervision signals and both aim to capture relatedness between $n_{q}$ and $\langle n_{p},n_{c}\rangle$ , $s_{primal}$ outputs matching score based on the internal feature representations of all auxiliary scorers including $s_{3}$ , which enables $s_{primal}$ to rely on $s_{1}$ or $s_{2}$ when $s_{3}$ struggle to differentiate positive candidates from partially-correct candidates negatives.

Channel-wise Gating Mechanism

As the nature of taxonomy, concepts under the same ancestor are semantically related to each other, which makes it challenging for model to learn the true taxonomic relations based on concept embeddings, especially in bottom-level of a taxonomy. For instance, in Fig 1, the model needs to learn that “Disk” is the true parent of “SSD” but “Memory” is not. However, “Disk” and “Memory” are siblings in taxonomy, which makes them highly-related, and therefore hard to distinguish based on their embeddings learned from a more general corpus.

To mitigate this problem, instead of directly using initial embedding vectors of concepts, we propose a novel channel-wise gating mechanism to regulate the information stored in initial embedding vectors, reducing the negative effects of irrelevant or spurious information on learning taxonomic relations. Specially, to distinguish “Disk” and “Memory” that both belong to “Desktop”, we would like to filter out the shared information stored in their embeddings related to “Desktop” in order to push the model to focus on the remaining more specific information. Formally, we give the formulation of channel-wise gating mechanism as follows:

With the gated embedding $\hat{\mathbf{x}}_{p}$ and $\hat{\mathbf{x}}_{c}$ in hand, we now replace the initial embedding vectors in Eq. 3 with it to facilitate TMN. Notably, this simple channel-wise gating mechanism is ready to be plugged in any matching models.

Jointly Learning Primal and Auxiliary Scorers

In this section, we first introduce how to generate self-supervision data as well as primal and auxiliary signals from the existing taxonomy, and then propose to jointly learn the primal and auxiliary scorers.

where $\mathcal{L}_{p}$ represents the loss for primal scorer and $\mathcal{L}_{1}$ , $\mathcal{L}_{2}$ , $\mathcal{L}_{3}$ are auxiliary losses for auxiliary scorers $s_{1}$ , $s_{2}$ and $s_{3}$ respectively. The hyperparameters $\lambda_{1}$ , $\lambda_{2}$ and $\lambda_{3}$ are weights to adjust relative effects of each auxiliary loss. The above objective function is similar to multi-task learning at the first glance, but it is an auxiliary learning strategy that only cares the performance of primal task, i.e. primal scorer in our case, and the auxiliary loss are meant to augment the learning of primal task.

Here, $\mathcal{L}_{i}$ is loss function of choice. We choose binary cross entropy loss for simplicity. Take the primal loss as an example, it is formulated as:

Experiments

Dataset. We study the performance of TMN on four large-scale real-world taxonomies.

Microsoft Academic Graph (MAG). We evaluate TMN on the public Field-of-Study (FoS) Taxonomy in Microsoft Academic Graph (MAG) (Sinha et al. 2015). It contains over 660 thousand scientific concepts and more than 700 thousand taxonomic relations. Following (Shen et al. 2020), we construct two datasets which we refer to as MAG-Psychology and MAG-CS based on the subgraph related to the “Psychology” and “Computer Science” domain, respectively. We compute a 250-dimension word word2vec embedding on a related paper abstracts corpus.

WordNet. Based on WordNet 3.0, we collect verbs and nouns along with the relations among them to form two datasets which we refer to as WordNet-Verb and WordNet-Noun, respectively. The reason for limiting our choice to only verbs and nouns is that only these parts of speech have fully-developed taxonomies in WordNet (Jurgens and Pilehvar 2016). We obtain the 300-dimension fasttext embeddingsWe use the wiki-news-300d-1M-subword.vec.zip version on official website. as initial feature vectors.

For each dataset, we randomly sample 1,000 nodes for validation and another 1,000 for test. Then we build the initial taxonomy using remaining nodes and associated edges. Notice that new edges will be added into initial taxonomy to avoid the taxonomy from breaking into multiple directed acyclic graphs. Table 1 lists the statistics of these four datasets.

Evaluation Metrics. As our model returns a rank list of candidate positions for each query concept, we evaluate its performance using the following ranking-based metrics.

Mean Rank (MR) measures the average rank position of a query concept’s true positions among all candidates. For queries with multiple positive edges, we first calculate the rank position of each individual edge and then take the average of all rank positions.

Mean Reciprocal Rank (MRR) calculates the reciprocal rank of a query concept’s true positions. We follow (Ying et al. 2018) and scale the original MRR by a factor 10 to amplify the performance gap between different methods.

Recall@ $k$ is the number of query concepts’ true positions ranked in the top $k$ , divided by the total number of true positions of all query concepts.

Precision@ $k$ is the number of query concepts’ true positions ranked in the top $k$ , divided by the total number of queries times $k$ .

Compared Methods. To the best of our knowledge, we are the first to study taxonomy completion task and there is no directly comparable previous method. Thus, we adapt the following related methods to our problem setting and compare TMN with them:

Closest-Position: A rule-based method which ranks candidate positions based on the cosine similarity:

Single Layer Model: A model that scores tuple by a standard single layer neural network which inputs the concatenation of the concept embeddings.

Multiple Layer Model: An extension of Single Layer Model that replaces the single layer neural network with multiple layer neural network.

Bilinear Model (Sutskever, Salakhutdinov, and Tenenbaum 2009; Jenatton et al. 2012): It incorporates the interaction of two concept embeddings through a simple and efficient bilinear form.

Nerual Tensor Network (Socher et al. 2013): It incorporates Single Layer Model with a bilinear tensor layer that directly relates the two concept embeddings across multiple dimensions and a bias vector.

TaxoExpan (Shen et al. 2020): One state-of-the-art taxonomy expansion framework which leverages position-enhanced graph neural network to capture local information and InfoNCE loss(Oord, Li, and Vinyals 2018) for robust training.

ARBORIST (Manzoor et al. 2020): One state-of-the-art taxonomy expansion model which aims for taxonomies with heterogeneous edge semantics and optimizes a large-margin ranking loss with a dynamic margin function.

Notably, except for the rule-based method Closest-Position, other baselines are learning-based method and designed for one-to-one matching. Thus we concatenate the embeddings of candidate’s constituting concepts as candidate embedding to fit our one-to-pair setting. For fair comparison, we replace the GNN encoder of TaxoExpan with initial feature vector to align with other compared methods. There are other recently-proposed taxonomy expansion methods, e.g., HiExpan (Shen et al. 2018b) and STEAM (Yu et al. 2020). We do not include them as baselines because they leverage external sources, e.g., text corpus, to extract complicated features, while TMN and other baselines only take initial feature vectors as input.

Parameter Settings. For learning-based methods, we use Adam optimizer with initial learning rate 0.001 and ReduceLROnPlateau schedulerhttps://pytorch.org/docs/stable/optim.html“#torch.optim.lr“˙scheduler.ReduceLROnPlateau with ten patience epochs. During model training, the batch size and negative sample size is set to 128 and 31, respectively. We set $k$ , i.e., the dimension of internal feature representation, to be 5. For TMN, we simply set $\lambda_{1}=\lambda_{2}=\lambda_{3}=1$ to avoid heavy hyperparameter tuning.

Experimental Results

Overall Performance. Table 2 presents the results of all compared methods on the four datasets. First, we find that learning-based methods clearly outperform rule-based Closest-Position method. Second, there is no baseline that could consistently outperform others in all taxonomies, which indicates the diversity of taxonomies of different domains and the difficulty of taxonomy completion task. Third, ARBORIST and TaxoExpan do not work well in taxonomy completion, which indicates that methods carefully designed for taxonomy expansion task will struggle in taxonomy completion task. Finally, our proposed TMN has the overall best performance across all the metrics and defeats the second best method by a large margin.

Performance on Taxonomy Expansion. As taxonomy expansion being a special case of our novel taxonomy completion task, we are curious about how TMN performs on previous task. Thus, we compare TMN with ARBORIST and TaxoExpan on taxonomy expansion taskWe sample validation/test set from leaf nodes for taxonomy expansion task.. The results are presented in Table 3. Notice that ARBORIST and TaxoExpan is trained directly on taxonomy expansion task, while TMN is trained solely on taxonomy completion task. From the results, we can see TMN outperforms the others in both dataset with a large margin, which indicates that TMN is able to solve taxonomy expansion task better than previous state-of-the-arts.

Ablation Study. We conduct the ablation studies on two representative datasets MAG-Psychology and WordNet-Verb, and the results are presented in Table 4. The results show that without any of the key components of TMN, i.e., auxiliary scorers ( $s_{1}$ , $s_{2}$ and $s_{3}$ ) and channel-wise gating mechanism (CG), the overall performance will degrade by different extends, which indicates the effectiveness of the components.

Efficiency Analysis. At the training stage, our model uses $|\mathcal{N}^{(0)}|$ training instances every epoch and thus scales linearly to the number of concepts in the existing taxonomy. At inference stage, because the cardinality of candidate pairs is $|\mathcal{N}^{(0)}|^{2}$ without any restriction, for each query concept, we need to calculate $|\mathcal{N}^{(0)}|^{2}$ matching scores, one for every candidate pair. However, in practical, as we restrict the valid candidate pairs to be $\langle$ ancestor, descendant $\rangle$ concept pairs in existing taxonomy, the number of candidates need to be considered is substantially reduced and therefore the inference efficiency is largely improved. Also, the inference stage can be further accelerated using GPU. We list the number of valid candidate pairs and the average running time per query during inference stage of all datasets in Table 5. From the table, we can see the number of valid candidate pairs is no more than ten times of $|\mathcal{N}^{(0)}|$ and thus the inference stage is quite efficient.

Case Study. We illustrate the power of TMN via two real query concepts “Detective” and “Toyon” of WordNet-Noun in Fig.3. For internal concept “Detective”, TMN ranks the true positions $\langle$ “Investigator”, “Private Detective” $\rangle$ at top 1 and $\langle$ “Investigator”, “Sleuth” $\rangle$ at top 2, while Arborist can only rank the true parent “Investigator” at top 23. For leaf concept “Toyon”, TMN recovers its true parent “Shrub” but Arborist ranks “Shrub” at top 5. We can see that TMN works better than baseline in terms of recovering true positions.

Related Work

Taxonomy Construction and Expansion. Automatic taxonomy construction is a long-standing task in the literature. Existing taxonomy construction methods leverage lexical features from the resource corpus such as lexical-patterns (Nakashole, Weikum, and Suchanek 2012; Jiang et al. 2017; Hearst 1992; Agichtein and Gravano 2000) or distributional representations (Mao et al. 2018; Zhang et al. 2018; Jin, Barzilay, and Jaakkola 2018; Luu et al. 2016; Roller, Erk, and Boleda 2014; Weeds, Weir, and McCarthy 2004) to construct a taxonomy from scratch. However, in many real-world applications, some existing taxonomies may have already been laboriously curated and are deployed in online systems, which calls for solutions to the taxonomy expansion problem. To this end, multitudinous methods have been proposed recently to solve the taxonomy expansion problem (Vedula et al. 2018; Shen et al. 2018b; Manzoor et al. 2020; Shen et al. 2020; Yu et al. 2020; Mao et al. 2020). For example, Arborist (Manzoor et al. 2020) studies expanding taxonomies by jointly learning latent representations for edge semantics and taxonomy concepts; TaxoExpan (Shen et al. 2020) proposes position-enhanced graph neural networks to encode the relative position of terms and a robust InfoNCE loss; STEAM (Yu et al. 2020) re-formulates the taxonomy expansion task as a mini-path-based prediction task and proposes to solve it through a multi-view co-training objective. However, all the existing taxonomy expansion methods aim for solving the one-to-one matching problem, i.e. to find the true parent/hypernym, which is incompatible to our novel one-to-pair matching problem induced by taxonomy completion task.

Auxiliary Learning. Auxiliary learning refers to a learning strategy that facilitates training of a primal task with auxiliary tasks (Ruder 2017; Shen et al. 2018a). Different from multi-task learning, auxiliary learning only cares the performance of the primal task. The benefits of auxiliary learning have been proved in various applications (Standley et al. 2020; Tang et al. 2020; Trinh et al. 2018; Toshniwal et al. 2017; Hwang et al. 2020; Jaderberg et al. 2017; Odena, Olah, and Shlens 2017; Liu, Davison, and Johns 2019; Lin et al. 2019; Xiao et al. 2019). In most of these contexts, joint training with auxiliary tasks adds an inductive bias, encouraging the model to learn meaningful representations and avoid overfitting spurious correlations. Despite the numerous applications of auxiliary learning, its benefits on taxonomy construction remains less investigated. To our best knowledge, we are the first to leverage auxiliary learning to enhance taxonomy construction.

Conclusion

This paper studies taxonomy completion without manually labeled supervised data. We propose a novel TMN framework to solve the one-to-pair matching problem in taxonomy completion, which can be applied on other applications where one-to-pair matching problem exists. Extensive experiments demonstrate the effectiveness of TMN on various taxonomies. Interesting future work includes leveraging current method to cleaning the existing taxonomy, and incorporating feedback from downstream applications (e.g., searching & recommendation) to generate more diverse (auxiliary) supervision signals for taxonomy completion.

Acknowledgements

Thanks the anonymous reviewers for their helpful comments and suggestions. We also thank Jingjing Xu, Hao Zhou, and Jiawei Han for their valuable discussions.