sense2vec - A Fast and Accurate Method for Word Sense Disambiguation In Neural Word Embeddings

Introduction

NLP systems seek to automate the extraction of information from human language. A key challenge in this task is the complexity and sparsity in natural language, which leads to a phenomenon known as the curse of dimensionality. To overcome this, recent work has learned real valued, distributed representations for words using neural networks (G.E. Hinton, 1986; Bengio et al., 2003; Morin & Bengio, 2005; Mnih & Hinton, 2009). These ”neural language models” embed a vocabulary into a smaller dimensional linear space that models ”the probability function for word sequences, expressed in terms of these representations” (Bengio et al., 2003). The result is a vector-space model (VSM) that represents word meanings with vectors that capture the semantic and syntactic information of words (Maas & Ng, 2010). These distributed representations model shades of meaning across their dimensions, allowing for multiple words to have multiple real-valued relationships encoded in a single vector (Liang & Potts, 2015).

Various forms of distributed representations have shown to be useful for a wide variety of NLP tasks including Part-of-Speech tagging, Named Entity Recognition, Analogy/Similarity Querying, Transliteration, and Dependency Parsing (Al-Rfou et al., 2013; 2015; Mikolov et al., 2013a; b; Chen & Manning, 2014). Extensive research has been done to tune these embeddings to various tasks by incorporating features such as character (compositional) information, word order information, and multi-word (phrase) information (Ling et al., 2015; Mikolov et al., 2013c; Zhang et al., 2015; Trask et al., 2015).

Despite these advancements, most word embedding techniques share a common problem in that each word must encode all of its potential meanings into a single vector (Huang et al., 2012). For words with multiple meanings (or ”senses”), this creates a superposition in vector space where a vector takes on a mixture of its individual meanings. In this work, we will show that this superposition obfuscates the context specific meaning of a word and can have a negative effect on NLP classifiers leveraging the superposition as input data. Furthermore, we will show that disambiguating multiple word senses into separate embeddings alleviates this problem and the corresponding confusion to an NLP model.

Related Work

Mikolov et al. (2013a) proposed two simple methods for learning continuous word embeddings using neural networks based on Skip-gram or Continuous-Bag-of-Word (CBOW) models and named it word2vec. Word vectors built from these methods map words to points in space that effectively encode semantic and syntactic meaning despite ignoring word order information. Furthermore, the word vectors exhibited certain algebraic relations, as exemplified by example: ”v[man] - v[king] + v[queen] $\approx$ v[woman]”. Subsequent work leveraging such neural word embeddings has proven to be effective on a variety of natural language modeling tasks (Al-Rfou et al., 2013; 2015; Chen & Manning, 2014).

2 Wang2vec

Because word embeddings in word2vec are insensitive to word order, they are suboptimal when used for syntactic tasks like POS tagging or dependency parsing. Ling et al. (2015) proposed modifications to word2vec that incorporated word order. Consisting of structured skip-gram and continuous window methods that are together termed wang2vec, these models demonstrate significant ability to model syntactic representations. They come, however, at the cost of computation speed. Furthermore, because words have a single vector representation in wang2vec, the method is unable to model polysemic words with multiple meanings. For instance, the word ”work” in the sentence ”We saw her work” can be either a verb or noun depending on the broader context in surrounding this sentence. This technique encodes the co-occurrence statistics for each sense of a word into one or more fixed dimensional embeddings, generating embeddings that model multiple uses of a word.

3 Statistical Multi-Prototype Vector-Space Models of Word Meaning

Perhaps a seminal work to vector-space word-sense disambiguation, the approach by Reisinger & Mooney (2010) creates a vector-space model that encodes multiple meanings for words by first clustering the contexts in which a word appears. Once the contexts are clustered, several prototype vectors can be initialized by averaging the statistically generated vectors for each word in the cluster. This process of computing clusters and creating embeddings based on a vector for each cluster has become the canonical strategy for word-sense disambiguation in vector spaces. However, this approach presents no strategy for the context specific selection of potentially many vectors for use in an NLP classifier.

4 Clustering Weighted Average Context Embeddings

Our technique is inspired by the work of Huang et al. (2012), which uses a multi-prototype neural vector-space model that clusters contexts to generate prototypes. Unlike Reisinger & Mooney (2010), the context embeddings are generated by a neural network in the following way: given a pre-trained word embedding model, each context embedding is generated by computing a weighted sum of the words in the context (weighted by tf-idf). Then, for each term, the associated context embeddings are clustered. The clusters are used to re-label each occurrence of each word in the corpus. Once these terms have been re-labeled with the cluster’s number, a new word model is trained on the labeled embeddings (with a different vector for each) generating the word-sense embeddings.

In addition to the selection problem and clustering overhead described in the previous subsection, this model also suffers from the need to train neural word embeddings twice, which is a very expensive endeavor.

5 Clustering Convolutional Context Embeddings

Recent work has explored leveraging convolutional approaches to modeling the context embeddings that are clustered into word prototypes. Unlike previous approaches, Chen et al. (2015) selects the number of word clusters for each word based on the number of definitions for a word in the WordNet Gloss (as opposed to other approaches that commonly pick a fixed number of clusters). A variant on the MSSG model of Neelakantan et al. (2015), this work uses the WordNet Glosses dataset and convolutional embeddings to initialize the word prototypes.

In addition to the selection problem, clustering overhead, and the need to train neural embeddings multiple times, this higher-quality model is somewhat limited by the vocabulary present in the English WordNet resource. Furthermore, the majority of the WordNet’s relations connect words from the same Part-of-Speech (POS). ”Thus, WordNet really consists of four sub-nets, one each for nouns, verbs, adjectives and adverbs, with few cross-POS pointers.”https://wordnet.princeton.edu/

The sense2vec Model

We expand on the work of Huang et al. (2012) by leveraging supervised NLP labels instead of unsupervised clusters to determine a particular word instance’s sense. This eliminates the need to train embeddings multiple times, eliminates the need for a clustering step, and creates an efficient method by which a supervised classifier may consume the appropriate word-sense embedding.

Given a labeled corpus (either by hand or by a model) with one or more labels per word, the sense2vec model first counts the number of uses (where a unique word maps set of one or more labels/uses) of each word and generates a random ”sense embedding” for each use. A model is then trained using either the CBOW, Skip-gram, or Structured Skip-gram model configurations. Instead of predicting a token given surrounding tokens, this model predicts a word sense given surrounding senses.

For subjective evaluation of these word embeddings, we trained models using several datasets for comparison. First, we trained using Word2vec’s Continuous Bag of Words command line params: -size 500 -window 10 -negative 10 -hs 0 -sample 1e-5 -iter 3 -min-count 10 approach on the large unlabeled corpus used for the Google Word Analogy Task the data.txt file generated from http://word2vec.googlecode.com/svn/trunk/demo-train-big-model-v1.sh. Several word embeddings and their closest terms measured by cosine similarity are displayed in Table 1 below.

In this table, observe that the ”bank” column is similar to proper nouns (”hsbc”, ”citibank”), verbs (”lending”,”banking”), and nouns (”banks”,”lender”). This is because the term ”bank” is used in 3 different ways, as a proper noun, verb, and noun. This embedding for ”bank” has modeled a mixture of these three meanings. ”apple”, ”so”, ”bad”, and ”perfect” can also have a mixture of meanings. In some cases, such as ”apple”, one interpretation of the word is completely ignored (apple the fruit). In the case of ”so”, there is also an interjection sense of ”so” that is not well represented in the vector space.

2 Subjective Evaluation - Part-of-Speech Disambiguation

For Part-of-Speech disambiguation, we labeled the dataset from section 3.1 with Part-of-Speech tags using the Polyglot Universal Dependency Part-of-Speech tagger of Al-Rfou et al. (2013) and trained sense2vec with identical parameters as section 3.1. In table 2, we see that this method has successfully disambiguated the difference between the noun ”apple” referring to the fruit and the proper noun ”apple” referring to the company. In table 3, we see that all three uses of the word ”bank” have been disambiguated by their respective parts of speech, and in table 4, nuanced senses of the word ”so” have also been disambiguated.

3 Subjective Evaluation - Sentiment Disambiguation

For Sentiment disambiguation, the IMDB labeled training corpus was labeled with Part-of-Speech tags using the Polyglot Part-of-Speech tagger from Al-Rfou et al. (2013). Adjectives were then labeled with the positive or negative sentiment associated with each comment. A CBOW sense2vec model was then trained on the resulting dataset, disambiguating between both Part-of-Speech and Sentiment (for adjectives).

Table 5 shows the difference between the positive and negative vectors for the word ”bad”. The negative vector is most similar to word indicating the classical meaning of bad (including the negative version of ”good”, e.g. ”good grief!”). The positive ”bad” vector denotes a tone of sarcasm, most closely relating to the positive sense of ”good” (e.g. ”good job!”).

Table 6 shows the positive and negative senses of the word ”perfect”. The positive version of the word clusters most closely with words indicating excellence. The positive version clusters with the more sarcastic interpretation.

Named Entity Resolution

To evaluate the embeddings when disambiguating on named entity resolution (NER), we labeled the standard word2vec dataset from section 3.2 with named entity labels. This demonstrated how sense2vec can also disambiguate between multi-word sequences of text as well as single word sequences of text. Below, we see that the word ”Washington” is disambiguated with both a PERSON and a GPE sense of the word. Furthermore, we see that Hillary Clinton is very similar to titles that she has held within the time span of the dataset.

Neural Dependency Parsing

To quantitatively evaluate disambiguated sense embeddings relative to the current standard, we compared sense2vec embeddings and wang2vec embeddings on neural syntactic dependency parsing tasks in six languages. First, we trained two sets of embeddings on the Bulgarian, German, English, French, Italian, and Swedish Wikipedia datasets from the Polyglot websitehttps://sites.google.com/site/rmyeid/projects/polyglot. The baseline embeddings were trained without any Part-of-Speech disambiguation using the structured skip-gram approach of Ling et al. (2015). For each language, the sense2vec embeddings were trained by disambiguating terms using the language specific Polyglot Part-of-Speech tagger of Al-Rfou et al. (2013), and embedded in the same structured skip-gram approach. Both were trained using identical parametrization command line params: -size 50 -window 5 -negative 10 -hs 0 -sample 1e-4 -iter 5 -cap 0.

Each of these embeddings was used to train a dependency parse model using the parser outlined in (Chen & Manning, 2014). All were trained on the the respective language’s Universal Dependencies treebank. The standard splits were used.The German, French, and Italian treebanks had occasional tokens that both spanned multiple indices and overlapped with the index of the previous and following token (ex. 0, 0-1, 1,…), a property which is incompatible with the (Chen & Manning, 2014) parser. These tokens were removed. If their removal created a malformed tree, the sentence was removed automatically by the parser and logged accordingly. For the parser trained on the sense2vec emeddings, the POS specific embedding was used as the input. The Part-of-Speech label was determined using the gold-standard POS tags from the treebank. It should be noted that the parser of (Chen & Manning, 2014) uses trained Part-of-Speech embeddings as input which are indexed based on gold-standard POS tags. Thus, differences in quality between parsers trained on the two embedding styles are due to clarity in the word embeddings as opposed to the addition of Part-of-Speech information because both model styles train on gold standard POS information. For each language, the Unlabeled Attachment Scores are outlined in Table 7.

The ”Error Margin” section of table 7 describes the percentage reduction in error for each language. Disambiguating based on Part-of-Speech using sense2vec reduced the error in all six languages with an average reduction greater than 8%.

Conclusion and Future Work

In this work, we have proposed a new model for word sense disambiguation that uses supervised NLP labeling to disambiguate between word senses. Much like previous models, it leverages a form of context clustering to disambiguate the use of a term. However, instead of using unsupervised clustering methods, our approach clusters using supervised labels which can analyze a specific word’s context and assign a label. This significantly reduces the computational overhead of word-sense modeling and provides a natural mechanism for other NLP tasks to select the appropriate sense embedding. Furthermore, we show that disambiguated embeddings can increase the accuracy of syntactic dependency parsing in a variety of languages. Future work will explore how disambiguated embeddings perform using other varieties of supervised labels and consuming NLP tasks.

References