GloVeInit at SemEval-2020 Task 1: Using GloVe Vector Initialization for Unsupervised Lexical Semantic Change Detection

Introduction and Background

Lexical Semantic Change (LSC) Detection is an active research topic in the field of natural language processing, and has been applied for diachronic (across time) and synchronic (across domains) tasks [Schlechtweg et al., 2019]. Previously limited to manual ”close-reading” approaches, the availability of large-scale corpora have allowed the use of computational methods for this task [Tahmasebi et al., 2018]. This topic has found various applications in various disciplines such as improving information retrieval from historical documents [Morsy and Karypis, 2016], preventing cross-domain ambiguity in requirements elicitation interviews [Jain et al., 2020], and studying the impact of societal and cultural changes on word meanings and usage [Tahmasebi and Risse, 2017].

LSC detection involves the use of two corpora $C_{1}$ and $C_{2}$ which, in the diachronic case, belong to different time periods $t_{1}$ and $t_{2}$ respectively. The various approaches found in the literature usually involve the construction of a word embedding space specific to each corpus. The embeddings can be constructed through count-based methods such as Positive Pointwise Mutual Information and Random Indexing, or predictive methods such as Skip-Gram with Negative Sampling (SGNS). Most of these models are stochastic in nature which means that separately trained embedding models live in their own space. In order to project them onto a unified space, alignment techniques such as vector initialization and orthogonal Procrustes are used. The LSC of a word is then quantitatively determined by measuring the contextual dissimilarity between the word’s representations [Tahmasebi et al., 2018].

Recently, there have been efforts to evaluate these various methods by comparing their results with manually-annotated data [Schlechtweg et al., 2019, Ahmad et al., 2020]. The SemEval-2020 Task 1 is one such effort which is based on LSC detection in corpora of English, German, Latin, and Swedish languages [Schlechtweg et al., 2020]. Its aim is to provide an evaluation framework for unsupervised LSC detection systems by comparing their results against a ground truth, as annotated by native speakers or scholars. It consists of two subtasks: Given two corpora $C_{1}$ and $C_{2}$ (for time periods $t_{1}$ and $t_{2}$), for a set of target words,

decide which words lost or gained senses between $t_{1}$ and $t_{2}$, and which ones did not; as annotated by human judges.

rank them according to their degree of LSC between $t_{1}$ and $t_{2}$ as annotated by human judges. A higher rank means stronger change.

System Overview

The Vector Initialization (VI) alignment method, which was first used by ?), involves the embedding space for $t_{1}$ to be trained independently on $C_{1}$. It is then used to initialize the embedding space for $t_{2}$ which is then subsequently trained from $C_{2}$. The underlying idea is that the embedding for a word $w$ will get considerably updated if it is used within different contexts in $C_{1}$ and $C_{2}$, otherwise it will receive only a slight update. A study by ?), which applied the VI method on SGNS embeddings, found it to perform significantly weaker on LSC detection tasks than other methods such as orthogonal Procrustes (OP). They attributed this to the sensitivity of the VI method to the frequency of a word in $C_{2}$. The high frequency of a word in $C_{2}$ will result into the word’s embedding getting frequent updates away from its initial state. This leads to a large divergence between the word’s representations even if it does not undergo any LSC from $t_{1}$ to $t_{2}$.

In a recent shared task on LSC detection in the German language, a modified VI approach achieved the third-best result and outperformed certain OP approaches [Ahmad et al., 2020]. Instead of only initializing on the word embeddings obtained from $C_{1}$, the modified approach initializes on the complete SGNS model which includes the hidden layer. However, it also theoretically suffers from sensitivity to high frequency.

2 Comparison of GloVe and SGNS

GloVe (Global Vectors) and SGNS are unsupervised algorithms for obtaining word embedding spaces [Pennington et al., 2014, Mikolov et al., 2013]. GloVe makes use of co-occurrence matrix $X$; its $(i,j)$ entry, $X_{ij}$ is the number of times the word $w_{j}$ appears in the context of the word $w_{i}$ (as defined by the window-size $L$). It trains the word embeddings by minimizing the cost function

On the other hand, SGNS is a predictive model which relies on a shallow two-layer neural network which, given a word, predicts the set of its context words. To avoid the expensive softmax function in the training objective, negative sampling is used by drawing a few negative samples from a noise distribution.

An important distinction between GloVe and SGNS from this paper’s point of view is the number of updates that are made to a word embedding during training. In a single epoch, the number of updates to the SGNS embedding of a word is roughly equal to the number of words that appear in its context throughout the corpus. Hence, the number of updates is proportional to the frequency of the word which has an upper-bound of the total number of words in the corpus. However, this relationship is not exactly linear because of downsampling of frequent words and negative sampling. In the GloVe model, the number of updates received in a single epoch is equal to the number of distinct context words. This number is limited by an upper-bound of vocabulary size $V$, which is usually much less than the total number of words in the corpus.

As discussed in Section 2.1, the VI method can falsely give a high LSC score to words with high frequency. Our hypothesis is that the GloVe model is more suited for the VI method due to its lesser sensitivity to high frequency. To give an indication of the extent of this difference, the relationship between a word’s frequency and the number of updates to its embedding is depicted in Figure 1 for both GloVe and SGNS models trained on the $C_{2}$ corpora of all the four languages. These results are based on the official GloVe implementationgithub.com/stanfordnlp/GloVe and the gensimradimrehurek.com/gensim implementation of SGNS. The models were trained for a single epoch and only words with a frequency of greater than or equal to 5 were considered. It is clear that the proportion of updates for frequent and rare words is comparatively more balanced in the case of GloVe. On the other hand, there is a high rate of growth in the number of updates with respect to the frequency in the case of SGNS, which indicates that it is biased towards estimating a high LSC for frequent words.

Results and Experimental Setup

The dataset provided by the competition organizers consists of the corpora $C_{1}$ and $C_{2}$, and the list of target words for four languages: English, German, Latin, and Swedish [Schlechtweg et al., 2020]. All of the corpora were already prepossessed: they were in tokenized and lemmatized form with punctuation marks and one-word sentences removed.

Subtask 1 is concerned with identifying the loss or gain of one or more word sense(s), where subtask 2 tests a model’s ability to detect fine-grained changes in the two sense frequency distributions. For example, consider the word cell which historically referred to either a chamber or the smallest unit of an organism. However, the relative usages of both these senses have decreased with time, with mobile phone being the predominant meaning now. For subtask 2, the ground-truth ranking of the target words is determined by calculating the Jensen-Shannon divergence between their normalized sense frequency distributions from $t_{1}$ and $t_{2}$ [Donoso and Sánchez, 2017]. A submission is scored by its Spearman’s rank-order correlation coefficient against the ground-truth ranking.

We used cosine distance as the metric to calculate the distance between a word’s vectors in diachronic spaces, and every word with a distance of more than 0.55 (for German) or 0.45 (for other languages) were classified to have gained of lost word sense(s).

In all of our experiments, only words having a frequency of at least 5 were considered. Our best-performing solution during the evaluation phase achieved an accuracy of 60% in subtask 1 and a score of 0.352. It was obtained by training word embeddings with a dimensionality of 50 on both $C_{1}$ and $C_{2}$ for 60 epochs each with a window-size of 10. The subtask-2 score was close to but less than the modified SGNS-based VI approach discussed in Section 2.1, which was proposed by the team IMS and received a score of 0.372. We ranked 13th and 10th out of 33 participants in the two subtasks respectively.

Further experiments on subtask-2 were conducted after the competition for analyzing the impact of hyperparameters like window-size $L$ and embedding dimensionality $d$ and are reported in Table 1. Number of training epochs was limited to 20 for models with window-size 10 because of their high computational requirements. The results suggest that training the models for higher number of epochs can produce better results if the embedding dimensionality is high, but can backfire in the opposite case. Using a larger window-size improves the average result in most cases.

We defined the ranking error for each target word as the difference between its predicted rank and its true rank divided by the total number of target words. It lies in the range $(-1,1)$ and has an ideal value of zero. A positive value indicates that the model overestimated the LSC for a word, and a negative value indicates otherwise. A regression plot between the relative frequency of the target words and their ranking error as per our best performing model is depicted in Figure 2. A word’s relative frequency is defined as its frequency divided by the total frequency of all words in the corpus.

There is a statistically significant correlation between the ranking error and the relative frequency, with the result of the Spearman’s rank-order correlation test being $\rho=0.186$ and $p=0.020$. This indicates that the GloVe model’s relative insensitivity to high frequency can lead to it assigning a rank lower than the true rank to such words. We believe that the frequency of the target words is not large enough for SGNS’ sensitivity to high frequency to have a major impact on the results. This explains our model’s inferior performance as compared to SGNS-based model in this task.

Conclusion and Future Work

In this paper, we reported our work in the SemEval2020 Task 1. We proposed a GloVe-based VI approach which achieved the 11th rank out of 32 participating teams. We gave a theoretical reasoning behind why GloVe is less sensitive towards high frequency than SGNS and thus more suited for VI method, and empirically showed the magnitude of the difference between the number of updates to word embeddings in the two models. We believe that despite a lower-than-expected performance in this competition, our work presents a good case for the suitability of GloVe model when corpora of larger size are involved. However, proving this quantitatively is a challenging task because of the limitations associated with manual annotation-based evaluation.

Planned future work includes the study of how techniques such as dimension-wise mean-centering and length normalization, which have proved beneficial in OP-based approaches, can be applied for VI method.

Acknowledgments

We would like to thank the task organizers for preparing the dataset and organizing the competition, Jens Kaiser (member of the team IMS) from University of Stuttgart for sharing the results from his work on the SGNS-based VI alignment method, and the anonymous reviewers for their detailed and helpful feedback.

References