GM-CTSC at SemEval-2020 Task 1: Gaussian Mixtures Cross Temporal Similarity Clustering

Introduction

The recent development in word embeddings, and their increasing capability to capture lexical semantics has inspired the application of these methods to new tasks and introduced new challenges. The diachronic analysis of language is one of such linguistic tasks that has benefited from the advantages of these new methods, i.e. the capability to build semantic representations of words by skimming through large corpora spanning multiple time periods. SemEval 2020 Task 1 [Schlechtweg et al., 2020] addresses the current lack of a systematic approach for the evaluation of automatic methods for the diachronic analysis by proposing a common evaluation framework that comprises two tasks and covers four different languages (German, English, Latin, and Swedish). Given two corpora $C_{1}$ and $C_{2}$ for two periods $t_{1}$ and $t_{2}$, Subtask 1 requires participants to classify a set of target words in two categories: words that have lost or gained senses from $t_{1}$ to $t_{2}$ and words that did not, while Subtask 2 requires participants to rank the target words according to their degree of lexical semantic change between the two periods. We tackle the problem of automatically detecting lexical semantic changes with approaches that rely on temporal word embeddings. These approaches create a word vector representation for each time period by exploiting a shared semantic space. Similarity measures can then be used to capture the extent of a word semantic change between two time lapses. Some temporal word embedding techniques adopt a two-step approach, where they first learn separate word embeddings for each time period and then align the word vectors across multiple time periods [Hamilton et al., 2016]. Other dynamic approaches incorporate the alignment directly into the learning stage via the optimisation function [Tahmasebi et al., 2018]. Dynamic word embeddings can be further categorised according to the constraint imposed on the alignment. The explicit alignment adopts a conservative approach to the semantic drift that a word can undergo by posing a limit to the distance between the word vectors belonging to the two temporal spaces. In the implicit alignment, there is no need for explicit constraint since the alignment is automatically performed by sharing the same word context vectors across all the time periods.

In this work, we focus on dynamic word embeddings by exploring methods based on both explicit, such as Dynamic Word2Vec [Yao et al., 2018], and implicit alignment, namely Temporal Random Indexing [Basile et al., 2015] and Temporal Referencing [Dubossarsky et al., 2019]. We analyse the use of different similarity measures to determine the extent of a word semantic change and compare the cosine similarity with Pearson Correlation and the neighborhood similarity [Shoemark et al., 2019]. While these similarity measures can be directly employed to generate a ranked list of words for Subtask 2, their adoption in Subtask 1 requires further manipulation. We introduce a new method to classify changing vs. stable words by clustering the target similarity distributions via Gaussian Mixture Models. We describe the embedding models and the clustering algorithm in Section 2, while Section 3 provides details about the hyper-parameter selection. Section 4 reports the results of the task evaluation followed by some concluding remarks in Section 5.

GM-CTSC

Dynamic Word2Vec (DW2V) [Yao et al., 2018] simultaneously learns time-aware embeddings by aligning and reducing the dimensionality of time-binned Positive Point-wise Mutual Information matrices.

Temporal Random Indexing (TRI) [Basile et al., 2015] implicitly aligns co-occurrence matrices by using the same random projection for all the temporal bins.

Collocations extracts for each word and each time period the set of relevant collocations through Dice score. As similarity function, we measure the cosine similarity between the sets of collocations belonging to the two different time periods. More details are reported in ?).

Temporal Referencing (TR) [Dubossarsky et al., 2019] used only in the post-evaluation, it consists in a modified version of Word2Vec Skipgram that adds a temporal referencing to target vectors, keeping context vectors unchanged.

A similarity measure between vectors in the two temporal spaces is adopted to compute the extent of the semantic drift of the target words. We explored several similarity measures:

Cosine similarity (CS) is the cosine of the angle between two vectors.

Pearson correlation (PC) measures the linear correlation between two variables, in case of centred vectors (with zero means) is equivalent to the cosine similarity.

Neighborhood similarity (NS) computes two $k$-neighbour sets $nbrs_{k}(E_{1}(w))$ and $nbrs_{k}(E_{2}(w))$ and the union set $\mathcal{U}=nbrs_{k}(E_{1}(w))\cup nbrs_{k}(E_{2}(w))$. Two second-order vectors, one for each word representation $u_{j}$, are created. The components of $u_{i}$ are the cosine similarity between the vector $v_{j}$Where $v_{j}$ is the vector representation for the word generated by $E_{j}$ and $j$ is the time period. and the i-th element of $\mathcal{U}$: $u_{j_{i}}=cos(v_{j},\mathcal{U}(i))$. The Neighborhood similarity is the cosine similarity between the second-order vectors. In all the experiments we set $k=25$.

In Subtask 2, we use one of the three similarity measures ($CS$, $PC$, $NS$) to compute the set of target similarities $\mathcal{S}=\{sim(E_{1}(w),E_{2}(w))\mid w\in T\}$. Then, we rank the target words according to the distance, computed as: $1-\mid sim(E_{1}(w),E_{2}(w))\mid$.

2 Subtask 1: Gaussian Mixture Clustering

Subtask 1 requires a further step: given $\mathcal{S}$, the set of target similarities, we need to predict the target labels. The aim is to assign either of the two classes, 0 (stable) or 1 (change), to each target word of a given language. Once we compute the set of target similarities $\mathcal{S}$, we want to find a way to assign the corresponding label. We assume that low similarities suggest changing words and high similarities indicate stable words.

Gaussian Mixture Models (GMMs) allow to build probabilistic models for representing the Gaussian distribution of stable and changed targets. We use GMMshttps://scikit-learn.org/stable/modules/generated/sklearn.mixture.GaussianMixture.html to model the density of the distributions of the similarities of targets as a weighted sum of two Gaussian densities [Huang et al., 2017]:

where $M$ is the number of mixture components, $\phi(\mathcal{S}|\mu_{m},\Sigma_{m})$ is the Gaussian density with mean vector $\mu_{m}$ and covariance matrix $\Sigma_{m}$, and $\pi_{m}$ is the prior probability for the $m$-th component. Additional constraints can be applied to the covariance matrix in Eq. 1. In our experiments, we allow each component to have its own covariance matrix.

For our purpose, we speculate that the distribution of target similarities is a mixture of two densities, i.e. representing the stable and changing words. Consequently, we fixed the number of the mixture components in the GMMs to two. We initially randomly assign a label (stable/changing) to each density distribution. Let $\mu_{0}$ and $\mu_{1}$ be the means of the two Gaussians associated with the “stable” and “changing” labels respectively. If $\mu_{0}<\mu_{1}$ (i.e. the similarity mean of the distribution labelled as “stable” is lower than the mean of distribution labelled as “changing”), we invert the labels. Alg. 1 can be used for properly label each word of the target vocabulary.

In order to set the best parameters for each language and model, we rely on the GMMs log likelihood, which is generally used for estimating the clusters quality:

Experimental Setup

In all the runs, we do not pre-process data and we use a context window size of 5 while analyzing sentences. The $TR$ modelWe add this model during the post-evaluation. has been adopted into its original implementationhttps://github.com/Garrafao/TemporalReferencing, as the $TRI$https://github.com/pippokill/tri approach and $DW2V$https://github.com/yifan0sun/DynamicWord2Vec one. For runs involving $TRI$, we experimented with a varying vector size from $200$ to $1,000$. Moreover, we investigated (1) the initialization of the count matrix at time $j$ with the matrix at time $j-1$, (2) the contribution of positive-only projections, and (3) the application of PPMI weights, as explained in ?). For $DW2V$, we use the parameter setting proposed in ?). We set $\lambda=10$, $\tau=50$, $\gamma=100$, $\rho=50$ and experimented with a number of iterations from one to five. As vocabulary, we kept the top 50,000 most frequent tokens for both $TRI$ and $DW2V$. In the $TR$ runs, we set the vector size to $100$, and we experimented eight iterations for English and Latin, and four for German and Swedish. We use $20$ negative samples, keeping only the tokens that occur at least $10$ times. All the other parameters used for configuring the models are reported in Tab. 1.

Results

Tab. 2 reports the main results obtained by the different models. It shows the results obtained from the official submissions at the challenge and the results obtained by the $TR$ approach performed during the post-evaluation phase. The results obtained for the Subtask 1 are reported using the accuracy metric, while for the Subtask 2, the Spearman’s rank-order correlation coefficients are used.

Considering the results of the evaluation phase, the models show not consistent behaviors. $TRI$ showed the best performance when considering “all the languages” for both Subtasks, although in Subtask 1 it is not able to overcome Baseline2. Focusing on Subtask 1, if we consider each language in isolation, we see that $DW2V$ gives the best results for EnglishPlease, note that for EN, LA and SW $Overall_{CS}$ and $DW2V$ coincide while $Overall_{PC}$ (Collocation with cosine similarity) is our best system for German language, although it is not able to overcome Baseline2. $TRI$ is the best system for Latin, although outperformed by Baseline1, and Sweden languages. In Subtask 2, the best English score was reported by $Overall_{NS}$. Simlarly to Subtask 1, $Overall_{CS}$ performed the best in German language. For Latin and Sweden, $TRI$ provided the best results, and interestingly, it is one of the few systems that did not generate a negative correlation. For Sweden language in particular, it is interesting to notice that $TRI$ generated the best result among all the task participants.

At the end of the challenge, when the labelled test set was released, we performed more experiments reported in the post-evaluation row. In this phase we run an additional system, $TR$, which outperformed all the previous reported approaches, including both baselines. The only exception is for Latin, in which for Subtask 1 $Baseline1$ achieves $0.650$ accuracy in comparison to $0.525$ of $TR$. Comparing $TR$ and $TRI$, which are both based on implicit alignment, the former is a prediction-based model while the is latter a count-based one. Moreover, $TR$ creates a temporal word embedding only for the target words rather than for the whole vocabulary. Consequently, this results in better word embeddings for all the words in the vocabulary that do not have a temporal reference. These differences allow $TR$ to achieve better results than the other models.

During the post-evaluation we decided to investigate also the role of GMMs for class labeling (Sec. 2). We compared GMMs with semi-manual thresholds $\mu_{\mathcal{S}}$, $\mu_{\mathcal{S}}-\sigma_{\mathcal{S}}$, $\mu_{\mathcal{S}}+\sigma_{\mathcal{S}}$ and Winsorizing [Kokic and Bell, 1994] computing $\mu_{\boldsymbol{S}}$ and $\sigma_{\boldsymbol{S}}$ on data provided for Subtask 1, where $\mu_{\mathcal{S}}$, $\sigma_{\mathcal{S}}$ are the mean and the standard deviation computed on the similarity set $\mathcal{S}$. Figure 1 reports the different accuracy scores obtained by the five methods for the $TRI$, $Collocation$, $DW2V$, $TR$ approaches. The scores for the GMMs strategy are close to those obtained by $\mu_{\mathcal{S}}$ for TRI and Collocation. While GMMs outperforms $\mu_{\mathcal{S}}+\sigma_{\mathcal{S}}$ in every run, $\mu_{\mathcal{S}}-\sigma_{\mathcal{S}}$ seems to work better than GMMs except that in $TR$. Winsorizing work better in $TRI$ and $Collocation$ than GMMs. GMMs outperforms Winsorizing in $DW2V$ and $TR$. These results are not clear enough to advocate for a specific threshold. Consequently, further analysis will be part of future work in order to understand what is the better threshold that could be included in the GMMs process.

Conclusions

We described the runs we submitted to the SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection. This paper has two main contributions. We reported a comparison of some of the most recent approaches to model lexical semantic change with temporal word embeddings, and we experimented with an automatic unsupervised procedure to classify changing and stable words. Results show that implicit alignment works generally better in modelling the lexical semantic change. In future works we plan to carry out an analysis on unlemmatised corpora and gauge a better understanding of the impact of Gaussian Mixture Clustering for unsupervised lexical semantic change detection.

References