Cultural Shift or Linguistic Drift? Comparing Two Computational Measures of Semantic Change

Introduction

Distributional methods of embedding words in vector spaces according to their co-occurrence statistics are a promising new tool for diachronic semantics [Gulordava and Baroni, 2011, Jatowt and Duh, 2014, Kulkarni et al., 2014, Xu and Kemp, 2015, Hamilton et al., 2016]. Previous work, however, does not consider the underlying causes of semantic change or how to distentangle different types of change.

We show how two computational measures can be used to distinguish between semantic changes caused by cultural shifts (e.g., technological advancements) and those caused by more regular processes of semantic change (e.g., grammaticalization or subjectification). This distinction is essential for research on linguistic and cultural evolution. Detecting cultural shifts in language use is crucial to computational studies of history and other digital humanities projects. By contrast, for advancing historical linguistics, cultural shifts amount to noise and only the more regular shifts matter.

Our work builds on two intuitions: that distributional models can highlight syntagmatic versus paradigmatic relations with neighboring words [Schutze and Pedersen, 1993] and that nouns are more likely to undergo changes due to irregular cultural shifts while verbs more readily participate in regular processes of semantic change [Gentner and France, 1988, Traugott and Dasher, 2001]. We use this noun vs. verb mapping as a proxy to compare our two measures’ sensitivities to cultural vs. linguistic shifts. Sensitivity to nominal shifts indicates a propensity to capture irregular cultural shifts in language, such as those due to technological advancements [Traugott and Dasher, 2001]. Sensitivity to shifts in verbs (and other predicates) indicates a propensity to capture regular processes of linguistic drift [Gentner and France, 1988, Kintsch, 2000, Traugott and Dasher, 2001].

The first measure we analyze is based upon changes to a word’s local semantic neighborhood; we show that it is more sensitive to changes in the nominal domain and captures changes due to unpredictable cultural shifts. Our second measure relies on a more traditional global notion of change; we show that it better captures changes, like those in verbs, that are the result of regular linguistic drift.

Our analysis relies on a large-scale statistical study of six historical corpora in multiple languages, along with case-studies that illustrate the fine-grained differences between the two measures.

Methods

We use the diachronic word2vec embeddings constructed in our previous work [Hamilton et al., 2016] to measure how word meanings change between consecutive decades.http://nlp.stanford.edu/projects/histwords/. This URL also links to detailed dataset descriptions and the code needed to replicate the experiments in this paper. In these representations each word $w_{i}$ has a vector representation $\mathbf{w}^{(t)}$ [Turney and Pantel, 2010] at each time point, which captures its co-occurrence statistics for that time period. The vectors are constructed using the skip-gram with negative sampling (SGNS) algorithm [Mikolov et al., 2013] and post-processed to align the semantic spaces between years. Measuring the distance between word vectors for consecutive decades allows us to compute the rate at which the different words change in meaning [Gulordava and Baroni, 2011].

We analyzed the decades from 1800 to 1990 using vectors derived from the Google N-gram datasets [Lin et al., 2012] that have large amounts of historical text (English, French, German, and English Fiction). We also used vectors derived from the Corpus of Historical American English (COHA), which is smaller than Google N-grams but was carefully constructed to be genre balanced and contains word lemmas as well as surface forms [Davies, 2010]. We examined all decades from 1850 through 2000 using the COHA dataset and used the part-of-speech tags provided with the corpora.

We examine two different ways to measure semantic change (Figure 1).

The first measure analyzes global shifts in a word’s vector semantics and is identical to the measure used in most previous works [Gulordava and Baroni, 2011, Jatowt and Duh, 2014, Kim et al., 2014, Hamilton et al., 2016]. We simply take a word’s vectors for two consecutive decades and measure the cosine distance between them, i.e.

Local neighborhood measure

The second measure is based on the intuition that only a word’s nearest semantic neighbors are relevant. For this measure, we first find word $w_{i}$’s set of $k$ nearest-neighbors (according to cosine-similarity) within each decade, which we denote by the ordered set $\mathcal{N}_{k}(w^{(t)}_{i})$. Next, to measure the change between decades $t$ and $t+1$, we compute a “second-order” similarity vector for $w^{(t)}_{i}$ from these neighbor sets with entries defined as

and we compute an analogous vector for $w_{i}^{(t+1)}$. The second-order vector, $\mathbf{s}^{(t)}_{i}$, contains the cosine similarity of $\mathbf{w}_{i}$ and the vectors of all $w_{i}$’s nearest semantic neighbors in the the time-periods $t$ and $t+1$. Working with variants of these second-order vectors has been a popular approach in many recent works, though most of these works define these vectors against the full vocabulary and not just a word’s nearest neighbors [del Prado Martin and Brendel, 2016, Eger and Mehler, 2016, Rodda et al., 2016].

Finally, we compute the local neighborhood change as

This measures the extent to which $w_{i}$’s similarity with its nearest neighbors has changed.

The local neighborhood measure defined in (3) captures strong shifts in a word’s paradigmatic relations but is less sensitive to global shifts in syntagmatic contexts [Schutze and Pedersen, 1993]. We used $k=25$ in all experiments (though we found the results to be consistent for $k\in$).

2 Statistical methodology

To test whether nouns or verbs change more according to our two measures of change, we build on our previous work and used a linear mixed model approach [Hamilton et al., 2016]. This approach amounts to a linear regression where the model also includes “random” effects to account for the fact that the measurements for individual words will be correlated across time [McCulloch and Neuhaus, 2001].

We ran two regressions per datatset: one with the global $d^{G}$ values as the dependent variables (DVs) and one with the local neighborhood $d^{L}$ values. In both cases we examined the change between all consecutive decades and normalized the DVs to zero-mean and unit variance. We examined nouns/verbs within the top-10000 words by frequency rank and removed all words that occurred ${<}500$ times in the smaller COHA dataset. The independent variables are word frequency, the decade of the change (represented categorically), and variable indicating whether a word is a noun or a verb (proper nouns are excluded, as in ?).Frequency was included since it is known to strongly influence the distributional measures [Hamilton et al., 2016].

Results

Our results show that the two seemingly related measures actually result in drastically different notions of semantic change.

The local neighborhood measure assigns far higher rates of semantic change to nouns across all languages and datasets while the opposite is true for the global distance measure, which tends to assign higher rates of change to verbs (Figure 2).

We focused on verbs vs. nouns since they are the two major parts-of-speech and previous research has shown that verbs are more semantically mutable than nouns and thus more likely to undergo linguistic drift [Gentner and France, 1988], while nouns are far more likely to change due to cultural shifts like new technologies [Traugott and Dasher, 2001]. However, some well-known regular linguistic shifts include rarer parts of speech like adverbs (included in our case studies below). Thus we also confirmed that the differences shown in Figure 2 also hold when adverbs and adjectives are included along with the verbs. This modified analysis showed analogous significant trends, which fits with previous research arguing that adverbial and adjectival modifiers are also often the target of regular linguistic changes [Traugott and Dasher, 2001].

The results of this large-scale regression analysis show that the local measure is more sensitive to changes in the nominal domain, a domain in which change is known to be driven by cultural factors. In contrast, the global measure is more sensitive to changes in verbs, along with adjectives and adverbs, which are known to be the targets of many regular processes of linguistic change [Traugott and Dasher, 2001, Hopper and Traugott, 2003]

2 Case studies

We examined six case-study words grouped into two sets. These case studies show that three examples of well-attested regular linguistic shifts (set A) changed more according to the global measure, while three well-known examples of cultural changes (set B) change more according to the local neighborhood measure. Table 2 lists these words with some representative historical contexts [Davies, 2010].

Set A contains three words that underwent attested regular linguistic shifts detailed in ?): actually, must, and promise. These three words represent three different types of regular linguistic shifts: actually is a case of subjectification (detailed in Figure 1); must shifted from a deontic/obligation usage (“you must do X”) to a epistemic one (“X must be the case”), exemplifying a regular pattern of change common to many modal verbs; and promise represents the class of shifting “performative speech acts” that undergo rich changes due to their pragmatic uses and subjectification [Traugott and Dasher, 2001]. The contexts listed in Table 2 exemplify these shifts.

Set B contains three words that were selected because they underwent well-known cultural shifts over the last 150 years: gay, virus, and cell. These words gained new meanings due to uses in community-specific vernacular (gay) or technological advances (virus, cell). The cultural shifts underlying these changes in usage — e.g., the development of the mobile “cell phone” — were unpredictable in the sense that they were not the result of regularities in human linguistic systems.

Figure 3 shows how much the meaning of these word changed from the 1850s to the 1990s according to the two different measures on the English Google data. We see that the words in set A changed more when measurements were made using the global measure, while the opposite holds for set B.

Discussion

Our results show that our novel local neighborhood measure of semantic change is more sensitive to changes in nouns, while the global measure is more sensitive to changes in verbs. This mapping aligns with the traditional distinction between irregular cultural shifts in nominals and more regular cases of linguistic drift [Traugott and Dasher, 2001] and is further reinforced by our six case studies.

This finding emphasizes that researchers must develop and use measures of semantic change that are tuned to specific tasks. For example, a cultural change-point detection framework would be more successful using our local neighborhood measure, while an empirical study of grammaticalization would be better off using the traditional global distance approach. Comparing measurements made by these two approaches also allows researchers to assess the extent to which semantic changes are linguistic or cultural in nature.

Acknowledgements

The authors thank C. Manning, V. Prabhakaran, S. Kumar, and our anonymous reviewers for their helpful comments. This research has been supported in part by NSF CNS-1010921, IIS-1149837, IIS-1514268 NIH BD2K, ARO MURI, DARPA XDATA, DARPA SIMPLEX, Stanford Data Science Initiative, SAP Stanford Graduate Fellowship, NSERC PGS-D, Boeing, Lightspeed, and Volkswagen.

References