A Latent Variable Model Approach to PMI-based Word Embeddings

Introduction

Vector representations of words (word embeddings) try to capture relationships between words as distance or angle, and have many applications in computational linguistics and machine learning. They are constructed by various models whose unifying philosophy is that the meaning of a word is defined by “the company it keeps” Firth (1957), namely, co-occurrence statistics. The simplest methods use word vectors that explicitly represent co-occurrence statistics. Reweighting heuristics are known to improve these methods, as is dimension reduction Deerwester et al. (1990). Some reweighting methods are nonlinear, which include taking the square root of co-occurrence counts Rohde et al. (2006), or the logarithm, or the related Pointwise Mutual Information (PMI) Church and Hanks (1990). These are collectively referred to as Vector Space Models, surveyed in Turney and Pantel (2010).

Neural network language models Rumelhart et al. (1986; 1988); Bengio et al. (2006); Collobert and Weston (2008a) propose another way to construct embeddings: the word vector is simply the neural network’s internal representation for the word. This method is nonlinear and nonconvex. It was popularized via word2vec, a family of energy-based models in Mikolov et al. (2013b; c), followed by a matrix factorization approach called GloVe Pennington et al. (2014). The first paper also showed how to solve analogies using linear algebra on word embeddings. Experiments and theory were used to suggest that these newer methods are related to the older PMI-based models, but with new hyperparameters and/or term reweighting methods Levy and Goldberg (2014b).

But note that even the old PMI method is a bit mysterious. The simplest version considers a symmetric matrix with each row/column indexed by a word. The entry for $(w,w^{\prime})$ is $\mbox{PMI}(w,w^{\prime})=\log\frac{p(w,w^{\prime})}{p(w)p(w^{\prime})}$, where $p(w,w^{\prime})$ is the empirical probability of words $w,w^{\prime}$ appearing within a window of certain size in the corpus, and $p(w)$ is the marginal probability of $w$. (More complicated models could use asymmetric matrices with columns corresponding to context words or phrases, and also involve tensorization.) Then word vectors are obtained by low-rank SVD on this matrix, or a related matrix with term reweightings. In particular, the PMI matrix is found to be closely approximated by a low rank matrix: there exist word vectors in say $300$ dimensions, which is much smaller than the number of words in the dictionary, such that

where $\approx$ should be interpreted loosely.

There appears to be no theoretical explanation for this empirical finding about the approximate low rank of the PMI matrix. The current paper addresses this. Specifically, we propose a probabilistic model of text generation that augments the log-linear topic model of Mnih and Hinton (2007) with dynamics, in the form of a random walk over a latent discourse space. The chief methodological contribution is using the model priors to analytically derive a closed-form expression that directly explains (1.1); see Theorem 2.2 in Section 2. Section 3 builds on this insight to give a rigorous justification for models such as word2vec and GloVe, including the hyperparameter choices for the latter. The insight also leads to a mathematical explanation for why these word embeddings allow analogies to be solved using linear algebra; see Section 4. Section 5 shows good empirical fit to this model’s assumtions and predictions, including the surprising one that word vectors are pretty uniformly distributed (isotropic) in space.The code is available at https://github.com/PrincetonML/SemanticVector

Latent variable probabilistic models of language have been used for word embeddings before, including Latent Dirichlet Allocation (LDA) and its more complicated variants (see the survey Blei (2012)), and some neurally inspired nonlinear models Mnih and Hinton (2007); Maas et al. (2011). In fact, LDA evolved out of efforts in the 1990s to provide a generative model that “explains” the success of older vector space methods like Latent Semantic Indexing Papadimitriou et al. (1998); Hofmann (1999). However, none of these earlier generative models has been linked to PMI models.

Levy and Goldberg (2014b) tried to relate word2vec to PMI models. They showed that if there were no dimension constraint in word2vec, specifically, the “skip-gram with negative sampling (SGNS)” version of the model, then its solutions would satisfy (1.1), provided the right hand side were replaced by $\mbox{PMI}(w,w^{\prime})-\beta$ for some scalar $\beta$. However, skip-gram is a discriminative model (due to the use of negative sampling), not generative. Furthermore, their argument only applies to very high-dimensional word embeddings, and thus does not address low-dimensional embeddings, which have superior quality in applications.

Hashimoto et al. (2016) focuses on issues similar to our paper. They model text generation as a random walk on words, which are assumed to be embedded as vectors in a geometric space. Given that the last word produced was $w$, the probability that the next word is $w^{\prime}$ is assumed to be given by $h(|v_{w}-v_{w^{\prime}}|^{2})$ for a suitable function $h$, and this model leads to an explanation of (1.1). By contrast our random walk involves a latent discourse vector, which has a clearer semantic interpretation and has proven useful in subsequent work, e.g. understanding structure of word embeddings for polysemous words Arora et al. (2016). Also our work clarifies some weighting and bias terms in the training objectives of previous methods (Section 3) and also the phenomenon discussed in the next paragraph.

Researchers have tried to understand why vectors obtained from the highly nonlinear word2vec models exhibit linear structures Levy and Goldberg (2014a); Pennington et al. (2014). Specifically, for analogies like “man:woman::king:??,” queen happens to be the word whose vector $v_{queen}$ is the most similar to the vector $v_{king}-v_{man}+v_{woman}$. This suggests that simple semantic relationships, such as masculine vs feminine tested in the above example, correspond approximately to a single direction in space, a phenomenon we will henceforth refer to as relations=lines.

Section 4 surveys earlier attempts to explain this phenomenon and their shortcoming, namely, that they ignore the large approximation error in relationships like (1.1). This error appears larger than the difference between the best solution and the second best (incorrect) solution in analogy solving, so that this error could in principle lead to a complete failure in analogy solving. In our explanation, the low dimensionality of the word vectors plays a key role. This can also be seen as a theoretical explanation of the old observation that dimension reduction improves the quality of word embeddings for various tasks. The intuitive explanation often given —that smaller models generalize better—turns out to be fallacious, since the training method for creating embeddings makes no reference to analogy solving. Thus there is no a priori reason why low-dimensional model parameters (i.e., lower model capacity) should lead to better performance in analogy solving, just as there is no reason they are better at some other unrelated task like predicting the weather.

2 Benefits of generative approaches

In addition to giving some form of “unification” of existing methods, our generative model also brings more intepretability to word embeddings beyond traditional cosine similarity and even analogy solving. For example, it led to an understanding of how the different senses of a polysemous word (e.g., bank) reside in linear superposition within the word embedding Arora et al. (2016). Such insight into embeddings may prove useful in the numerous settings in NLP and neuroscience where they are used.

Another new explanatory feature of our model is that low dimensionality of word embeddings plays a key theoretical role —unlike in previous papers where the model is agnostic about the dimension of the embeddings, and the superiority of low-dimensional embeddings is an empirical finding (starting with Deerwester et al. (1990)). Specifically, our theoretical analysis makes the key assumption that the set of all word vectors (which are latent variables of the generative model) are spatially isotropic, which means that they have no preferred direction in space. Having $n$ vectors be isotropic in $d$ dimensions requires $d\ll n$. This isotropy is needed in the calculations (i.e., multidimensional integral) that yield (1.1). It also holds empirically for our word vectors, as shown in Section 5.

The isotropy of low-dimensional word vectors also plays a key role in our explanation of the relations=lines phenomenon (Section 4). The isotropy has a “purification” effect that mitigates the effect of the (rather large) approximation error in the PMI models.

Generative model and its properties

The model treats corpus generation as a dynamic process, where the $t$-th word is produced at step $t$. The process is driven by the random walk of a discourse vector $c_{t}\in\Re^{d}$. Its coordinates represent what is being talked about.This is a different interpretation of the term “discourse” compared to some other settings in computational linguistics. Each word has a (time-invariant) latent vector $v_{w}\in\Re^{d}$ that captures its correlations with the discourse vector. We model this bias with a log-linear word production model:

The discourse vector $c_{t}$ does a slow random walk (meaning that $c_{t+1}$ is obtained from $c_{t}$ by adding a small random displacement vector), so that nearby words are generated under similar discourses. We are interested in the probabilities that word pairs co-occur near each other, so occasional big jumps in the random walk are allowed because they have negligible effect on these probabilities.

A similar log-linear model appears in Mnih and Hinton (2007) but without the random walk. The linear chain CRF of Collobert and Weston (2008b) is more general. The dynamic topic model of Blei and Lafferty (2006) utilizes topic dynamics, but with a linear word production model. Belanger and Kakade (2015) have proposed a dynamic model for text using Kalman Filters, where the sequence of words is generated from Gaussian linear dynamical systems, rather than the log-linear model in our case.

The novelty here over such past works is a theoretical analysis in the method-of-moments tradition Hsu et al. (2012); Cohen et al. (2012). Assuming a prior on the random walk we analytically integrate out the hidden random variables and compute a simple closed form expression that approximately connects the model parameters to the observable joint probabilities (see Theorem 2.2). This is reminiscent of analysis of similar random walk models in finance Black and Scholes (1973).

Let $n$ denote the number of words and $d$ denote the dimension of the discourse space, where $1\leq d\leq n$. Inspecting (2.1) suggests word vectors need to have varying lengths, to fit the empirical finding that word probabilities satisfy a power law. Furthermore, we will assume that in the bulk, the word vectors are distributed uniformly in space, earlier referred to as isotropy. This can be quantified as a prior in the Bayesian tradition. More precisely, the ensemble of word vectors consists of i.i.d draws generated by $v=s\cdot\hat{v}$, where $\hat{v}$ is from the spherical Gaussian distribution, and $s$ is a scalar random variable. We assume $s$ is a random scalar with expectation $\tau=\Theta(1)$ and $s$ is always upper bounded by $\kappa$, which is another constant. Here $\tau$ governs the expected magnitude of $\left\langle v,c_{t}\right\rangle$, and it is particularly important to choose it to be $\Theta(1)$ so that the distribution $\Pr[w|c_{t}]\propto\exp(\left\langle v_{w},c_{t}\right\rangle)$ is interesting.A larger $\tau$ will make $\Pr[w|c_{t}]$ too peaked and a smaller one will make it too uniform. Moreover, the dynamic range of word probabilities will roughly equal $\exp(\kappa^{2})$, so one should think of $\kappa$ as an absolute constant like $5$. These details about $s$ are important for realistic modeling but not too important in our analysis. (Furthermore, readers uncomfortable with this simplistic Bayesian prior should look at Section 2.2 below.)

The following lemma (whose proof appears in the appendix) is central to the analysis. It says that under the Bayesian prior, the partition function $Z_{c}=\sum_{w}\exp(\left\langle v_{w},c\right\rangle)$, which is the implied normalization in equation (2.1), is close to some constant $Z$ for most of the discourses $c$. This can be seen as a plausible theoretical explanation of a phenomenon called self-normalization in log-linear models: ignoring the partition function or treating it as a constant (which greatly simplifies training) is known to often give good results. This has also been studied in Andreas and Klein (2014).

If the word vectors satisfy the Bayesian prior described in the model details, then

for $\epsilon_{z}=\widetilde{O}(1/\sqrt{n})$, and $\delta=\exp(-\Omega(\log^{2}n))$.

The concentration of the partition functions then leads to our main theorem (the proof is in the appendix). The theorem gives simple closed form approximations for $p(w)$, the probability of word $w$ in the corpus, and $p(w,w^{\prime})$, the probability that two words $w,w^{\prime}$ occur next to each other. The theorem states the result for the window size $q=2$, but the same analysis works for pairs that appear in a small window, say of size $10$, as stated in Corollary 2.3. Recall that $\mbox{PMI}(w,w^{\prime})=\log[p(w,w^{\prime})/(p(w)p(w^{\prime}))]$.

Suppose the word vectors satisfy the inequality (2.2), and window size $q=2$. Then,

for $\epsilon=O(\epsilon_{z})+\widetilde{O}(1/d)+O(\epsilon_{2})$. Jointly these imply:

Remarks 2. Variants of the expression for joint probability in (2.3) had been hypothesized based upon empirical evidence in Mikolov et al. (2013b) and also Globerson et al. (2007), and Maron et al. (2010) .

Remarks 3. Theorem 2.2 directly leads to the extension to a general window size $q$ as follows:

Let $p_{q}(w,w^{\prime})$ be the co-occurrence probability in windows of size $q$, and $\mbox{PMI}_{q}(w,w^{\prime})$ be the corresponding PMI value. Then

where $\gamma=\log\left(\frac{q(q-1)}{2}\right)$.

It is quite easy to see that Theorem 2.2 implies the Corollary 2.3, as when the window size is $q$ the pair $w,w^{\prime}$ could appear in any of ${q\choose 2}$ positions within the window, and the joint probability of $w,w^{\prime}$ is roughly the same for any positions because the discourse vector changes slowly. (Of course, the error term gets worse as we consider larger window sizes, although for any constant size, the statement of the theorem is correct.) This is also consistent with the shift $\beta$ for fitting PMI in Levy and Goldberg (2014b), which showed that without dimension constraints, the solution to skip-gram with negative sampling satisfies $\mbox{PMI}~{}(w,w^{\prime})-\beta=\langle v_{w},v_{w^{\prime}}\rangle$ for a constant $\beta$ that is related to the negative sampling in the optimization. Our result justifies via a generative model why this should be satisfied even for low dimensional word vectors.

1 Proof sketches

Here we provide the proof sketches, while the complete proof can be found in the appendix.

Let $w$ and $w^{\prime}$ be two arbitrary words. Let $c$ and $c^{\prime}$ denote two consecutive context vectors, where $c\sim\mathcal{C}$ and $c^{\prime}|c$ is defined by the Markov kernel $p(c^{\prime}\mid c)$.

We start by using the law of total expectation, integrating out the hidden variables $c$ and $c^{\prime}$:

An expectation like (2.6) would normally be difficult to analyze because of the partition functions. However, we can assume the inequality (2.2), that is, the partition function typically does not vary much for most of context vectors $c$. Let $F$ be the event that both $c$ and $c^{\prime}$ are within $(1\pm\epsilon_{z})Z$. Then by (2.2) and the union bound, event $F$ happens with probability at least $1-2\exp(-\Omega(\log^{2}n))$. We will split the right-hand side (RHS) of (2.6) into the parts according to whether $F$ happens or not.

where $\bar{F}$ denotes the complement of event $F$ and $\mathbf{1}_{F}$ and $\mathbf{1}_{\bar{F}}$ denote indicator functions for $F$ and $\bar{F}$, respectively. When $F$ happens, we can replace $Z_{c}$ by $Z$ with a $1\pm\epsilon_{z}$ factor loss: The first term of the RHS of (2.7) equals to

Therefore it suffices to only consider (2.8). Our model assumptions state that $c$ and $c^{\prime}$ cannot be too different. We leverage that by rewriting (2.8) a little, and get that it equals

Furthermore, by our model assumptions, $\|c-c^{\prime}\|\leq\epsilon_{2}/\sqrt{d}$. So

and thus $\displaystyle A(c)=(1\pm O(\epsilon_{2}))\exp(\langle v_{w^{\prime}},c\rangle)$. Plugging the simplification of $A(c)$ to (2.10),

Bounding the difference between $\left\langle v_{w}+v_{w^{\prime}},c\right\rangle$ from Gaussian random variable, we can show that for $\epsilon=\widetilde{O}(1/d)$,

Therefore, the series of simplification/approximation above (concretely, combining equations (2.6), (2.7), (2.9), (2.11), and (2.13)) lead to the desired bound on $\log p(w,w^{\prime})$ for the case when the window size $q=2$. The bound on $\log p(w)$ can be shown similarly.

Proof sketch of Lemma 2.1

Note that for fixed $c$, when word vectors have Gaussian priors assumed as in our model, $Z_{c}=\sum_{w}\exp(\langle v_{w},c\rangle)$ is a sum of independent random variables.

where the second line is just an application of the law of total expectation, if we pick the norm of the (random) vector $v_{w}$ first, followed by its direction. Conditioned on $s_{w}$, $\langle v_{w},c\rangle$ is a Gaussian random variable with variance $\|c\|_{2}^{2}s_{w}^{2}$, and therefore using similar calculation as in (2.12), we have

Proof of Theorem 4.1

Indeed, $\sum_{i=1}^{d}\sigma^{2}_{i}=O(nd)$, since the average squared norm of a word vector is $d$. The claim then follows from the first assumption. Furthermore, by the second assumption, $\|P^{T}\zeta^{\prime}\|_{\infty}\leq\frac{c_{2}}{\sqrt{n}}\|\zeta^{\prime}\|_{2}$, so

Plugging (2.16) and (2.17) into (2.15), we get

as desired. The last statement follows because the norm of the signal, which is $d\log(\nu_{R})$ originally and is $V^{\dagger}d\log(\nu_{R})=v_{a}-v_{b}$ after dimension reduction, also gets reduced by a factor of $\sqrt{n}$.

2 Weakening the model assumptions

For readers uncomfortable with Bayesian priors, we can replace our assumptions with concrete properties of word vectors that are empirically verifiable (Section 5.1) for our final word vectors, and in fact also for word vectors computed using other recent methods.

The word meanings are assumed to be represented by some “ground truth” vectors, which the experimenter is trying to recover. These ground truth vectors are assumed to be spatially isotropic in the bulk, in the following two specific ways: (i) For almost all unit vectors $c$ the sum $\sum_{w}\exp(\langle v_{w},c\rangle)$ is close to a constant $Z$; (ii) Singular values of the matrix of word vectors satisfy properties similar to those of random matrices, as formalized in the paragraph before Theorem 4.1. Our Bayesian prior on the word vectors happens to imply that these two conditions hold with high probability. But the conditions may hold even if the prior doesn’t hold. Furthermore, they are compatible with all sorts of local structure among word vectors such as existence of clusterings, which would be absent in truly random vectors drawn from our prior.

Training objective and relationship to other models

Assuming this approximation, we show below that the maximum likelihood values for the word vectors correspond to the following optimization,

As is usual, empirical performance is improved by weighting down very frequent word pairs, possibly because very frequent words such as “the” do not fit our model. This is done by replacing the weighting $X_{w,w^{\prime}}$ by its truncation $\min\{X_{w,w^{\prime}},X_{\text{max}}\}$ where $X_{\text{max}}$ is a constant such as $100$. We call this objective with the truncated weights SN (Squared Norm).

We now give its derivation. Maximizing the likelihood of $\{X_{w,w^{\prime}}\}$ is equivalent to maximizing

Denote the logarithm of the ratio between the expected count and the empirical count as

Then with some calculation, we obtain the following where $c$ is independent of the empirical observations $X_{w,w^{\prime}}$’s.

On the other hand, using $e^{x}\approx 1+x+x^{2}/2$ when $x$ is small,This Taylor series approximation has an error of the order of $x^{3}$, but ignoring it can be theoretically justified as follows. For a large $X_{w,w^{\prime}}$, its value approaches its expectation and thus the corresponding $\Delta_{w,w^{\prime}}$ is close to 0 and thus ignoring $\Delta_{w,w^{\prime}}^{3}$ is well justified. The terms where $\Delta_{w,w^{\prime}}$ is significant correspond to $X_{w,w^{\prime}}$’s that are small. But empirically, $X_{w,w^{\prime}}$’s obey a power law distribution (see, e.g. Pennington et al. (2014)) using which it can be shown that these terms contribute a small fraction of the final objective (3.3). So we can safely ignore the errors. Full details appear in the ArXiv version of this paper Arora et al. (2015). we have

So maximizing the likelihood is approximately equivalent to minimizing the right hand side, which (by examining (3.1)) leads to our objective.

A similar objective PMI can be obtained from (2.5), by computing an approximate MLE, using the fact that the error between the empirical and true value of $\mbox{PMI}(w,w^{\prime})$ is driven by the smaller term $p(w,w^{\prime})$, and not the larger terms $p(w),p(w^{\prime})$.

This is of course very analogous to classical VSM methods, with a novel reweighting method.

Fitting to either of the objectives involves solving a version of Weighted SVD which is NP-hard, but empirically seems solvable in our setting via AdaGrad Duchi et al. (2011).

Connection to GloVe.

Compare SN with the objective used by GloVe Pennington et al. (2014):

with $f(X_{w,w^{\prime}})=\min\{X_{w,w^{\prime}}^{3/4},100\}.$ Their weighting methods and the need for bias terms $s_{w},s_{w^{\prime}},C$ were derived by trial and error; here they are all predicted and given meanings due to Theorem 2.2, specifically $s_{w}=\left\|v_{w}\right\|^{2}$.

Connection to word2vec(CBOW).

The CBOW model in word2vec posits that the probability of a word $w_{k+1}$ as a function of the previous $k$ words $w_{1},w_{2},\ldots,w_{k}$:

This expression seems mysterious since it depends upon the average word vector for the previous $k$ words. We show it can be theoretically justified. Assume a simplified version of our model, where a small window of $k$ words is generated as follows: sample $c\sim\mathcal{C}$, where $\mathcal{C}$ is a uniformly random unit vector, then sample $\left(w_{1},w_{2},\dots,w_{k}\right)\sim\exp(\langle\sum^{k}_{i=1}v_{w_{i}},c\rangle)/Z_{c}$. Furthermore, assume $Z_{c}=Z$ for any $c$.

In the simplified version of our model, the Maximum-a-Posteriori (MAP) estimate of $c$ given $\left(w_{1},w_{2},\dots,w_{k}\right)$ is $\frac{\sum_{i=1}^{k}v_{w_{i}}}{\|\sum_{i=1}^{k}v_{w_{i}}\|_{2}}$.

The $c$ maximizing $p\left(c|w_{1},w_{2},\dots,w_{k}\right)$ is the maximizer of $p(c)p\left(w_{1},w_{2},\dots,w_{k}|c\right)$. Since $p(c)=p(c^{\prime})$ for any $c,c^{\prime}$, and we have $p\left(w_{1},w_{2},\dots,w_{k}|c\right)=\exp(\langle\sum_{i}v_{w_{i}},c\rangle)/Z,$ the maximizer is clearly $c=\frac{\sum_{i=1}^{k}v_{w_{i}}}{\|\sum_{i=1}^{k}v_{w_{i}}\|_{2}}$. ∎

Thus using the MAP estimate of $c_{t}$ gives essentially the same expression as CBOW apart from the rescaling, which is often omitted due to computational efficiency in empirical works.

Explaining relations=lines

As mentioned, word analogies like “$a$:$b$::$c$:??” can be solved via a linear algebraic expression:

where vectors have been normalized such that $\left\|v_{d}\right\|_{2}=1$. This suggests that the semantic relationships being tested in the analogy are characterized by a straight line,Note that this interpretation has been disputed; e.g., it is argued in Levy and Goldberg (2014a) that (4.1) can be understood using only the classical connection between inner product and word similarity, using which the objective (4.1) is slightly improved to a different objective called 3COSMUL. However, this “explanation” is still dogged by the issue of large termwise error pinpointed here, since inner product is only a rough approximation to word similarity. Furthermore, the experiments in Section 5 clearly support the relations=lines interpretation. referred to earlier as relations=lines.

Using our model we will show the following for low-dimensional embeddings: for each such relation $R$ there is a direction $\mu_{R}$ in space such that for any word pair $a,b$ satisfying the relation, $v_{a}-v_{b}$ is like $\mu_{R}$ plus some noise vector. This happens for relations satisfying a certain condition described below. Empirical results supporting this theory appear in Section 5, where this linear structure is further leveraged to slightly improve analogy solving.

A side product of our argument will be a mathematical explanation of the empirically well-established superiority of low-dimensional word embeddings over high-dimensional ones in this setting Levy and Goldberg (2014a). As mentioned earlier, the usual explanation that smaller models generalize better is fallacious.

We first sketch what was missing in prior attempts to prove versions of relations=lines from first principles. The basic issue is approximation error: the difference between the best solution and the 2nd best solution to (4.1) is typically small, whereas the approximation error in the objective in the low-dimensional solutions is larger. For instance, if one uses our PMI objective, then the weighted average of the termwise error in (2.5) is $17\%$, and the expression in (4.1) above contains six inner products. Thus in principle the approximation error could lead to a failure of the method and the emergence of linear relationship, but it does not.

Pennington et al. (2014) try to propose a model where such linear relationships should occur by design. They posit that queen is a solution to the analogy “man:woman::king:??” because

where $p(\chi\mid king)$ denotes the conditional probability of seeing word $\chi$ in a small window of text around ${king}$. Relationship (4.2) is intuitive since both sides will be $\approx 1$ for gender-neutral $\chi$ like “walks” or “food”, will be $>1$ when $\chi$ is like “he, Henry” and will be $<1$ when $\chi$ is like “dress, she, Elizabeth.” This was also observed by Levy and Goldberg (2014a). Given (4.2), they then posit that the correct model describing word embeddings in terms of word occurrences must be a homomorphism from $({\Re^{d},+})$ to $(\Re^{+},\times)$, so vector differences map to ratios of probabilities. This leads to the expression

and their method is a (weighted) least squares fit for this expression. One shortcoming of this argument is that the homomorphism assumption assumes the linear relationships instead of explaining them from a more basic principle. More importantly, the empirical fit to the homomorphism has nontrivial approximation error, high enough that it does not imply the desired strong linear relationships.

Levy and Goldberg (2014b) show that empirically, skip-gram vectors satisfy

up to some shift. They also give an argument suggesting this relationship must be present if the solution is allowed to be very high-dimensional. Unfortunately, that argument does not extend to low-dimensional embeddings. Even if it did, the issue of termwise approximation error remains.

Our explanation.

The current paper has introduced a generative model to theoretically explain the emergence of relationship (4.3). However, as noted after Theorem 2.2, the issue of high approximation error does not go away either in theory or in the empirical fit. We now show that the isotropy of word vectors (assumed in the theoretical model and verified empirically) implies that even a weak version of (4.3) is enough to imply the emergence of the observed linear relationships in low-dimensional embeddings.

This argument will assume the analogy in question involves a relation that obeys Pennington et al.’s suggestion in (4.2). Namely, for such a relation $R$ there exists function $\nu_{R}(\cdot)$ depending only upon $R$ such that for any $a,b$ satisfying $R$ there is a noise function $\xi_{a,b,R}(\cdot)$ for which:

For different words $\chi$ there is huge variation in (4.4), so the multiplicative noise may be large.

Our goal is to show that the low-dimensional word embeddings have the property that there is a vector $\mu_{R}$ such that for every pair of words $a,b$ in that relation, $v_{a}-v_{b}=\mu_{R}+\mbox{noise vector}$, where the noise vector is small.

Theorem 2.2 implies that the left-hand side simplifies to $\log\left(\frac{p(\chi\mid a)}{p(\chi\mid b)}\right)=\frac{1}{d}\left\langle v_{\chi},v_{a}-v_{b}\right\rangle+\epsilon_{a,b}(\chi)$ where $\epsilon$ captures the small approximation errors induced by the inexactness of Theorem 2.2. This adds yet more noise! Denoting by $V$ the $n\times d$ matrix whose rows are the $v_{\chi}$ vectors, we rewrite (4.5) as:

where $\log(\nu_{R})$ in the element-wise log of vector $\nu_{R}$ and $\zeta^{\prime}_{a,b,R}=d(\zeta_{a,b,R}-\epsilon_{a,b,R})$ is the noise.

In essence, (4.6) shows that $v_{a}-v_{b}$ is a solution to a linear regression in $d$ variables and $m$ constraints, with $\zeta^{\prime}_{a,b,R}$ being the “noise.” The design matrix in the regression is $V$, the matrix of all word vectors, which in our model (as well as empirically) satisfies an isotropy condition. This makes it random-like, and thus solving the regression by left-multiplying by $V^{\dagger}$, the pseudo-inverse of $V$, ought to “denoise” effectively. We now show that it does.

Our model assumed the set of all word vectors satisfies bulk properties similar to a set of Gaussian vectors. The next theorem will only need the following weaker properties. (1) The smallest non-zero singular value of $V$ is larger than some constant $c_{1}$ times the quadratic mean of the singular values, namely, $\|V\|_{F}/\sqrt{d}$. Empirically we find $c_{1}\approx 1/3$ holds; see Section 5. (2) The left singular vectors behave like random vectors with respect to $\zeta^{\prime}_{a,b,R}$, namely, have inner product at most $c_{2}\|\zeta^{\prime}_{a,b,R}\|/\sqrt{n}$ with $\zeta^{\prime}_{a,b,R}$, for some constant $c_{2}$. (3) The max norm of a row in $V$ is $O(\sqrt{d})$. The proof is included in the appendix.

Under the conditions of the previous paragraph, the noise in the dimension-reduced semantic vector space satisfies

As a corollary, the relative error in the dimension-reduced space is a factor of $\sqrt{d/n}$ smaller.

Experimental verification

In this section, we provide experiments empirically supporting our generative model.

All word embedding vectors are trained on the English Wikipedia (March 2015 dump). It is pre-processed by standard approach (removing non-textual elements, sentence splitting, and tokenization), leaving about $3$ billion tokens. Words that appeared less than $1000$ times in the corpus are ignored, resulting in a vocabulary of $68,430$. The co-occurrence is then computed using windows of 10 tokens to each side of the focus word.

Training method.

Our embedding vectors are trained by optimizing the SN objective using AdaGrad Duchi et al. (2011) with initial learning rate of $0.05$ and 100 iterations. The PMI objective derived from (2.5) was also used. SN has average (weighted) term-wise error of $5$%, and PMI has $17$%. We observed that SN vectors typically fit the model better and have better performance, which can be explained by larger errors in PMI, as implied by Theorem 2.2. So, we only report the results for SN.

For comparison, GloVe and two variants of word2vec (skip-gram and CBOW) vectors are trained. GloVe’s vectors are trained on the same co-occurrence as SN with the default parameter values.http://nlp.stanford.edu/projects/glove/ word2vec vectors are trained using a window size of 10, with other parameters set to default values.https://code.google.com/p/word2vec/

1 Model verification

Experiments were run to test our modeling assumptions. First, we tested two counter-intuitive properties: the concentration of the partition function $Z_{c}$ for different discourse vectors $c$ (see Theorem 2.1), and the random-like behavior of the matrix of word embeddings in terms of its singular values (see Theorem 4.1). For comparison we also tested these properties for word2vec and GloVe vectors, though they are trained by different objectives. Finally, we tested the linear relation between the squared norms of our word vectors and the logarithm of the word frequencies, as implied by Theorem 2.2.

Our theory predicts the counter-intuitive concentration of the partition function $Z_{c}=\sum_{w^{\prime}}\exp(c^{\top}v_{w^{\prime}})$ for a random discourse vector $c$ (see Lemma 2.1). This is verified empirically by picking a uniformly random direction, of norm $\|c\|=4/\mu_{w}$, where $\mu_{w}$ is the average norm of the word vectors.Note that our model uses the inner products between the discourse vectors and word vectors, so it is invariant if the discourse vectors are scaled by $s$ while the word vectors are scaled by $1/s$ for any $s>0$. Therefore, one needs to choose the norm of $c$ properly. We assume $\|c\|\mu_{w}=\sqrt{d}/\kappa\approx 4$ for a constant $\kappa=5$ so that it gives a reasonable fit to the predicted dynamic range of word frequencies according to our theory; see model details in Section 2. Figure 1(a) shows the histogram of $Z_{c}$ for $1000$ such randomly chosen $c$’s for our vectors. The values are concentrated, mostly in the range $[0.9,1.1]$ times the mean. Concentration is also observed for other types of vectors, especially for GloVe and CBOW.

Isotropy with respect to singular values.

Our theoretical explanation of relations=lines assumes that the matrix of word vectors behaves like a random matrix with respect to the properties of singular values. In our embeddings, the quadratic mean of the singular values is 34.3, while the minimum non-zero singular value of our word vectors is 11. Therefore, the ratio between them is a small constant, consistent with our model. The ratios for GloVe, CBOW, and skip-gram are 1.4, 10.1, and 3.1, respectively, which are also small constants.

Squared norms v.s. word frequencies.

Figure 2 shows a scatter plot for the squared norms of our vectors and the logarithms of the word frequencies. A linear relationship is observed (Pearson correlation 0.75), thus supporting Theorem 2.2. The correlation is stronger for high frequency words, possibly because the corresponding terms have higher weights in the training objective.

This correlation is much weaker for other types of word embeddings. This is possibly because they have more free parameters (“knobs to turn”), which imbue the embeddings with other properties. This can also cause the difference in the concentration of the partition function for the two methods.

2 Performance on analogy tasks

We compare the performance of our word vectors on analogy tasks, specifically the two testbeds GOOGLE and MSR Mikolov et al. (2013a; c). The former contains $7874$ semantic questions such as “man:woman::king:??”, and $10167$ syntactic ones such as “run:runs::walk:??.” The latter has $8000$ syntactic questions for adjectives, nouns, and verbs.

To solve these tasks, we use linear algebraic queries.One can instead use the 3COSMUL in Levy and Goldberg (2014a), which increases the accuracy by about $3\%$. But it is not linear while our focus here is the linear algebraic structure. That is, first normalize the vectors to unit norm and then solve “a:b::c:??” by

The algorithm succeeds if the best $d$ happens to be correct.

The performance of different methods is presented in Table 1. Our vectors achieve performance comparable to the state of art on semantic analogies (similar accuracy as GloVe, better than word2vec). On syntactic tasks, they achieve accuracy $0.04$ lower than GloVe and skip-gram, while CBOW typically outperforms the others.It was earlier reported that skip-gram outperforms CBOW Mikolov et al. (2013a); Pennington et al. (2014). This may be due to the different training data sets and hyperparameters used. The reason is probably that our model ignores local word order, whereas the other models capture it to some extent. For example, a word “she” can affect the context by a lot and determine if the next word is “thinks” rather than “think”. Incorporating such linguistic features in the model is left for future work.

3 Verifying relations=lines

The theory in Section 4 predicts the existence of a direction for a relation, whereas earlier Levy and Goldberg (2014a) had questioned if this phenomenon is real. The experiment uses the analogy testbed, where each relation is tested using $20$ or more analogies. For each relation, we take the set of vectors $v_{ab}=v_{a}-v_{b}$ where the word pair $(a,b)$ satisfies the relation. Then calculate the top singular vectors of the matrix formed by these $v_{ab}$’s, and compute the cosine similarity (i.e., normalized inner product) of individual $v_{ab}$ to the singular vectors. We observed that most $(v_{a}-v_{b})$’s are correlated with the first singular vector, but have inner products around 0 with the second singular vector. Over all relations, the average projection on the first singular vector is 0.51 (semantic: 0.58; syntactic: 0.46), and the average on the second singular vector is 0.035. For example, Table 2 shows the mean similarities and standard deviations on the first and second singular vectors for 4 relations. Similar results are also obtained for word embedings by GloVe and word2vec. Therefore, the first singular vector can be taken as the direction associated with this relation, while the other components are like random noise, in line with our model.

The above linear structure suggests a better (but cheating) way to solve the analogy task. This uses the fact that the same semantic relationship (e.g., masculine-feminine, singular-plural) is tested many times in the testbed. If a relation $R$ is represented by a direction $\mu_{R}$ then the cheating algorithm can learn this direction (via rank 1 SVD) after seeing a few examples of the relationship. Then use the following method of solving “a:b::c:??”: look for a word $d$ such that $v_{c}-v_{d}$ has the largest projection on $\mu_{R}$, the relation direction for $(a,b)$. This can boost success rates by about $10\%$.

The testbed can try to combat such cheating by giving analogy questions in a random order. But the cheating algorithm can just cluster the presented analogies to learn which of them are in the same relation. Thus the final algorithm, named analogy solver with relation direction (RD), is: take all vectors $v_{a}-v_{b}$ for all the word pairs $(a,b)$ presented among the analogy questions and do $k$-means clustering on them; for each $(a,b)$, estimate the relation direction by taking the first singular vector of its cluster, and substitute that for $v_{a}-v_{b}$ in (5.1) when solving the analogy. Table 3 shows the performance on GOOGLE with different values of $k$; e.g. using our SN vectors and $k=30$ leads to $0.79$ accuracy. Thus future designers of analogy testbeds should remember not to test the same relationship too many times! This still leaves other ways to cheat, such as learning the directions for interesting semantic relations from other collections of analogies.

Non-cheating solver for analogy testbeds.

Now we show that even if a relationship is tested only once in the testbed, there is a way to use the above structure. Given “a:b::c:??,” the solver first finds the top $300$ nearest neighbors of $a$ and those of $b$, and then finds among these neighbors the top $k$ pairs $(a^{\prime},b^{\prime})$ so that the cosine similarities between $v_{a^{\prime}}-v_{b^{\prime}}$ and $v_{a}-v_{b}$ are largest. Finally, the solver uses these pairs to estimate the relation direction (via rank 1 SVD), and substitute this (corrected) estimate for $v_{a}-v_{b}$ in (5.1) when solving the analogy. This algorithm is named analogy solver with relation direction by nearest neighbors (RD-nn). Table 4 shows its performance, which consistently improves over the old method by about $3\%$.

Conclusions

A simple generative model has been introduced to explain the classical PMI based word embedding models, as well as recent variants involving energy-based models and matrix factorization. The model yields an optimization objective with essentially “no knobs to turn”, yet the embeddings lead to good performance on analogy tasks, and fit other predictions of our generative model. A model with fewer knobs to turn should be seen as a better scientific explanation (Occam’s razor), and certainly makes the embeddings more interpretable.

The spatial isotropy of word vectors is both an assumption in our model, and also a new empirical finding of our paper. We feel it may help with further development of language models. It is important for explaining the success of solving analogies via low dimensional vectors (relations=lines). It also implies that semantic relationships among words manifest themselves as special directions among word embeddings (Section 4), which lead to a cheater algorithm for solving analogy testbeds.

Our model is tailored to capturing semantic similarity, more akin to a log-linear dynamic topic model. In particular, local word order is unimportant. Designing similar generative models (with provable and interpretable properties) with linguistic features is left for future work.

Acknowledgements

We thank the editors of TACL for granting a special relaxation of the page limit for our paper. We thank Yann LeCun, Christopher D. Manning, and Sham Kakade for helpful discussions at various stages of this work.

This work was supported in part by NSF grants CCF-1527371, DMS-1317308, Simons Investigator Award, Simons Collaboration Grant, and ONR-N00014-16-1-2329. Tengyu Ma was supported in addition by Simons Award in Theoretical Computer Science and IBM PhD Fellowship.

References

Appendix A Proofs of Theorem 1

In this section we prove Theorem 2.2 and Lemma 2.1 (restated below).

Suppose the word vectors satisfy equation (2.2), and window size $q=2$. Then,

for $\epsilon=O(\epsilon_{z})+\widetilde{O}(1/d)+O(\epsilon_{2})$. Jointly these imply:

If the word vectors satisfy the bayesian prior $v=s\cdot\hat{v}$, where $\hat{v}$ is from the spherical Gaussian distribution, and $s$ is a scalar random variable, then with high probability the entire ensemble of word vectors satisfies that

for $\epsilon_{z}=\widetilde{O}(1/\sqrt{n})$, and $\delta=\exp(-\Omega(\log^{2}n))$.

We first prove Theorem 2.2 using Lemma 2.1, and Lemma 2.1 will be proved in Section A.1. Please see Section 2 of the main paper for the intuition of the proof and a cleaner sketch without too many technicalities.

Let $c$ be the hidden discourse that determines the probability of word $w$, and $c^{\prime}$ be the next one that determines $w^{\prime}$. We use $p(c^{\prime}|c)$ to denote the Markov kernel (transition matrix) of the Markov chain. Let $\mathcal{C}$ be the stationary distribution of discourse vector $c$, and $\mathcal{D}$ be the joint distribution of $(c,c^{\prime})$. We marginalize over the contexts $c,c^{\prime}$ and then use the independence of $w,w^{\prime}$ conditioned on $c,c^{\prime}$,

We first get rid of the partition function $Z_{c}$ using Lemma 2.1. As sketched in the main paper, essentially we will replace $Z_{c}$ by $Z$ in equation (A.5), though a very careful control of the approximation error is required. Formally, Let $\mathcal{F}_{1}$ be the event that $c$ satisfies

We first decompose the integral (A.5) into the two parts according to whether event $\mathcal{F}$ happens,

We bound the first quantity on the right hand side using (2.2) and the definition of $\mathcal{F}$.

For the second quantity of the right hand side of (A.7), we have by Cauchy-Schwartz,

Using the fact that $Z_{c}\geq 1$, then we have that

The second term of (A.10) is upper bounded by

We proceed to the first term of (A.10) and observe the following property of it:

where $\alpha>1$. Therefore, it’s sufficient to bound

Let’s denote by $z$ the random variable $2\left\langle v_{w},c\right\rangle$.

If $c$ were distributed as $\mathcal{N}(0,\frac{1}{d}I)$, this would be a simple tail bound. However, as $c$ is distributed uniformly on the sphere, this requires special care, and the claim follows by applying Lemma A.1 instead. Finally, applying Corollary A.3, we have:

We have the same bound for $c^{\prime}$ as well. Hence, for the second quantity of the right hand side of (A.7), we have

where the first inequality follows from Cauchy-Schwartz, and the second from the calculation above. Combining (A.7), (A.8) and (A.13), we obtain

where $\delta_{0}=\exp(-\Omega(\log^{1.8}n))Z^{2}\leq\exp(-\Omega(\log^{1.8}n))$ by the fact that $Z\leq\exp(2\kappa)n=O(n)$. Note that $\kappa$ is treated as an absolute constant throughout the paper. On the other hand, we can lowerbound similarly

Taking logarithm, the multiplicative error translates to a additive error

For the purpose of exploiting the fact that $c,c^{\prime}$ should be close to each other, we further rewrite $\log p(w,w^{\prime})$ by re-organizing the expectations above,

where the inner integral which is denoted by $A(c)$,

Since $\|v_{w}\|\leq\kappa\sqrt{d}$. Therefore we have that $\langle v_{w},c-c^{\prime}\rangle\leq\|v_{w}\|\|c-c^{\prime}\|\leq\kappa\sqrt{d}\|c-c^{\prime}\|$.

where the last inequality follows from our model assumptions. To derive a lower bound of $A(c)$, observe that

Therefore, our model assumptions imply that

Therefore, we obtain that $A(c)=(1\pm\epsilon_{2})\exp(\langle v_{w^{\prime}},c\rangle)$. Plugging the just obtained estimate of $A(c)$ into the equation (A.14), we get that

Plugging in equation (A.16) into equation (A.15), we have that

where the last equality holds since $\|v\|=\Omega(\sqrt{d})$ and $t=\omega(1)$. ∎

Next, we show that $r$ is lower bounded by a constant with probability $1-\exp(-\Omega(d))$. Indeed, $r$ is in distribution equal to $\frac{1}{d}\chi^{2}_{d-1}$, where $\chi^{2}_{k}$ is a Chi-squared distribution with $k$ degrees of freedom. Standard concentration bounds laurent2000adaptive imply that $\forall\xi\geq 0,\Pr[r-1\leq-2\sqrt{\frac{\xi}{d}}]\leq\exp(-\xi)$. Taking $\xi=\alpha d$ for $\alpha$ a constant implies that with probability $1-\exp(-\Omega(d))$, $r\geq M$ for some constant $M$. We can now rewrite

The first term is clearly bounded by $e^{-\Omega(x^{2})}$ and the second by $\exp(-\Omega(d))$. Therefore,

Putting A.17 and A.18 together, we get that

For any random variable $X$ which has non-negative support, it’s easy to check that

To bound this integral, we split into the following two cases:

where the last inequality follows since $\|v\|=O(\sqrt{d})$.

Case $t^{2}<d$: In the second case, we will split the integral into two portions: $u\in[0,\exp(t)]$ and $u\in[\exp(t),\exp(\|v\|)]$.

where the last inequality is the usual Gaussian tail bound.

As a corollary to the above lemma, we get the following:

We claim the proof is trivial if $d=o(\log^{4}n)$. Indeed, in this case, $\exp(\langle v,c\rangle)\leq\exp(\|v\|)=\exp(O(\sqrt{d}))$. Hence,

Since by Lemma A.1, $\Pr[z\geq t]\leq\exp(-\Omega(\log^{2}n)$, we get

So, we may, without loss of generality assume that $d=\Omega(\log^{4}n)$. In this case, Lemma A.2 implies

where the last inequality holds because $d=\Omega(\log^{4}n)$ and $t^{2}=\Omega(\log^{.9}n)$, so we get the claim we wanted.

where at the third line we use the fact that $u(x)G(x)\rightarrow 0$ as $x\rightarrow\infty$ and that $u^{\prime}(x)\geq 0$, and at the last line we integrate by parts again. ∎

where $\epsilon=\widetilde{O}(\frac{1}{d})$.

Let $g\in\mathcal{N}(0,I)$, then $g/\|g\|$ has the same distribution as $c$. Let $r=\|v\|$. Since $c$ is spherically symmetric, we could, we can assume without loss of generality that $v=(r,0,\dots,0)$. Let $x=g_{1}$ and $y=\sqrt{g_{2}^{2}+\dots+g_{d}^{2}}$. Therefore $x\in\mathcal{N}(0,1)$ and $y^{2}$ has $\chi^{2}$ distribution with mean $d-1$ and variance $O(d)$.

Conditioned on event $\mathcal{F}$, we have

where we used the fact that $r\leq\kappa\sqrt{d}$. Let $\mathcal{E}$ be the event that $1.5\sqrt{d}\geq y\geq 0.5\sqrt{d}$. By using Proposition 1, we have that

Then let $z=y^{2}/(d-1)$ and $w=z-1$. Therefore $z$ has $\chi^{2}$ distribution with mean 1 and variance $1/(d-1)$, and $w$ has mean 0 and variance $1/(d-1)$.

We finally provide the proofs for a few helper propositions on conditional probabilities for high probability events used in the lemma above.

Let’s denote by $\bar{\mathcal{E}}$ the complement of the event $\mathcal{E}$. We will consider the upper and lower bound separately. Since

where the last equality follows from the change of variables $x=\sigma x^{\prime}$. However,

is nothing more than $\Pr[x^{\prime}>\frac{t}{\sigma}]$, where $x^{\prime}$ is distributed like a univariate gaussian with mean $\frac{\sigma}{2}$ and variance 1. Bearing in mind that $\sigma=O(1)$

by the usual Gaussian tail bounds, which proves the lower bound we need.

where the last equality follows from the same change of variables $x=\sigma x^{\prime}$ as before. Since $\int_{t=-\infty}^{+\infty}e^{-(x^{\prime}-\frac{\sigma}{2})^{2}}dx^{\prime}=\sqrt{2\pi}$, we get

Let $z=\left\langle v,c\right\rangle$. We proceed similarly as in the proof of Proposition 1. We have

We again proceed by separating the upper and lower bound.

We proceed to the first term of (A.10) and observe the following property of it:

where $\alpha>1$. Therefore, it’s sufficient to bound

Indeed, this follows by directly applying Lemma A.1. Afterward, applying Lemma A.2, we have:

Consider the event $\mathcal{E}^{\prime}:z\leq t$, for $t=\Theta(\log^{.9}d)$, which by Lemma A.1 satisfies $\Pr[\mathcal{E}^{\prime}]\geq 1-\exp(-\Omega(\log^{2}d))$. By the upper bound we just showed,

where the last equality follows since conditioned on $\mathcal{E}^{\prime}$, $z=O(\log^{.9}d)$. Finally, this implies

where the last equality follows since $\Pr[\mathcal{E}]\geq 1-\exp(-\Omega(\log^{2}n))$. Putting this together with A.27, we get

In this section, we prove Lemma 2.1. We basically first prove that for the means of $Z_{c}$ are all $(1+o(1))$-close to each other, and then prove that $Z_{c}$ is concentrated around its mean. It turns out the concentration part is non trivial because the random variable of concern, $\exp(\langle v_{w},c\rangle)$ is not well-behaved in terms of the tail. Note that $\exp(\langle v_{w},c\rangle)$ is NOT sub-gaussian for any variance proxy. This essentially disallows us to use an existing concentration inequality directly. We get around this issue by considering the truncated version of $\exp(\langle v_{w},c\rangle)$, which is bounded, and have similar tail properties as the original one, in the regime that we are concerning.

We bound the mean and variance of $Z_{c}$ first in the Lemma below.

We fix context $c$ and view $v_{w}$’s as random variables throughout this proof. Recall that $v_{w}$ is composed of $v_{w}=s_{w}\cdot\hat{v}_{w}$, where $s_{w}$ is the scaling and $\hat{v}_{w}$ is from spherical Gaussian with identity covariance $I_{d\times d}$. Let $s$ be a random variable that has the same distribution as $s_{w}$.

We lowerbound the mean of $Z_{c}$ as follows:

where the last equality holds because of the symmetry of the spherical Gaussian distibution. On the other hand, to upperbound the mean of $Z_{c}$, we condition on the scaling $s_{w}$,

Note that conditioned on $s_{w}$, we have that $\langle v_{w},c\rangle$ is a Gaussian random variable with variance $\sigma^{2}=s_{w}^{2}$. Therefore,

We calculate the variance of $Z_{c}$ as follows:

By a very similar calculation as above, using the fact that $2\langle v_{w},c\rangle$ is a Gaussian random variable with variance $4\sigma^{2}=4s_{w}^{2}$,

for $\Lambda=\exp(8\kappa^{2})$ a constant, and at the last step we used the facts that $s\leq\kappa$ a.s. ∎

We fix the choice of $c$, and the proving the concentration using the randomness of $v_{w}$’s first. Note that that $\exp(\langle v_{w},c\rangle)$ is neither sub-Gaussian nor sub-exponential (actually the Orlicz norm of random variable $\exp(\langle v_{w},c\rangle)$ is never bounded). This prevents us from applying the usual concentration inequalities. The proof deals with this issue in a slightly more specialized manner.

Let’s define $\mathcal{F}_{w}$ be the event that $|\langle v_{w},c\rangle|\leq\frac{1}{2}\log n$. We claim that $\Pr[\mathcal{F}_{w}]\geq 1-\exp(-\Omega(\log^{2}n))$. Indeed note that $\left\langle v_{w},c\right\rangle\mid s_{w}$ has a Gaussian distribution with standard deviation $s_{w}\|c\|=s_{w}\leq 2\kappa$ a.s. Therefore by the Gaussianity of $\left\langle v_{w},c\right\rangle$ we have that

where $\Omega(\cdot)$ hides the dependency on $\kappa$ which is treated as absolute constants. Taking expectations over $s_{w}$, we obtain that

Note that by definition, we in particular have that conditioned on $\mathcal{F}_{w}$, it holds that $\exp(\left\langle v_{w},c\right\rangle)\leq\sqrt{n}$.

Let the random variable $X_{w}$ have the same distribution as $\exp(\langle v_{w},c\rangle)|_{\mathcal{F}_{w}}$. We prove that the random variable $Z^{\prime}_{c}=\sum_{w}X_{w}$ concentrates well. By convexity of the exponential function, we have that the mean of $Z_{c}^{\prime}$ is lowerbounded

Moreover, by definition, for any $w$, $|X_{w}|\leq\sqrt{n}$. Therefore by Bernstein’s inequality, we have that

Let $\mathcal{F}=\cup_{w}\mathcal{F}_{w}$ be the union of all $\mathcal{F}_{w}$. Then by union bound, it holds that $\Pr[\bar{\mathcal{F}}]\leq\sum_{w}\Pr[\bar{\mathcal{F}_{w}}]\leq n\cdot\exp(-\Omega(\log^{2}n))=\exp(-\Omega(\log^{2}n))$. We have that by definition, $Z_{c}^{\prime}$ has the same distribution as $Z_{c}|_{\mathcal{F}}$. Therefore, we have that

where at the last line we used the fact that $\Pr[\bar{\mathcal{F}}]\leq\exp(-\Omega(\log^{2}n))$ and equation (A.30).

Taking expectation over the randomness of $c$, we have that

Therefore by a standard averaging argument (using Markov inequality), we have

For now on we fix a choice of $v_{w}$’s so that \Pr_{c}[\textrm{~{}\eqref{eqn:lem_zc_proof} holds}]\geq 1-\exp(-\Omega(\log^{2}n)) is true. Therefore in the rest of the proof, only $c$ is random variable, and with probability $1-\exp(-\Omega(\log^{2}n))$ over the randomness of $c$, it holds that,

Appendix B Maximum likelihood estimator for co-occurrence

Assuming this approximation, we show below that the maximum likelihood values for the word vectors correspond to the following optimization,

Now we give the derivation of the objective. According to the multinomial distribution, maximizing the likelihood of $\{X_{w,w^{\prime}}\}$ is equivalent to maximizing

To reason about the likelihood, denote the logarithm of the ratio between the expected count and the empirical count as

When the last term is much smaller than the first term on the right hand side, maximizing the likelihood is approximately equivalent to minimizing the first term on the right hand side, which is our objective:

We now argue that the last term is much smaller than the first term on the right hand side in (B.2). For a large $X_{w,w^{\prime}}$, the $\Delta_{w,w^{\prime}}$ is close to 0 and thus the induced approximation error is small. Small $X_{w,w^{\prime}}$’s only contribute a small fraction of the final objective (3.3), so we can safely ignore the errors. To see this, note that the objective $\sum_{(w,w^{\prime})}X_{w,w^{\prime}}\Delta_{w,w^{\prime}}^{2}$ and the error term $\sum_{(w,w^{\prime})}X_{w,w^{\prime}}O(|\Delta_{w,w^{\prime}}|^{3})$ differ by a factor of $|\Delta_{w,w^{\prime}}|$ for each $X_{w,w^{\prime}}$. For large $X_{w,w^{\prime}}$’s, $|\Delta_{w,w^{\prime}}|\ll 1$, and thus their corresponding errors are much smaller than the objective. So we only need to consider the $X_{w,w^{\prime}}$’s that are small constants. The co-occurrence counts obey a power law distribution (see, e.g. Pennington et al. ). That is, if one sorts $\{X_{w,w^{\prime}}\}$ in decreasing order, then the $r$-th value in the list is roughly

where $k$ is some constant. Some calculation shows that