Improving Topic Models with Latent Feature Word Representations

Introduction

Topic modeling algorithms, such as Latent Dirichlet Allocation [Blei et al., 2003] and related methods [Blei, 2012], are often used to learn a set of latent topics for a corpus, and predict the probabilities of each word in each document belonging to each topic [Teh et al., 2006, Newman et al., 2006, Toutanova and Johnson, 2008, Porteous et al., 2008, Johnson, 2010, Xie and Xing, 2013, Hingmire et al., 2013].

Conventional topic modeling algorithms such as these infer document-to-topic and topic-to-word distributions from the co-occurrence of words within documents. But when the training corpus of documents is small or when the documents are short, the resulting distributions might be based on little evidence. ?) and ?) show that it helps to exploit external knowledge to improve the topic representations. ?) employed web search results to improve the information in short texts. ?) assumed that the small corpus is a sample of topics from a larger corpus like Wikipedia, and then use the topics discovered in the larger corpus to help shape the topic representations in the small corpus. However, if the larger corpus has many irrelevant topics, this will “use up” the topic space of the model. In addition, ?) proposed an extension of LDA that uses external information about word similarity, such as thesauri and dictionaries, to smooth the topic-to-word distribution.

Topic models have also been constructed using latent features [Salakhutdinov and Hinton, 2009, Srivastava et al., 2013, Cao et al., 2015]. Latent feature (lf) vectors have been used for a wide range of NLP tasks [Glorot et al., 2011, Socher et al., 2013, Pennington et al., 2014]. The combination of values permitted by latent features forms a high dimensional space which makes it is well suited to model topics of very large corpora.

Rather than relying solely on a multinomial or latent feature model, as in ?), ?) and ?), we explore how to take advantage of both latent feature and multinomial models by using a latent feature representation trained on a large external corpus to supplement a multinomial topic model estimated from a smaller corpus.

Our main contribution is that we propose two new latent feature topic models which integrate latent feature word representations into two Dirichlet multinomial topic models: a Latent Dirichlet Allocation (lda) model [Blei et al., 2003] and a one-topic-per-document Dirichlet Multinomial Mixture (dmm) model [Nigam et al., 2000]. Specifically, we replace the topic-to-word Dirichlet multinomial component which generates the words from topics in each Dirichlet multinomial topic model by a two-component mixture of a Dirichlet multinomial component and a latent feature component.

In addition to presenting a sampling procedure for the new models, we also compare using two different sets of pre-trained latent feature word vectors with our models. We achieve significant improvements on topic coherence evaluation, document clustering and document classification tasks, especially on corpora of short documents and corpora with few documents.

Background

The Latent Dirichlet Allocation (lda) topic model [Blei et al., 2003] represents each document $d$ as a probability distribution $\boldsymbol{\theta}_{d}$ over topics, where each topic $z$ is modeled by a probability distribution $\boldsymbol{\phi}_{z}$ over words in a fixed vocabulary $W$.

As presented in Figure 1, where $\alpha$ and $\beta$ are hyper-parameters and $T$ is number of topics, the generative process for lda is described as follows:

Notation: $N^{t,w}_{d}$ is the rank-3 tensor that counts the number of times that word $w$ is generated from topic $t$ in document $d$ by the Dirichlet multinomial component, which in section 2.1 belongs to the lda model, while in section 2.2 belongs to the dmm model. When an index is omitted, it indicates summation over that index (so $N_{d}$ is the number of words in document $d$).

$V$ is the size of the vocabulary, $V=|W|$.

2 dmm model for short texts

Applying topic models for short or few documents for text clustering is more challenging because of data sparsity and the limited contexts in such texts. One approach is to combine short texts into long pseudo-documents before training lda [Hong and Davison, 2010, Weng et al., 2010, Mehrotra et al., 2013]. Another approach is to assume that there is only one topic per document [Nigam et al., 2000, Zhao et al., 2011, Yin and Wang, 2014].

In the Dirichlet Multinomial Mixture (dmm) model [Nigam et al., 2000], each document is assumed to only have one topic. The process of generating a document $d$ in the collection $D$, as shown in Figure 1, is to first select a topic assignment for the document, and then the topic-to-word Dirichlet multinomial component generates all the words in the document from the same selected topic:

Notation: $M_{\neg d}^{t}$ is the number of documents assigned to topic $t$ excluding the current document $d$; $\Gamma$ is the Gamma function.

3 Latent feature vector models

Traditional count-based methods [Deerwester et al., 1990, Lund and Burgess, 1996, Bullinaria and Levy, 2007] for learning real-valued latent feature (lf) vectors rely on co-occurrence counts. Recent approaches based on deep neural networks learn vectors by predicting words given their window-based context [Collobert and Weston, 2008, Mikolov et al., 2013, Pennington et al., 2014, Liu et al., 2015].

?)’s method maximizes the log likelihood of each word given its context. ?) used back-propagation to minimize the squared error of a prediction of the log-frequency of context words within a fixed window of each word. Word vectors can be trained directly on a new corpus. In our new models, however, in order to incorporate the rich information from very large datasets, we utilize pre-trained word vectors that were trained on external billion-word corpora.

New latent feature topic models

In this section, we propose two novel probabilistic topic models, which we call the lf-lda and the lf-dmm, that combine a latent feature model with either an lda or dmm model. We also present Gibbs sampling procedures for our new models.

In general, lf-lda and lf-dmm are formed by taking the original Dirichlet multinomial topic models lda and dmm, and replacing their topic-to-word Dirichlet multinomial component that generates words from topics with a two-component mixture of a topic-to-word Dirichlet multinomial component and a latent feature component.

Informally, the new models have the structure of the original Dirichlet multinomial topic models, as shown in Figure 2, with the addition of two matrices $\boldsymbol{\tau}$ and $\boldsymbol{\omega}$ of latent feature weights, where $\boldsymbol{\tau}_{t}$ and $\boldsymbol{\omega}_{w}$ are the latent-feature vectors associated with topic $t$ and word $w$ respectively.

In the next two sections 3.1 and 3.2, we explain the generative processes of our new models lf-lda and lf-dmm. We then present our Gibbs sampling procedures for the models lf-lda and lf-dmm in the sections 3.3 and 3.4, respectively, and explain how we estimate $\boldsymbol{\tau}$ in section 3.5.

The lf-lda model generates a document as follows: a distribution over topics $\boldsymbol{\theta}_{d}$ is drawn for document $d$; then for each $i^{\mathit{th}}$ word $w_{d_{i}}$ (in sequential order that words appear in the document), the model chooses a topic indicator $z_{d_{i}}$, a binary indicator variable $s_{d_{i}}$ is sampled from a Bernoulli distribution to determine whether the word $w_{d_{i}}$ is to be generated by the Dirichlet multinomial or latent feature component, and finally the word is generated from the chosen topic by the determined topic-to-word model. The generative process is:

2 Generative process for the lf-dmm model

Our lf-dmm model uses the dmm model assumption that all the words in a document share the same topic. Thus, the process of generating a document in a document collection with our lf-dmm is as follows: a distribution over topics $\boldsymbol{\theta}$ is drawn for the document collection; then the model draws a topic indicator $z_{d}$ for the entire document $d$; for every $i^{\mathit{th}}$ word $w_{d_{i}}$ in the document $d$, a binary indicator variable $s_{d_{i}}$ is sampled from a Bernoulli distribution to determine whether the Dirichlet multinomial or latent feature component will be used to generate the word $w_{d_{i}}$, and finally the word is generated from the same topic $z_{d}$ by the determined component. The generative process is summarized as:

3 Inference in lf-lda model

From the generative model of lf-lda in Figure 2, by integrating out $\boldsymbol{\theta}$ and $\boldsymbol{\phi}$, we use the Gibbs sampling algorithm [Robert and Casella, 2004] to perform inference to calculate the conditional topic assignment probabilities for each word. The outline of the Gibbs sampling algorithm for the lf-lda model is detailed in Algorithm 1.

Here, $\mathbfit{S}$ denotes the distribution indicator variables for the whole document collection $D$. Instead of sampling $\boldsymbol{\tau}_{t}$ from the posterior, we perform MAP estimation as described in the section 3.5.

For sampling the topic $z_{d_{i}}$ and the binary indicator variable $s_{d_{i}}$ of the $i^{\mathit{th}}$ word $w_{d_{i}}$ in the document $d$, we integrate out $s_{d_{i}}$ in order to sample $z_{d_{i}}$ and then sample $s_{d_{i}}$ given $z_{d_{i}}$. We sample the topic $z_{d_{i}}$ using the conditional distribution as follows:

Then we sample ${s}_{d_{i}}$ conditional on $z_{d_{i}}=t$ with:

Notation: Due to the new models’ mixture architecture, we separate out the counts for each of the two components of each model. We define the rank-3 tensor $K^{t,w}_{d}$ as the number of times a word $w$ in document $d$ is generated from topic $t$ by the latent feature component of the generative lf-lda or lf-dmm model.

We also extend the earlier definition of the tensor $N^{t,w}_{d}$ as the number of times a word $w$ in document $d$ is generated from topic $t$ by the Dirichlet multinomial component of our combined models, which in section 3.3 refers to the lf-lda model, while in section 3.4 refers to the lf-dmm model. For both tensors $K$ and $N$, omitting an index refers to summation over that index and negation $\neg$ indicates exclusion as before. So $N^{w}_{d}+K^{w}_{d}$ is the total number of times the word type $w$ appears in the document $d$.

4 Inference in lf-dmm model

For the lf-dmm model, we integrate out $\boldsymbol{\theta}$ and $\boldsymbol{\phi}$, and then sample the topic $z_{d}$ and the distribution selection variables $\mathbfit{s}_{d}$ for document $d$ using Gibbs sampling as outlined in Algorithm 2.

As before in Algorithm 1, we also use MAP estimation of $\boldsymbol{\tau}$ as detailed in section 3.5 rather than sampling from the posterior. The conditional distribution of topic variable and selection variables for document $d$ is:

Then we sample the binary indicator variable $s_{d_{i}}$ for each $i^{\mathit{th}}$ word $w_{d_{i}}$ in document $d$ conditional on $z_{d}=t$ from the following distribution:

5 Learning latent feature vectors for topics

To estimate the topic vectors after each Gibbs sampling iteration through the data, we apply regularized maximum likelihood estimation. Applying MAP estimation to learn log-linear models for topic models is also used in SAGE [Eisenstein et al., 2011] and SPRITE [Paul and Dredze, 2015]. However, unlike our models, those models do not use latent feature word vectors to characterize topic-word distributions. The negative log likelihood of the corpus $L$ under our model factorizes topic-wise into factors $L_{t}$ for each topic. With $L_{2}$ regularizationThe $L_{2}$ regularizer constant was set to $\mu=0.01$. for topic $t$, these are:

The MAP estimate of topic vectors $\boldsymbol{\tau}_{t}$ is obtained by minimizing the regularized negative log likelihood. The derivative with respect to the $j^{\mathit{th}}$ element of the vector for topic $t$ is:

We used l-bfgsWe used the L-BFGS implementation from the Mallet toolkit [McCallum, 2002].[Liu and Nocedal, 1989] to find the topic vector $\boldsymbol{\tau}_{t}$ that minimizes $L_{t}$.

Experiments

To investigate the performance of our new lf-lda and lf-dmm models, we compared their performance against baseline lda and dmm models on topic coherence, document clustering and document classification evaluations. The topic coherence evaluation measures the coherence of topic-word associations, i.e. it directly evaluates how coherent the assignment of words to topics is. The document clustering and document classification tasks evaluate how useful the topics assigned to documents are in clustering and classification tasks.

Because we expect our new models to perform comparatively well in situations where there is little data about topic-to-word distributions, our experiments focus on corpora with few or short documents. We also investigated which values of $\lambda$ perform well, and compared the performance when using two different sets of pre-trained word vectors in these new models.

We experimented with two state-of-the-art sets of pre-trained word vectors here.

Google word vectorsDownload at: https://code.google.com/p/word2vec/ are pre-trained 300-dimensional vectors for 3 million words and phrases. These vectors were trained on a 100 billion word subset of the Google News corpus by using the Google Word2Vec toolkit [Mikolov et al., 2013]. Stanford vectorsDownload at: http://www-nlp.stanford.edu/projects/glove/ are pre-trained 300-dimensional vectors for 2 million words. These vectors were learned from 42-billion tokens of Common Crawl web data using the Stanford GloVe toolkit [Pennington et al., 2014].

We refer to our lf-lda and lf-dmm models using Google and Stanford word vectors as w2v-lda, glove-lda, w2v-dmm and glove-dmm.

1.2 Experimental datasets

We conducted experiments on the 20-Newsgroups dataset, the TagMyNews news dataset and the Sanders Twitter corpus.

The 20-Newsgroups datasetWe used the “all-terms” version of the 20-Newsgroups dataset available at http://web.ist.utl.pt/acardoso/datasets/ [Cardoso-Cachopo, 2007]. contains about 19,000 newsgroup documents evenly grouped into 20 different categories. The TagMyNews news datasetThe TagMyNews news dataset is unbalanced, where the largest category contains 8,200 news items while the smallest category contains about 1,800 items. Download at: http://acube.di.unipi.it/tmn-dataset/ [Vitale et al., 2012] consists of about 32,600 English RSS news items grouped into 7 categories, where each news document has a news title and a short description. In our experiments, we also used a news title dataset which consists of just the news titles from the TagMyNews news dataset.

Each dataset was down-cased, and we removed non-alphabetic characters and stop-words found in the stop-word list in the Mallet toolkit [McCallum, 2002]. We also removed words shorter than 3 characters and words appearing less than 10 times in the 20-Newsgroups corpus, and under 5 times in the TagMyNews news and news titles datasets. In addition, words not found in both Google and Stanford vector representations were also removed.1366, 27 and 12 words were correspondingly removed out of the 20-Newsgroups, TagMyNews news and news title datasets. We refer to the cleaned 20-Newsgroups, TagMyNews news and news title datasets as N20, TMN and TMNtitle, respectively.

We also performed experiments on two subsets of the N20 dataset. The N20short dataset consists of all documents from the N20 dataset with less than 21 words. The N20small dataset contains 400 documents consisting of 20 randomly selected documents from each group of the N20 dataset.

Finally, we also experimented on the publicly available Sanders Twitter corpus.Download at: http://www.sananalytics.com/lab/index.php This corpus consists of 5,512 Tweets grouped into four different topics (Apple, Google, Microsoft, and Twitter). Due to restrictions in Twitter’s Terms of Service, the actual Tweets need to be downloaded using 5,512 Tweet IDs. There are 850 Tweets not available to download. After removing the non-English Tweets, 3,115 Tweets remain. In addition to converting into lowercase and removing non-alphabetic characters, words were normalized by using a lexical normalization dictionary for microblogs [Han et al., 2012]. We then removed stop-words, words shorter than 3 characters or appearing less than 3 times in the corpus. The four words apple, google, microsoft and twitter were removed as these four words occur in every Tweet in the corresponding topic. Moreover, words not found in both Google and Stanford vector lists were also removed.There are 91 removed words. In all our experiments, after removing words from documents, any document with a zero word count was also removed from the corpus. For the Twitter corpus, this resulted in just 2,520 remaining Tweets.

1.3 General settings

The hyper-parameter $\beta$ used in baseline lda and dmm models was set to 0.01, as this is a common setting in the literature [Griffiths and Steyvers, 2004]. We set the hyper-parameter $\alpha=0.1$, as this can improve performance relative to the standard setting $\alpha=\frac{50}{T}$, as noted by ?) and ?).

We ran each baseline model for 2000 iterations and evaluated the topics assigned to words in the last sample. For our models, we ran the baseline models for 1500 iterations, then used the outputs from the last sample to initialize our models, which we ran for 500 further iterations.

We report the mean and standard deviation of the results of ten repetitions of each experiment (so the standard deviation is approximately 3 standard errors, or a 99% confidence interval).

2 Topic coherence evaluation

This section examines the quality of the topic-word mappings induced by our models. In our models, topics are distributions over words. The topic coherence evaluation measures to what extent the high-probability words in each topic are semantically coherent [Chang et al., 2009, Stevens et al., 2012].

?), ?) and ?) describe methods for automatically evaluating the semantic coherence of sets of words. The method presented in ?) uses the normalized pointwise mutual information (npmi) score and has a strong correlation with human-judged coherence. A higher npmi score indicates that the topic distributions are semantically more coherent. Given a topic $t$ represented by its top-$N$ topic words $w_{1},w_{2},...,w_{N}$, the npmi score for $t$ is:

where the probabilities in equation (14) are derived from a 10-word sliding window over an external corpus.

The npmi score for a topic model is the average score for all topics. We compute the npmi score based on top-15 most probable words of each topic and use the English WikipediaWe used the Wikipedia-articles dump of July 8, 2014. of 4.6 million articles as our external corpus.

Figures 3 and 4 show npmi scores computed for the lda, w2v-lda and glove-lda models on the N20short dataset for 20 and 40 topics. We see that $\lambda=1.0$ gives the highest npmi score. In other words, using only the latent feature model produces the most coherent topic distributions.

Tables 2, 3 and 4 present the npmi scores produced by the models on the other experimental datasets, where we vary We perform with $T=6$ on the N20 and N20small datasets as the 20-Newsgroups dataset could be also grouped into 6 larger topics instead of 20 fine-grained categories. the number of topics in steps from $4$ to $80$. Tables 3 and 4 show that the dmm model performs better than the lda model on the TMN, TMNtitle and Twitter datasets. These results show that our latent feature models produce significantly higher scores than the baseline models on all the experimental datasets.

Google word2vec vs. Stanford glove word vectors: In general, our latent feature models obtain competitive npmi results in using pre-trained Google word2vec and Stanford glove word vectors for a large value of $T$, for example $T=80$. With small values of $T$, for example $T\leq 7$ , using Google word vectors produces better scores than using Stanford word vectors on the small N20small dataset of normal texts and on the short text TMN and TMNtitle datasets. However, the opposite pattern holds on the full N20 dataset. Both sets of the pre-trained word vectors produce similar scores on the small and short Twitter dataset.

2.2 Qualitative analysis

This section provides an example of how our models improve topic coherence. Table 5 compares the top-15 wordsIn the baseline model, the top-15 topical words output from the $1500^{\mathit{th}}$ sample are similar to top-15 words from the $2000^{\mathit{th}}$ sample if we do not take the order of the most probable words into account. produced by the baseline dmm model and our w2v-dmm model with $\lambda=1.0$ on the TMNtitle dataset with $T=20$ topics.

In table 5, topic $1$ of the dmm model consists of words related to “nuclear crisis in Japan” together with other unrelated words. The w2v-dmm model produced a purer topic $1$ focused on “Japan earthquake and nuclear crisis,” presumably related to the “Fukushima Daiichi nuclear disaster.” Topic $3$ is about “oil prices” in both models. However, all top-15 words are qualitatively more coherent in the w2v-dmm model. While topic $4$ of the dmm model is difficult to manually label, topic $4$ of the w2v-dmm model is about the “Arab Spring” event.

Topics $5$, $19$ and $14$ of the dmm model are not easy to label. Topic $5$ relates to “entertainment”, topic $19$ is generally a mixture of “entertainment” and “sport”, and topic $14$ is about “sport” and “politics.” However, the w2v-dmm model more clearly distinguishes these topics: topic $5$ is about “entertainment”, topic $19$ is only about “sport” and topic $14$ is only about “politics.”

3 Document clustering evaluation

We compared our models to the baseline models in a document clustering task. After using a topic model to calculate the topic probabilities of a document, we assign every document the topic with the highest probability given the document [Cai et al., 2008, Lu et al., 2011, Xie and Xing, 2013, Yan et al., 2013]. We use two common metrics to evaluate clustering performance: Purity and normalized mutual information (nmi): see [Manning et al., 2008, Section 16.3] for details of these evaluations. Purity and nmi scores always range from 0.0 to 1.0, and higher scores reflect better clustering performance.

Figures 5 and 6 present Purity and nmi results obtained by the lda, w2v-lda and glove-lda models on the N20short dataset with the numbers of topics $T$ set to either 20 or 40, and the value of the mixture weight $\lambda$ varied from 0.0 to 1.0.

We found that setting $\lambda$ to 1.0 (i.e. using only the latent features to model words), the glove-lda produced 1%+ higher scores on both Purity and nmi results than the w2v-lda when using $20$ topics. However, the two models glove-lda and w2v-lda returned equivalent results with $40$ topics where they gain 2%+ absolute improvementUsing the Student’s t-Test, the improvement is significant ($p<0.01$). on the two Purity and nmi against the baseline lda model.

By varying $\lambda$, as shown in Figures 5 and 6, the w2v-lda and glove-lda models obtain their best results at $\lambda=0.6$ where the w2v-lda model does slightly better than the glove-lda. Both models significantly outperform their baseline lda models; for example with $40$ topics, the w2v-lda model attains 4.4% and 4.3% over the lda model on Purity and nmi metrics, respectively.

We fix the mixture weight $\lambda$ at 0.6, and report experimental results based on this value for the rest of this section. Tables 6, 7 and 8 show clustering results produced by our models and the baseline models on the remaining datasets with different numbers of topics. As expected, the dmm model is better than the lda model on the short datasets of TMN, TMNtitle and Twitter. For example with 80 topics on the TMNtitle dataset, the dmm achieves about 7+% higher Purity and nmi scores than lda.

New models vs. baseline models: On most tests, our models score higher than the baseline models, particularly on the small N20small dataset where we get 6.0% improvement on nmi at $T=6$, and on the short text TMN and TMNtitle datasets we obtain 6.1% and 2.5% higher Purity at $T=80$. In addition, on the short and small Twitter dataset with $T=4$, we achieve 3.9% and 5.3% improvements in Purity and nmi scores, respectively. Those results show that an improved model of topic-word mappings also improves the document-topic assignments.

For the small value of $T\leq 7$, on the large datasets of N20, TMN and TMNtitle, our models and baseline models obtain similar clustering results. However, with higher values of $T$, our models perform better than the baselines on the short TMN and TMNtitle datasets, while on the N20 dataset, the baseline lda model attains a slightly higher clustering results than ours. In contrast, on the short and small Twitter dataset, our models obtain considerably better clustering results than the baseline models with a small value of $T$.

Google word2vec vs. Stanford glove word vectors: On the small N20short and N20small datasets, using the Google pre-trained word vectors produces higher clustering scores than using Stanford pre-trained word vectors. However, on the large datasets N20, TMN and TMNtitle, using Stanford word vectors produces higher scores than using Google word vectors when using a smaller number of topics, for example $T\leq 20$. With more topics, for instance $T=80$, the pre-trained Google and Stanford word vectors produce similar clustering results. In addition, on the Twitter dataset, both sets of pre-trained word vectors produce similar results.

4 Document classification evaluation

Unlike the document clustering task, the document classification task evaluates the distribution over topics for each document. Following ?), ?), ?) and ?), we used Support Vector Machines (SVM) to predict the ground truth labels from the topic-proportion vector of each document. We used the WEKA’s implementation [Hall et al., 2009] of the fast Sequential Minimal Optimization algorithm [Platt, 1999] for learning a classifier with ten-fold cross-validation and WEKA’s default parameters. We present the macro-averaged $F_{1}$ score [Manning et al., 2008, Section 13.6] as the evaluation metric for this task.

Just as in the document clustering task, the mixture weight $\lambda=0.6$ obtains the highest classification performances on the N20short dataset. For example with $T=40$, our w2v-lda and glove-lda obtain $F_{1}$ scores at 40.0% and 38.9% which are 4.5% and 3.4% higher than $F_{1}$ score at 35.5% obtained by the lda model, respectively.

We report classification results on the remaining experimental datasets with mixture weight $\lambda=0.6$ in tables 9, 10 and 11. Unlike the clustering results, the lda model does better than the dmm model for classification on the TMN dataset.

New models vs. baseline models: On most evaluations, our models perform better than the baseline models. In particular, on the small N20small and Twitter datasets, when the number of topics $T$ is equal to number of ground truth labels (i.e. 20 and 4 correspondingly), our w2v-lda obtains $5^{+}$% higher $F_{1}$ score than the lda model. In addition, our w2v-dmm model achieves 5.4% and 2.9% higher $F_{1}$ score than the dmm model on short TMN and TMNtitle datasets with $T=80$, respectively.

Google word2vec vs. Stanford glove word vectors: The comparison of the Google and Stanford pre-trained word vectors for classification is similar to the one for clustering.

5 Discussion

We found that the topic coherence evaluation produced the best results with a mixture weight $\lambda=1$, which corresponds to using topic-word distributions defined in terms of the latent-feature word vectors. This is not surprising, since the topic coherence evaluation we used [Lau et al., 2014] is based on word co-occurrences in an external corpus (here, Wikipedia), and it is reasonable that the billion-word corpora used to train the latent feature word vectors are more useful for this task than the much smaller topic-modeling corpora, from which the topic-word multinomial distributions are trained.

On the other hand, the document clustering and document classification tasks depend more strongly on possibly idiosyncratic properties of the smaller topic-modeling corpora, since these evaluations reflect how well the document-topic assignments can group or distinguish documents within the topic-modeling corpus. Smaller values of $\lambda$ enable the models to learn topic-word distributions that include an arbitrary multinomial topic-word distribution, enabling the models to capture idiosyncratic properties of the topic-modeling corpus. Even in these evaluations we found that an intermediate value of $\lambda=0.6$ produced the best results, indicating that better word-topic distributions were produced when information from the large external corpus is combined with corpus-specific topic-word multinomials. We found that using the latent feature word vectors produced significant performance improvements even when the domain of the topic-modeling corpus was quite different to that of the external corpus from which the word vectors were derived, as was the case in our experiments on Twitter data.

We found that using either the Google or the Stanford latent feature word vectors produced very similar results. As far as we could tell, there is no reason to prefer either one of these in our topic modeling applications.

Conclusion and future work

In this paper, we have shown that latent feature representations can be used to improve topic models. We proposed two novel latent feature topic models, namely lf-lda and lf-dmm, that integrate a latent feature model within two topic models lda and dmm. We compared the performance of our models lf-lda and lf-dmm to the baseline lda and dmm models on topic coherence, document clustering and document classification evaluations. In the topic coherence evaluation, our model outperformed the baseline models on all 6 experimental datasets, showing that our method for exploiting external information from very large corpora helps improve the topic-to-word mapping. Meanwhile, document clustering and document classification results show that our models improve the document-topic assignments compared to the baseline models, especially on datasets with few or short documents.

As an anonymous reviewer suggested, it would be interesting to identify exactly how the latent feature word vectors improve topic modeling performance. We believe that they provide useful information about word meaning extracted from the large corpora that they are trained on, but as the reviewer suggested, it is possible that the performance improvements arise because the word vectors are trained on context windows of size 5 or 10, while the lda and dmm models view documents as bags of words, and effectively use a context window that encompasses the entire document. In preliminary experiments where we train latent feature word vectors from the topic-modeling corpus alone using context windows of size 10 we found that performance was degraded relative to the results presented here, suggesting that the use of a context window alone is not responsible for the performance improvements we reported here. Clearly it would be valuable to investigate this further.

In order to use a Gibbs sampler in section 3.4, the conditional distributions needed to be distributions we can sample from cheaply, which is not the case for the ratios of Gamma functions. While we used a simple approximation, it is worth exploring other sampling techniques that can avoid approximations, such as Metropolis-Hastings sampling [Bishop, 2006, Section 11.2.2].

In order to compare the pre-trained Google and Stanford word vectors, we excluded words that did not appear in both sets of vectors. As suggested by anonymous reviewers, it would be interesting to learn vectors for these unseen words. In addition, it is worth fine-tuning the seen-word vectors on the dataset of interest.

Although we have not evaluated our approach on very large corpora, the corpora we have evaluated on do vary in size, and we showed that the gains from our approach are greatest when the corpora are small. A drawback of our approach is that it is slow on very large corpora. Variational Bayesian inference may provide an efficient solution to this problem [Jordan et al., 1999, Blei et al., 2003].

Acknowledgments

This research was supported by a Google award through the Natural Language Understanding Focused Program, and under the Australian Research Council’s Discovery Projects funding scheme (project numbers DP110102506 and DP110102593). The authors would like to thank the three anonymous reviewers, the action editor and Dr. John Pate at the Macquarie University, Australia for helpful comments and suggestions.

References