Nested Hierarchical Dirichlet Processes

Introduction

Organizing things hierarchically is a natural aspect of human activity. Walking into a large department store, one might first find the men’s section, followed by men’s casual, and then see the t-shirts hanging along the wall. Or being hungry, one might choose to eat Italian food, decide whether to spring for the better, more authentic version or go to one of the cheaper chain options, and then end up at the Olive Garden. Similarly with data analysis, a hierarchical tree-structured representation of data can provide an illuminating means for understanding and reasoning about the information it contains.

In this paper, we focus on developing hierarchical topic models to construct tree-structured representations for text data. Hierarchical topic models use a structured prior on the topics underlying a corpus of documents, with the aim of bringing more order to an unstructured set of thematic concepts . They do this by learning a tree structure for the underlying topics, with the inferential goal being that topics closer to the root are more general, and gradually become more specific in thematic content when following a path down the tree.

Our work builds on the nested Chinese restaurant process (nCRP) . The nCRP is a Bayesian nonparametric prior for hierarchical topic models, but is limited in that it assumes each document selects topics from one path in the tree. We illustrate this limitation in Figure 1. This assumption has practical drawbacks; for trees truncated to a small number of levels this does not allow for many topics per document, and for trees of many levels there are too many nodes to infer.

The nCRP also has drawbacks from a modeling standpoint. As a simple example, consider an article on ESPN.com about an injured player, compared with an article in a sports medicine journal about a specific type of athletic injury. Both documents will contain words about medicine and words about sports. These areas are different enough, however, that one cannot be considered to be a subset of the other. Yet the single-path structure of the nCRP will require this to be the case in order to model the relevant words in the documents, or it will learn a new “sports/medicine” topic rather than a mixture of separate sports and medicine topics. Continuing this analogy, other documents may only be about sports or medicine. As a result, medical terms in the nCRP will need to appear in multiple places within the tree: in its own subtree separate from sports, and also affiliated with sports, perhaps as a child of the general sports topic (in the case of the ESPN article). A similar fractionation of sports-related terms results from the sports medicine article, where the medical terms dominate and sports can be considered a topic underneath the main medicine topic. The result is a tree where topics appear in multiple places, and so the full statistical power within the corpus is not being used to model each topic; the tree will not be as compact as it could be.

Though the nCRP is a Bayesian nonparametric prior, it performs nonparametric clustering of document-specific paths, which reduces the number of topics available to a document by restricting them to lie on a single path, leading to drawbacks as illustrated above. Our goal is to develop a related Bayesian nonparametric prior that performs word-specific path clustering. We illustrate this objective in Figure 1. In this case, each word has access to the entire tree, but with document-specific distributions on the paths within the tree. To this end, we make use of the hierarchical Dirichlet process , developing a novel prior that we refer to as the nested hierarchical Dirichlet process (nHDP). The HDP can be viewed as a nonparametric elaboration of the classical topic model, latent Dirichlet allocation (LDA) , providing a mechanism whereby a global Dirichlet process defines a base distribution for a collection of local Dirichlet processes, one for each document. With the nHDP, we extend this idea by letting a global nCRP become a base distribution for a collection of local nCRPs, one for each document. As illustrated in Figure 1, the nested HDP provides the opportunity for cross-thematic borrowing while keeping general topic areas in separate subtrees, which is not possible with the nCRP.

Hierarchical topic models have thus far been applied to corpora of small size. A significant issue, not just with topic models but with Bayesian models in general, is to scale up inference to massive data sets . Recent developments in stochastic variational inference methods have shown promising results for LDA and the HDP topic model . We continue this development for hierarchical topic modeling with the nested HDP. Using stochastic variational inference, we demonstrate an ability to efficiently handle very large corpora. This is a major benefit to complex models such as tree-structured topic models, which require significant amounts of data to support their large size.

We organize the paper as follows: In Section 2 we review the Bayesian nonparametric priors that we incorporate in our model—the Dirichlet process, nested Chinese restaurant process and hierarchical Dirichlet process. In Section 3 we present our proposed nested HDP model for hierarchical topic modeling. In Section 4 we review stochastic variational inference and present an inference algorithm for nHDPs that scales well to massive data sets. We present empirical results in Section 5. We first compare the nHDP with the nCRP on three relatively small data sets. We then evaluate our stochastic algorithm on 1.8 million documents from The New York Times and 2.7 million documents from Wikipedia, comparing performance with stochastic LDA and HDP.

Background: Bayesian nonparametric priors for topic models

The nested hierarchical Dirichlet process (nHDP) builds on a collection of existing Bayesian nonparametric priors. In this section, we review these priors: the Dirichlet process, nested Chinese restaurant process and hierarchical Dirichlet process. We also review constructive representations for these processes that we will use for posterior inference of the nHDP topic model.

The Dirichlet process (DP) is the foundation for a large collection of Bayesian nonparametric models that rely on mixtures to represent distributions on data. Mixture models work by partitioning a data set according to statistical traits shared by members of the same cell. Dirichlet process priors are effective in learning a suitable number of traits for representing the data, in addition to the parameters of the mixture. The basic form of a Dirichlet process mixture model is

With this representation, data $W_{1},\dots,W_{N}$ are distributed according to a family of distributions $F_{W}$ with respective parameters $\varphi_{1},\dots,\varphi_{N}$. These parameters are drawn from the distribution $G$, which is discrete and potentially infinite, as the DP allows it to be. This discreteness induces a partition of the data $W$ according to the sharing of the atoms $\{\theta_{i}\}$ among the parameters $\{\varphi_{n}\}$ that are selected.

The Dirichlet process is a stochastic process for generating $G$. To briefly review, let $(\Theta,\mathcal{B})$ be a measurable space, $G_{0}$ a probability measure on it and $\alpha>0$. Ferguson proved the existence of a stochastic process $G$ where, for all measurable partitions $\{B_{1},\dots,B_{k}\}$ of $\Theta$, with $B_{i}\in\mathcal{B}$,

abbreviated as $G\sim\mbox{DP}(\alpha G_{0})$. It has been shown that $G$ is discrete (with probability one) even when $G_{0}$ is non-atomic . Thus the DP prior is a good candidate for $G$ in Eq. (1) since it generates discrete distributions on continuous parameter spaces. For most applications $G_{0}$ is diffuse, and so representations of $G$ at the granularity of the atoms are necessary for inference; we next review two of these approaches to working with this infinite-dimensional distribution.

The Chinese restaurant process (CRP) avoids directly working with $G$ by integrating it out . In doing so, the values of $\varphi_{1},\dots,\varphi_{N}$ become dependent, with the value of $\varphi_{n+1}$ given $\varphi_{1},\dots,\varphi_{n}$ distributed as

That is, $\varphi_{n+1}$ takes the value of one of the previously observed $\varphi_{i}$ with probability $\frac{n}{\alpha+n}$, and a value drawn from $G_{0}$ with probability $\frac{\alpha}{\alpha+n}$, which will be unique when $G_{0}$ is continuous. This displays the clustering property of the CRP and also gives insight into the impact of $\alpha$, since it is evident that the number of unique $\varphi_{i}$ grows like $\alpha\ln n$. In the limit $n\rightarrow\infty$, the distribution in Eq. (2) converges to a random measure distributed according to a Dirichlet process . The CRP is so-called because of an analogy to a Chinese restaurant, where a new customer (datum) sits at a table (selects a parameter) with probability proportional to the number of previous customers at that table, or selects a new table with probability proportional to $\alpha$.

1.2 A stick-breaking construction

Where the Chinese restaurant process works with $G\sim\mbox{DP}(\alpha G_{0})$ implicitly through $\varphi$, a stick-breaking construction allows one to directly construct $G$ before drawing any $\varphi_{n}$. Sethuraman showed that if $G$ is constructed as follows:

2 Nested Chinese restaurant processes

Nested Chinese restaurant processes (nCRP) are a tree-structured extension of the CRP that are useful for hierarchical topic modeling . They extend the CRP analogy to a nesting of restaurants in the following way: After selecting a table (parameter) according to a CRP, the customer departs for another restaurant uniquely indicated by that table. Upon arrival, the customer acts according to the CRP for the new restaurant, and again departs for a restaurant only accessible through the table selected. This occurs for a potentially infinite sequence of restaurants, which generates a sequence of parameters for the customer according to the selected tables.

A natural interpretation of the nCRP is as a tree where each parent has an infinite number of children. Starting from the root node, a path is traversed down the tree. Given the current node, a child node is selected with probability proportional to the previous number of times it was selected among its siblings, or a new child is selected with probability proportional to $\alpha$. As with the CRP, the underlying mixing measure of the nCRP also has a constructive representation useful for variational inference, which we will use in our nHDP construction.

If the next node is child $j$, then the nCRP transitions to DP $G_{\textbf{{i}}_{l+1}}$, where $\textbf{{i}}_{l+1}$ has index $j$ appended to $\textbf{{i}}_{l}$, that is $\textbf{{i}}_{l+1}=(\textbf{{i}}_{l},j)$. A path down the tree givens a sequence of parameters $\boldsymbol{\varphi}=(\varphi_{1},\varphi_{2},\dots)$, where the parameter $\varphi_{l}$ correspond to an atom $\theta_{\textbf{{i}}_{l}}$ at level $l$. Hierarchical topic models use these sequences of parameters to give the topics for generating documents. Other nested DPs have been considered as well, such as a two-leveled nDP where all parameters are selected from the leaves .

2.2 Nested CRP topic models

Hierarchical topic models based on the nested CRP use a globally shared tree to generate a corpus of documents. Starting with the construction of nested Dirichlet processes as described above, each document selects a path down the tree according to a Markov process, which produces a sequence of topics $\boldsymbol{\varphi}_{d}=(\varphi_{d,1},\varphi_{d,2},\dots)$ used to generate the $d$th document. As with other topic models, each word in a document, $W_{d,n}$, is represented by an index in the set $\{1,\dots,\mathcal{V}\}$ and the topics $\theta_{\textbf{{i}}_{l}}$ appearing in $\boldsymbol{\varphi}_{d}$ are $\mathcal{V}$-dimensional probability vectors with Dirichlet prior $G_{0}=\mbox{Dirichlet}(\lambda_{0}\boldsymbol{1}_{\mathcal{V}})$.

For each document $d$, an additional stick-breaking process provides a distribution on the topics in $\boldsymbol{\varphi}_{d}$,

Since this is not a DP, $U_{d,j}$ has two free parameters, $\gamma_{1}$ and $\gamma_{2}$. Following the standard method, words for document $d$ are generated by first drawing a topic i.i.d. from $G^{(d)}$, and then drawing the word index from the discrete distribution with the selected topic.

2.3 Issues with the nCRP

As discussed in the introduction, a significant drawback of the nCRP for topic modeling is that each document follows one path down the tree. Therefore, all thematic content of a document must be contained within that single sequence of topics. Since the nCRP is meant to characterize the thematic content of a corpus in increasing levels of specificity, this creates a combinatorial problem, where similar topics will appear in many parts of the tree to account for the possibility that they appear as a topic of the document (e.g., the sport/medicine example given in the introduction). In practice, nCRP trees are typically truncated at three levels , since learning deeper levels becomes difficult due to the exponential increase in nodes.This includes a root node topic, which is shared by all documents and is intended to collect stop words. In this situation each document has three topics for modeling its entire thematic content, which is likely insufficient, and so a blending of multiple topics is bound to occur during inference.

The nCRP is a Bayesian nonparametric (BNP) prior, but it performs nonparametric clustering of the paths selected at the document level, rather than at the word level. Though the same distribution on a tree is shared by a corpus, each document can differentiate itself only by the path it choses, as well as the distribution on topics in that path. The key issue with the nCRP is the restrictiveness of this single path allowed to a document. However, if instead each word were allowed to follow its own path according to an nCRP, the distribution on paths would be the same for all documents, which is clearly not desired. Our goal is to develop a hierarchical topic model that does not prohibit a document from using topics in different parts of the tree. Our solution to this problem is to employ the hierarchical Dirichlet process (HDP).

3 Hierarchical Dirichlet processes

The HDP is a multi-level version of the Dirichlet process . It makes use of the idea that the base distribution on the continuous space $\Theta$ can be discrete, which is useful because a discrete distribution allows for multiple draws from the DP prior to place probability mass on the same subset of atoms. Hence different groups of data can share the same atoms, but have different probability distributions on them. A discrete base is needed, but the atoms are unknown in advance. The HDP models these atoms by drawing the base from a DP prior. This leads to the hierarchical process

for groups $d=1,\dots,D$. This prior has been used to great effect in topic modeling as a nonparametric extension of LDA and related LDA-based models .

As with the DP, explicit representations of the HDP are necessary for inference. The representation we use relies on two levels of Sethuraman’s stick breaking construction. For this construction, first sample $G$ as in Eq. (3), and then sample $G_{d}$ in the same way,

This form is identical to Eq. (3), with the key difference that $G$ is discrete, and so atoms $\phi_{i}$ will repeat. An advantage of this representation is that all random variables are i.i.d., which aids variational inference strategies.

Nested hierarchical Dirichlet processes for topic modeling

In building on the nCRP framework, our goal is to allow for each document to have access to the entire tree, while still learning document-specific distributions on topics that are thematically coherent. Ideally, each document will still exhibit a dominant path corresponding to its main themes, but with off-shoots allowing for other topics. Our two major changes to the nCRP formulation toward this end are that ($i$) each word follows its own path to a topic, and ($ii$) each document has its own distribution on paths in a shared tree. The BNP tools discussed above make this a straightforward task.

In the proposed nested hierarchical Dirichlet process (nHDP), we split the process of generating a document’s distribution on topics into two parts: first, generating a document’s distribution on paths down the tree, and second, generating a word’s distribution on terminating at a particular node within those paths.

With the nHDP, all documents share a global nCRP drawn according to the stick-breaking construction in Section 2.2.1. Denote this tree by $\mathcal{T}$. As discussed, $\mathcal{T}$ is simply an infinite collection of Dirichlet processes with a continuous base distribution $G_{0}$ and a transition rule between DPs. According to this rule, from a root Dirichlet process $G_{\textbf{{i}}_{0}}$, a path is followed by drawing $\varphi_{l+1}\sim G_{\textbf{{i}}_{l}}$ for $l=0,1,2,\dots$, where $\textbf{{i}}_{0}$ is a constant root index, and $\textbf{{i}}_{l}=(i_{1},\dots,i_{l})$ indexes the DP associated with the topic $\varphi_{l}=\theta_{\textbf{{i}}_{l}}$. With the nested HDP, instead of following paths according to the global $\mathcal{T}$, we use each Dirichlet process in $\mathcal{T}$ as a base distribution for a local DP drawn independently for each document.

That is, for document $d$ we construct a tree $\mathcal{T}_{d}$ where, for each $G_{\textbf{{i}}_{l}}\in\mathcal{T}$, we draw a corresponding $G^{(d)}_{\textbf{{i}}_{l}}\in\mathcal{T}_{d}$ according to the Dirichlet process

As discussed in Section 2.3, $G^{(d)}_{\textbf{{i}}_{l}}$ will have the same atoms as $G_{\textbf{{i}}_{l}}$, but with different probability weights on them. Therefore, the tree $\mathcal{T}_{d}$ will have the same nodes as $\mathcal{T}$, but the probability of a path in $\mathcal{T}_{d}$ will vary with $d$, giving each document its own distribution on the tree.

We represent this document-specific DP with a stick-breaking construction as in Section 2.3,

This representation retains full independence among random variables, and will lead to a simpler stochastic variational inference algorithm. We note that the atoms from the global DP are randomly permuted and copied with this construction; $\phi^{(d)}_{\textbf{{i}}_{l},j}$ does not correspond to the node with parameter $\theta_{(\textbf{{i}}_{l},j)}$. To find the probability mass that $G^{(d)}_{\textbf{{i}}_{l}}$ places on $\theta_{(\textbf{{i}}_{l},j)}$, one can calculate

Using this nesting of HDPs to construct $\mathcal{T}_{d}$, each document has a tree with transition probabilities defined over the same subset of nodes since $\mathcal{T}$ is discrete, but with values for these probabilities that are document specific. To see how this allows each word to follow its own path while still producing a thematically coherent document, consider each $G^{(d)}_{\textbf{{i}}_{l}}$ when $\beta$ is small. In this case, most of the probability will be placed on one atom selected from $G_{\textbf{{i}}_{l}}$ since the first proportion $V^{(d)}_{\textbf{{i}}_{l},1}$ will be large with high probability. This will leave little probability remaining for other atoms, a feature shared by all DPs in $\mathcal{T}_{d}$. Starting from the root node of $\mathcal{T}_{d}$, each word in the document will have high probability of transitioning to the same node when moving down the tree, with some small probability of diverging into a different topic. In the limit $\beta\rightarrow 0$, each $G^{(d)}_{\textbf{{i}}_{l}}$ will be a delta function on a $\phi^{(d)}_{\textbf{{i}}_{l},j}\sim G_{\textbf{{i}}_{l}}$, and the same path will be selected by each word with probability one, thus recovering the nCRP.

2 Generating a document

With the tree $\mathcal{T}_{d}$ for document $d$ we have a method for selecting word-specific paths that are thematically coherent, meaning they tend to reuse the same path while allowing for off-shoots. We next discuss how to generate a document with this tree. As discussed in Section 2.2.2, with the nCRP the atoms selected for a document by its path through $\mathcal{T}$ have a unique stick-breaking distribution that determines which level any particular word comes from. We generalize this idea to the tree $\mathcal{T}_{d}$ with an overlapping stick-breaking construction as follows.

For each node $\textbf{{i}}_{l}$, we draw a document-specific beta random variable that acts as a stochastic switch. Given a pointer that is currently at node $\textbf{{i}}_{l}$, the beta random variable determines the probability that we draw from the topic at that node or continue further down the tree. That is, given that the path for word $W_{d,n}$ is at node $\textbf{{i}}_{l}$, stop with probability $U_{d,\textbf{{i}}_{l}}$, where

If we don’t select topic $\theta_{\textbf{{i}}_{l}}$, then continue by selecting node $\textbf{{i}}_{l+1}$ according to $G^{(d)}_{\textbf{{i}}_{l}}$. We observe the stick-breaking construction implied by this construction; for word $n$ in document $d$, the probability that its topic $\varphi_{d,n}=\theta_{\textbf{{i}}_{l}}$ is

Here it is implied that $\textbf{{i}}_{m}$ equals the first $m$ values in $\textbf{{i}}_{l}$ for $m\leq l$. The leftmost term in this expression is the probability of path $\textbf{{i}}_{l}$, the right term is the probability that the word does not select the first $l-1$ topics, but does select the $l$th. Since all random variables are independent, a simple product form results that will significantly aid the development of a posterior inference algorithm. The overlapping nature of this stick-breaking construction on the levels of a sequence is evident from the fact that the random variables $U$ are shared for the first $l$ values by all paths along the subtree starting at node $\textbf{{i}}_{l}$. A similar tree-structured prior distribution was presented by Adams, et al. in which all groups shared the same distribution on a tree and entire objects (e.g., images or documents) were clustered within a single node. We summarize our model for generating documents with the nHDP in Algorithm 1.

Stochastic variational inference for the Nested HDP

Many text corpora can be viewed as “Big Data”—they are large data sets for which standard inference algorithms can be prohibitively slow. For example, Wikipedia currently indexes several million entries and The New York Times has published almost two million articles in the last 20 years. With so much data, fast inference algorithms are essential. Stochastic variational inference is a development in this direction for hierarchical Bayesian models in which ideas from stochastic optimization are applied to approximate Bayesian inference using mean-field variational Bayes (VB) . Stochastic inference algorithms have provided a significant speed-up in inference for probabilistic topic models . In this section, after reviewing the ideas behind stochastic variational inference, we present a stochastic variational inference algorithm for the nHDP topic model.

Stochastic variational inference exploits the difference between local variables, or those associated with a single unit of data, and global variables, which are shared over an entire data set. In brief, stochastic VB works by splitting a large data set into smaller groups, processing the local variables of one group, updating the global variables, and then moving to another group. This is in contrast to batch inference, which processes all local variables at once before updating the global variables. In the context of probabilistic topic models, the unit of data is a document, and the global variables include the topics (among other possible variables), while the local variables relate to the distribution on these topics for each document. We next briefly review the relevant ideas from variational inference and its stochastic variant.

Mean-field variational inference is a method for approximate posterior inference in Bayesian models . It approximates the full posterior of a set of model parameters $P(\Phi|W)$ with a factorized distribution $Q(\Phi|\Psi)=\prod_{i}q_{i}(\phi_{i}|\psi_{i})$. It does this by searching the space of variational approximations for one that is close to the posterior according to their Kullback-Leibler divergence. Algorithmically, this is done by maximizing a variational objective function $\mathcal{L}$ with respect to the variational parameters $\Psi$ of $Q$, where

We are interested in conjugate exponential models, where the prior and likelihood of all nodes of the model fall within the conjugate exponential family. In this case, variational inference has a simple optimization procedure , which we illustrate with the following example—this generic example gives the general form exploited by the stochastic variational inference algorithm that we apply to the nHDP.

Consider $D$ independent samples from an exponential family distribution $P(W|\eta)$, where $\eta$ is the natural parameter vector. The likelihood under this model has the generic form

The sum of vectors $t(W_{d})$ forms the sufficient statistics of the likelihood. The conjugate prior on $\eta$ has a similar form

Conjugacy between these two distributions motivates selecting a $q$ distribution in this same family to approximate the posterior of $\eta$,

The variational parameters $\chi^{\prime}$ and $\nu^{\prime}$ are free and are modified to maximize the lower bound in Eq. (12).A closed form expression for the lower bound is readily derived for this example. Inference proceeds by taking the gradient of $\mathcal{L}$ with respect to the variational parameters of a particular $q$, in this case the vector $\psi:=[\chi^{\prime T},\nu^{\prime}]^{T}$, and setting to zero to find their updated values. For the conjugate exponential example we are considering, this gradient is

Setting this to zero, one can immediately read off the variational parameter updates from the rightmost vector. In this case $\chi^{\prime}=\chi+\sum_{d=1}^{D}t(W_{d})$ and $\nu^{\prime}=\nu+D$, which are the sufficient statistics calculated from the data.

1.2 A stochastic extension

Stochastic optimization of the variational lower bound modifies batch inference by forming a noisy gradient of $\mathcal{L}$ at each iteration. The variational parameters for a random subset of the data are optimized first, followed by a step in the direction of the noisy gradient of the global variational parameters. Let $C_{s}\subset\{1,\dots,D\}$ index a subset of the data at step $s$. Also let $\phi_{d}$ be the hidden local variables associated with observation $W_{d}$ and let $\Phi_{W}$ be the global variables shared among all observations. The stochastic variational objective function $\mathcal{L}_{s}$ is the noisy version of $\mathcal{L}$ formed by selecting a subset of the data,

Stochastic variational inference proceeds by optimizing the objective in (14) with respect to $\psi_{d}$ for $d\in C_{s}$, followed by an update to $\Psi_{W}$ that blends the new information with the old. The update of a global variational parameter $\psi$ at step $s$ is $\psi_{s}=\psi_{s-1}+\rho_{s}B\nabla_{\psi}\mathcal{L}_{s}(W_{C_{s}},\Psi)$, where the matrix $B$ is a positive definite preconditioning matrix and $\rho_{s}$ is a step size satisfying $\sum_{s=1}^{\infty}\rho_{s}=\infty$ and $\sum_{s=1}^{\infty}\rho_{s}^{2}<\infty$ to ensure convergence .

The gradient $\nabla_{\psi}\mathcal{L}_{s}(W_{C_{s}},\Psi)$ has a similar form as Eq. (13), with the exception that the sum is taken over a subset of the data. Though the matrix in Eq. (13) is often very complicated, it is superfluous to batch variational inference for conjugate exponential family models. In the stochastic optimization of Eq. (12), however, this matrix cannot be ignored. The key for conjugate exponential models is in selecting the preconditioning matrix $B$. Since the gradient of $\mathcal{L}_{s}$ has the same form as Eq. (13), $B$ can be set to the inverse of the matrix in (13) to allow for cancellation. An interesting observation is that this matrix is

which is the inverse Fisher information of the variational distribution $q(\eta|\psi)$. Using this setting for $B$, the step direction is the natural gradient of the lower bound, and therefore gives an efficient step direction in addition to simplifying the algorithm . The resulting variational update is a weighted combination of the old sufficient statistics for $q$ with the new ones calculated over data indexed by $C_{s}$.

2 The inference algorithm

We develop a stochastic variational inference algorithm for approximate posterior inference of the nHDP topic model. As discussed in our general review of stochastic inference, this entails optimizing the local variational parameters for a subset of documents, followed by a step along the natural gradient of the global variational parameters. We distinguish between local and global variables for the nHDP in Table II. In Table II we also give the variational $q$ distributions selected for each variable. In almost all cases we select this distribution to be in the same family as the prior. We point out two additional latent indicator variables for inference: $c_{d,n}$, which indicates the topic of word $W_{d,n}$, and $z_{\textbf{{i}},j}^{(d)}$, which points to the atom in $G_{\textbf{{i}}}$ associated with the $j$th break in $G^{(d)}_{\textbf{{i}}}$ using the construction given in Eq. (9).

Since we wish to consider large trees, and because there is slightly more overhead in calculating the distribution for each document than in models such as LDA and the HDP, the word allocation step is more time consuming for the nHDP. Additionally, we seek an efficient means for learning the indicators $z_{\textbf{{i}},j}^{(d)}$. Since each document will use a small subset of topics, which translates to a small subtree of the entire tree, our goal is to pick out a subtree in advance for the document to work with. This will reduce the number of topics to do inference over for each document, speeding up the algorithm, and determine the delta-function indicators for $z_{\textbf{{i}},j}^{(d)}$, which point to the “activated” nodes.

To this end, we introduce a third aspect to our inference algorithm in which we pick a small subtree for each document in advance. By this we mean that we only allow words in a document to be allocated to the subtree selected for that document and fix the probability that the indicator $c_{d,n}$ corresponds to topics outside this subtree to zero. As we will show, by selecting a subtree we are in effect learning a truncated stick-breaking construction of the tree for each document. If a node has two children in the subtree, then algorithmically we will have a two-node truncated construction for that DP of the specific document we are considering.

We select the subtree from $\mathcal{T}$ for each document using a greedy algorithm. This greedy algorithm is performed with respect to maximizing the variational objective function. Being an optimization method with one requirement (that we maximize a fixed objective), variational inference has considerable freedom in this regard. We discuss this greedy algorithm below, followed by the variational parameter updates for the local and global $q$ distributions. Algorithm 2 gives an outline.

As mentioned, we perform a greedy algorithm with respect to the variational objective function to determine a subtree from $\mathcal{T}$ for each document. We first describe the algorithm followed by a mathematical representation. Starting from the root node, we sequentially add nodes from $\mathcal{T}$, selecting from those currently “activated.” An activated node is one whose parent is contained within the subtree but which is not itself in the subtree.

To determine which node to add, we look at which node will give the greatest increase in the variational objective when the $q$ distributions for the document-specific beta distributions are fixed to their priors and the variational distribution for each word’s topic indicator $q$ distribution ($\nu_{d,n}$ in Table II) is zero on the remaining unactivated nodes. That is, we then ask the question: Which of the activated nodes not currently in the subtree will lead to the greatest increase in the variational objective under this restricted $q$ distribution?

The reason we consider this restricted distribution is that there is a closed form calculation for each node, and so no iterations are required in this step and the algorithm is much faster. Calculating this score only involves optimizing the variational parameter $\nu_{d,n}$ for each word over the current subtree plus the candidate node. We continue adding the maximizing node until the marginal increase in the objective falls below a threshold. We give a more formal description of this below.

As defined in Table II, $z^{(d)}_{\textbf{{i}},j}$ is the variable that indicates the index of the atom from the global DP $G_{\textbf{{i}}}$ pointed to by the $j$th stick-breaking weight in $G^{(d)}_{\textbf{{i}}}$. We select a delta $q$ distribution for this variable, meaning we make a hard assignment for this value. These values also define the subtree for document $d$. Starting with an empty tree, all atoms in $G_{\textbf{{i}}_{0}}$ constitute the activated set. Adding the first node is equivalent to determining the value for $z^{(d)}_{\textbf{{i}}_{0},1}$; in general, creating a subtree for $\mathcal{T}_{d}$, which we denote as $\mathcal{T}^{\prime}_{d}$, is equivalent to determining which $z^{(d)}_{\textbf{{i}},j}$ to include in $\mathcal{T}^{\prime}_{d}$ and the atoms to which they point.

For a subtree of size $t$ corresponding to document $d$, let the set $\mathcal{I}_{d,t}$ contain the index values of the included nodes, let $\mathcal{S}_{d,t}=\{\textbf{{i}}\,:\,pa(\textbf{{i}})\in\mathcal{I}_{d,t},\textbf{{i}}\not\in\mathcal{I}_{d,t}\}$ be the set of candidate nodes to add to $\mathcal{T}^{\prime}$. Then provided the marginal increase in the variational objective is above a preset threshold, we increment the subtree by letting $\mathcal{I}_{d,t+1}\leftarrow\mathcal{I}_{d,t}\cup\textbf{{i}}^{*}$, where

We let $\mathcal{C}_{d,t,\textbf{{i}}^{\prime}}$ denote the discussed conditions, that $\nu_{d,n}(\textbf{{i}})=0$ for all $\textbf{{i}}\not\in\mathcal{I}_{d,t}\cup\textbf{{i}}^{\prime}$ and that $q(\cdot)$ is fixed to the prior for all other distributions. The optimal values for $\nu_{d,n}$ are given below in Eq. (17).

We note two aspects of this greedy algorithm. First, though the stick-breaking construction of the document-level DP given in Eq. (9) allows for atoms to repeat, in this algorithm each additional atom is new, since there is no advantage in duplicating atoms. Therefore, the algorithm approximates each $G^{(d)}_{\textbf{{i}}}$ by selecting and reordering a subset of atoms from $G_{\textbf{{i}}}$ for its stick-breaking construction. (The subtree $\mathcal{T}^{\prime}_{d}$ may also contain zero atoms or one atom from a $G_{\textbf{{i}}}$.) The second aspect we point out is the changing prior on the same node in $\mathcal{T}$. If the atom $\theta_{(\textbf{{i}},m)}$ is a candidate for addition, then it remains a candidate until it is either selected by a $z^{(d)}_{\textbf{{i}},j}$, or the algorithm terminates. The prior on selecting this atom changes, however, depending on whether it is a candidate for $z^{(d)}_{\textbf{{i}},j}$ or $z^{(d)}_{\textbf{{i}},j^{\prime}}$. Therefore, incorporating a sibling of $\theta_{(\textbf{{i}},m)}$ impacts the prior on incorporating $\theta_{(\textbf{{i}},m)}$.

2.2 Coordinate updates for document variables

Given the subtree $\mathcal{T}^{\prime}_{d}$ selected for document $d$, we optimize the variational parameters for the $q$ distributions on $c_{d,n}$, $V^{(d)}_{\textbf{{i}},j}$ and $U_{d,\textbf{{i}}}$ over that subtree.

The variational distribution on the path for word $W_{d,n}$ is

where the prior term $\pi_{d,\textbf{{i}}}$ is the tree-structured prior of the nHDP,

We note that this has a familiar feel as LDA, but where LDA uses a flat Dirichlet prior on $\pi_{d}$, the nHDP uses a prior that is a tree-structured product of beta random variables. Though the form of the prior is more complicated, the independence results in simple closed-form updates for these beta variables that only depend on $\nu_{d,n}$.

The variational parameter updates for the document-level stick-breaking proportions are

In words, the statistic for the first parameter is the expected number of words in document $d$ that pass through or stop at node $(\textbf{{i}},j)$. The statistic for the second parameter is the expected number of words from document $d$ whose paths pass through the same parent i, but then transition to a node with index greater than $j$ according to the indicators $z^{(d)}_{\textbf{{i}},m}$ from the document-level stick-breaking construction of $G_{\textbf{{i}}}^{(d)}$.

The variational parameter updates for the switching probabilities are similar to those of the document-level stick-breaking process, but collect the statistics from $\nu_{d,n}$ in a slightly different way,

In words, the statistic for the first parameter is the expected number of words that use the topic at node i. The statistic for the second parameter is the expected number of words that pass through node i but do not terminate there.

2.3 Stochastic updates for corpus variables

After selecting the subtrees and updating the local document-specific variational parameters for each document $d$ in sub-batch $s$, we take a step in the direction of the natural gradient of the parameters of the $q$ distributions on the global variables. These include the topics $\theta_{\textbf{{i}}}$ and the global stick-breaking proportions $V_{\textbf{{i}}_{l},j}$.

For the stochastic update of the Dirichlet $q$ distributions on each topic $\theta_{\textbf{{i}}}$, first form the vector $\lambda^{\prime}_{\textbf{{i}}}$ of sufficient statistics using the data in sub-batch $s$,

for $w=1,\dots,\mathcal{V}.$ This vector contains the expected number of words with index $w$ that originate from topic $\theta_{\textbf{{i}}}$ over documents indexed by $C_{s}$. According to the discussion on stochastic inference in Section 4.1.2, we scale this to a corpus of size $D$. The update for the associated $q$ distribution is

We see a blending of the old statistics with the new in this update. Since $\rho_{s}\rightarrow 0$ as $s$ increases, the algorithm uses less and less information from new sub-groups of documents, which reflects the increasing confidence in this parameter value as more data is seen.

As with $\theta_{\textbf{{i}}}$, we first collect the sufficient statistics for the $q$ distribution on $V_{\textbf{{i}}_{l},j}$ from the documents in sub-batch $s$,

The first value scales up the number of documents in sub-batch $s$ that include atom $\theta_{(\textbf{{i}},j)}$ in their subtree; the second value scales up the number of times an atom of higher index value in the same DP is used by a document in sub-batch $s$. The update to the global variational parameters are

Again, we see a blending of old information with new.

Experiments

We present an empirical evaluation of the nested HDP topic model in the stochastic and the batch inference settings. We first present batch results on three smaller data sets to verify that our multi-path approach gives an improvement over the single-path nested CRP. We then move to the stochastic inference setting, where we perform experiments on 1.8 million documents from The New York Times and 2.7 million documents from Wikipedia. We compare with other recent stochastic inference algorithms for topic models: stochastic LDA and the stochastic HDP . As is fairly standard with the optimization-based variational inference, we use truncated stick-breaking processes for all DPs . With this method, we truncate the posterior approximation by not allowing words to come from topics beyond the truncation index (i.e., fixing $c_{d,n}((\textbf{{i}},j))=0$ for all $j>n$). The truncation is set to something reasonably large, and the posterior inference procedure then shrinks the number of used topics to something smaller than the number provided. In our large-scale experiments, we truncate to $n_{1}=20$ first level nodes, $n_{2}=10$ children for each of these nodes and $n_{3}=5$ children of each of these second level nodes. We consider three level trees, corresponding intuitively to “general”, “specific” and “specialized” levels of words. Though the nHDP is nonparametric in level as well, we are more interested in the nonparametric aspect of the Dirichlet process here.

Before presenting our results, we discuss our method for initializing the topic distributions of the tree. As with most Bayesian models, inference for hierarchical topic models can benefit greatly from a good initialization. Our goal is to find a method for quickly centering the posterior mean of each topic so that they contain some information about their hierarchical relationships. We briefly discuss our approach for initializing the global variational topic parameters $\lambda_{\textbf{{i}}}$ of the nHDP.

Using a small set of documents (e.g, 10,000) from the training set, we form the empirical distribution for each document on the vocabulary. We then perform k-means clustering of these probability vectors using the $L_{1}$ distance measure (i.e., total variation). At the top level, we partition the data into $n_{1}$ groups, corresponding to $n_{1}$ children of the root node from the truncated stick-breaking process. We then subtract the mean of a group (a probability vector) from all data within that group, set any negative values to zero and renormalize. We loosely think of this as the “probability of what remains”—a distribution on words not captured by the parent distributions. Within each group we again perform k-means clustering, obtaining $n_{2}$ probability vectors for each of the $n_{1}$ groups, and again subtracting, setting negative values to zero and renormalizing the remainder of each probability vector for a document.

Through this hierarchical k-means clustering, we obtain $n_{1}$ probability vectors at the top level, $n_{2}$ probability vectors beneath each top-level vector for the second level, $n_{3}$ probability vectors beneath each of these second-level vectors, etc. The $n_{i}$ vectors obtained from any sub-group of data are refinements of an already coherent sub-group of data, since that sub-group is itself a cluster from a larger group. Therefore, the resulting tree will have some thematic coherence. The clusters from this algorithm are used to initialize the nodes within the nHDP tree. For a mean probability vector $\hat{\lambda}_{\textbf{{i}}}$ obtained from this algorithm, we set the corresponding variational parameter for the topic Dirichlet distribution $q$ to $\lambda_{\textbf{{i}}}=N(\kappa\hat{\lambda}_{\textbf{{i}}}+(1-\kappa)(\boldsymbol{1}/\mathcal{V}+v_{\textbf{{i}}}))$ for $\kappa\in$, $N$ a scaling factor and $v_{\textbf{{i}}}\stackrel{{\scriptstyle iid}}{{\sim}}\mbox{Dirichlet}(100\boldsymbol{1}_{\mathcal{V}}/\mathcal{V})$. This initializes the mean of $\theta_{\textbf{{i}}}$ to be slightly peaked around $\hat{\lambda}_{\textbf{{i}}}$, while the uniform vector and $\kappa$ help determine the variance and $v_{\textbf{{i}}}$ provides some randomness. In our algorithms we set $\kappa=0.5$ and $N$ equal to the number of documents.

2 A batch comparison

Before comparing our stochastic inference algorithm for the nHDP with similar algorithms for LDA and the HDP, we compare a batch version with the nCRP on three smaller data sets. This will verify the advantage of giving each document access to the entire tree versus forcing each document to follow one path. We compare the variational nHDP topic model with both the variational nCRP and the Gibbs sampling nCRP , using the parameter settings in those papers to facilitate comparison. We consider three corpora for our experiments: ($i$) The Journal of the ACM, a collection of 536 abstracts from the years 1987–2004 with vocabulary size 1,539; ($ii$) The Psychological Review, a collection of 1,272 abstracts from the years 1967–2003 with vocabulary size 1,971; and ($iii$) The Proceedings of the National Academy of Science, a collection of 5,000 abstracts from the years 1991–2001 with a vocabulary size of 7,762. The average number of words per document for the three corpora are 45, 108 and 179, respectively.

As mentioned, variational inference for Dirichlet process priors uses a truncation of the variational distribution, which limits the number of topics that are learned . This truncation is set to a number larger than the anticipated number of topics necessary for modeling the data set, but can be increased if more are needed . We use a truncated tree of $(10,7,5)$ for modeling these corpora, where 10 children of the root node each have 7 children, which themselves each have 5 children for a total of 420 nodes. Because these three data sets contain stop words, we follow and by including a root node shared by all documents for this batch problem only. Following , we perform five-fold cross validation to evaluate performance on each corpus.

We present our results in Table II, where we show the predictive log likelihood on a held-out test set. We see that for all data sets, the variational nHDP outperforms the variational nCRP. For the two larger data sets, the variational nHDP also outperforms Gibbs sampling for the nCRP. Given the relative sizes of these corpora, we see that the benefit of learning a per-document distribution on the full tree rather than a shared distribution on paths appears to increase as the corpus size and document size increase. Since we are interested in the “Big Data” regime, this strongly hints at an advantage of our nHDP approach over the nCRP. We omit a comparison with Gibbs nHDP since MCMC methods are not amenable to large data sets for this problem.

3 Stochastic inference for large corpora

We next present an evaluation of our stochastic variational inference algorithm on The New York Times and Wikipedia. These are both very large data sets, with The New York Times containing roughly 1.8 million articles and Wikipedia roughly 2.7 million web pages. The average document size is somewhat larger than those considered in our batch experiments as well, with an article from The New York Times containing 254 words on average taken from a vocabulary size of 8,000, and Wikipedia 164 words on average taken from a vocabulary size of 7,702. For this problem we remove stop words and rare words.

We use the algorithm discussed in Section 5.1 to initialize a three-level tree with $(20,10,5)$ child nodes per level, giving a total of 1,220 initial topics. For the Dirichlet processes, we set all top-level DP concentration parameters to $\alpha=5$ and the second-level DP concentration parameters to $\beta=1$. For the switching probabilities $U$, we set the beta distribution hyperparameters for the tree level prior to $\gamma_{1}=1/3$ and $\gamma_{2}=2/3$, slightly encouraging a word to continue down the tree. We set the base Dirichlet parameter $\lambda_{0}=0.1$. For our greedy subtree selection algorithm, we stop adding nodes to the subtree when the marginal improvement to the lower bound falls below $10^{-3}$. When optimizing the local variational parameters of a document given its subtree, we continue iterating until the fractional change in the $L_{1}$ distance of the empirical distribution of words falls below $10^{-2}$.

We hold out a data set for each corpus for testing, 14,268 documents for testing The New York Times and 8,704 documents for testing Wikipedia. To quantitatively assess the performance, at various points in the learning process we calculate the predictive log likelihood on a fraction of the test set as follows: Holding the top-level variational parameters fixed, for each test document we randomly partition the words into a 90/10 percent split. We then learn document-specific variational parameters for the 90% portion. Following , we use the mean of each $q$ distribution to form a predictive distribution for the remaining words of that document. With this distribution, we calculate the average predictive log likelihood of the 10% portion to assess performance. For comparison, we evaluate stochastic inference algorithms for LDA and the HDP in the same manner. In all algorithms, we use an algorithm for adaptively learning the step size $\rho_{s}$ as presented by Ranganath, et al. .

3.2 The New York Times

We first present our results for The New York Times. In Figure 2 we show the average predictive log likelihood on unseen words as a function of the number of documents processed during model learning. We see an improvement in performance as the amount of data processed increases. We also note an improvement in the performance of the nHDP compared with LDA and the HDP. In Figure 5 we give a sense of the size of the tree as a function of documents seen. Since all topics aren’t used equally, we show the minimum number of nodes containing $95\%$, $99\%$ and $99.9\%$ of all data in the posterior. In Figure 5 we show document-level statistics from the test set at the final step of the algorithm. These include the word allocations by level and the number of topics used per level. We note that while the tree has three levels, roughly 12 topics are being used (in varying degrees) per document. This is in contrast to the three topics that would be available to any document with the nCRP. Thus there is a clear advantage in allowing each document to have access to the entire tree. We show the adaptively learned step size in Figure 5.

In Figure 6 we show example topics from the model and their relative structure. For each node we show the most probable words according to the approximate posterior $q$ distribution of the topic. We show four topics from the top level of the tree (shaded), and connect topics according to parent/child relationship. The model learns a meaningful hierarchical structure; for example, the sports subtree branches into the various sports, which themselves appear to branch by teams. In the foreign affairs subtree, children tend to group by major subregion and then branch out into subregion or issue. If a sports document incorporated topics on foreign affairs, the nHDP would allow words to split into both parts of the tree, but with the nCRP a document would have to pick one or the other, and so a tree could not be learned that distinguished topics with this level of precision.

The algorithm took roughly 20 hours to make one pass through the data set using a single desktop computer, which was sufficient for the model to converge to a set of topics. Runtime for Wikipedia was comparable.

3.3 Wikipedia

We show similar results for Wikipedia as for The New York Times. In Figures 7, 10, 10 and 10 we show results corresponding to Figures 2, 5, 5 and 5, respectively for The New York Times. We again see an improvement in performance for the nHDP over LDA and the HDP, as well as the increased usage of the tree with the nHDP than would be available in the nCRP.

In Figure 11, we see example subtrees used by three documents. We note that the topics contain many more function words than for The New York Times, but an underlying hierarchical structure is uncovered that would be unlikely to arise along one path, as the nCRP would require. As with The New York Times, we see the nonparametric nature of the model in Figure 10. Though the model has an 1,220 initial nodes, a small subset are ultimately used by the data.

3.4 Sensitivity analysis

We present a brief sensitivity analysis of some parameters of the nHDP topic model using the Wikipedia corpus. In general, we find that the results were not sensitive to the parameter $\lambda_{0}$ of the base Dirichlet distribution, which is consistent with . We note that this is typically not the case for topic models, but because of the massive quantity of data we are working with, the data overwhelms the prior in this case. This was similarly found with the global DP parameter $\alpha$.

The document-specific variables have a more significant impact since they only use the data from a single document in their posteriors. In Figures 12–14 we show the sensitivity of the model to the parameters $\beta$ and $(\gamma_{1},\gamma_{2})$. We consider several values for these parameters, holding $\gamma_{1}+\gamma_{2}=1$. As can be seen, the model structure is fairly robust to these values. The tree structure does respond as would be expected from the prior, but there is no major change. The quantitative results in Figure 14 indicate that the quality of the model is robust as well. We note that this relative insensitivity is within a parameter range that we believe a priori to be reasonable.

Conclusion

We have presented the nested hierarchical Dirichlet process (nHDP), an extension of the nested Chinese restaurant process (nCRP) that allows each observation to follow its own path to a topic in the tree. Starting with a stick-breaking construction for the nCRP, the new model samples document-specific path distributions for a shared tree using a nested hierarchy of Dirichlet processes. By giving a document access to the entire tree, we are able to borrow thematic content from various parts of the tree in constructing a document. We developed a stochastic variational inference algorithm that is scalable to very large data sets. We compared the stochastic nHDP topic model with stochastic LDA and HDP and showed how the nHDP can learn meaningful topic hierarchies.

References