Bibliographic Analysis on Research Publications using Authors, Categorical Labels and the Citation Network

Introduction

Models of bibliographic data need to consider many kinds of information. Articles are usually accompanied by metadata such as authors, publication data, categories and time. Cited papers can also be available. When authors’ topic preferences are modelled, we need to associate the document topic information somehow with the authors’. Jointly modelling text data with citation network information can be challenging for topic models, and the problem is confounded when also modelling author-topic relationships.

In this paper, we propose a topic model to jointly model authors’ topic preferences, text contentAbstract and publication title. and the citation network. The model is a nonparametric extension of previous models discussed in Section 2. Using simple assumptions and approximations, we derive a novel algorithm that allows the probability vectors in the model to be integrated out. This yields a Markov chain Monte Carlo (MCMC) inference via discrete sampling.

As an extension of our previous work (Lim and Buntine, 2014), we propose a supervised approach to improve document clustering, by making use of categorical information that is available. Our method allows the level of supervision to be adjusted through a variable, giving us a model with no supervision, semi-supervised or fully supervised. Additionally, we present a more extensive qualitative analysis of the learned topic models, and display a visualisation snapshot of the learned author-topics network. We also perform additional diagnostic tests to assess our proposed topic model. For example, we study the convergence of the proposed learning algorithm and report on the computation complexity of the algorithm.

In the next section, we discuss the related work. Section 3, 4 and 5 detail our topic model and its inference algorithm. We describe the datasets in Section 6 and report on experiments in LABEL:sec:experiment. Applying our model on research publication data, we demonstrate the model’s improved performance, on both model fitting and a clustering task, compared to several baselines. Additionally, in LABEL:sec:qualitative_analysis, we qualitatively analyse the inference results produced by our model. We find that the learned topics have high comprehensibility. Additionally, we present a visualisation snapshot of the learned topic models. Finally, we perform diagnostic assessment of the topic model in LABEL:sec:diagnostic and conclude the paper in LABEL:sec:conclusion.

Related Work

Latent Dirichlet Allocation (LDA) (Blei et al., 2003) is the simplest Bayesian topic model used in modelling text, which also allows easy learning of the model. Teh and Jordan (2010) proposed the Hierarchical Dirichlet process (HDP) LDA, which utilises the Dirichlet process (DP) as a nonparametric prior which allows a non-symmetric, arbitrary dimensional topic prior to be used. Furthermore, one can replace the Dirichlet prior on the word vectors with the Pitman-Yor Process (PYP, also known as the two-parameter Poisson Dirichlet process) (Teh, 2006b), which models the power-law of word frequency distributions in natural language (Goldwater et al., 2011), yielding significant improvement (Sato and Nakagawa, 2010).

Variants of LDA allow incorporating more aspects of a particular task and here we consider authorship and citation information. The author-topic model (ATM) (Rosen-Zvi et al., 2004) uses the authorship information to restrict topic options based on author. Some recent work jointly models the document citation network and text content. This includes the relational topic model (Chang and Blei, 2010), the Poisson mixed-topic link model (PMTLM) (Zhu et al., 2013) and Link-PLSA-LDA (Nallapati et al., 2008). An extensive review of these models can be found in Zhu et al. (2013). The Citation Author Topic (CAT) model (Tu et al., 2010) models the author-author network on publications based on citations using an extension of the ATM. Note that our work is different to CAT in that we model the author-document-citation network instead of author-author network.

The Topic-Link LDA (Liu et al., 2009) jointly models author and text by using the distance between the document and author topic vectors. Similarly the Twitter-Network topic model (Lim et al., 2013) models the author networkThe author network here corresponds to the Twitter follower network. based on author topic distributions, but using a Gaussian process to model the network. Note that our work considers the author-document-citation of Liu et al. (2009). We use the PMTLM of Zhu et al. (2013) to model the network, which lets one integrate PYP hierarchies with the PMTLM using efficient MCMC sampling.

There is also existing work on analysing the degree of authors’ influence. On publication data, Kataria et al. (2011) and Mimno and McCallum (2007) analyse influential authors with topic models, while Weng et al. (2010), Tang et al. (2009), and Liu et al. (2010) use topic models to analyse users’ influence on social media.

Supervised Citation Network Topic Model

In our previous work (Lim and Buntine, 2014), we proposed the Citation Network Topic Model (CNTM) that jointly models the text, authors, and the citation network of research publications (documents). The CNTM allows us to both model the authors and text better by exploiting the correlation between the authors and their research topics. However, the benefit of the above modelling is not realised when the author information is simply missing from the data. This could be due to error in data collection (e.g. metadata not properly formatted), or even simply that the author information is lost during preprocessing.

In this section, we propose an extension of the CNTM that remedies the above issue, by making use of additional metadata that is available. For example, the metadata could be the research areas or keywords associated with the publications, which are usually provided by the authors during the publication submission. However, this information might not always be reliable as it is not standardised across different publishers or conferences. In this paper, rather than using the mentioned metadata, we will instead incorporate the categorical labels that were previously used as ground truth for evaluation. As such, our extension gives rise to a supervised model, which we will call the Supervised Citation Network Topic Model (SCNTM).

We first describe the topic model part of SCNTM for which the citations are not considered, it will be used for comparison later in LABEL:sec:experiment. We then complete the SCNTM with the discussion on its network component. The full graphical model for SCNTM is displayed in Figure 1.

The SCNTM uses both the Griffiths-Engen-McCloskey (GEM) distribution (Pitman, 1996) and the Pitman-Yor process (PYP) (Teh, 2006b) to generate probability vectors. Both the GEM distribution and the PYP are parameterised by a discount parameter $\alpha$ and a concentration parameter $\beta$. The PYP is additionally parameterised by a base distribution $H$, which is also the mean of the PYP when it can be represented by a probability vector. Note that the base distribution can also be a PYP. This gives rise to the hierarchical Pitman-Yor process (HPYP).

In modelling authorship, the SCNTM modifies the approach of the author-topic model (Rosen-Zvi et al., 2004) which assumes that the words in a publication are equally attributed to the different authors. This is not reflected in practice since publications are often written more by the first author, excepting when the order is alphabetical. Thus, we assume that the first author is dominant and attribute all the words in a publication to the first author. Although, we could model the contribution of each author on a publication by, say, using a Dirichlet distribution, we found that considering only the first author gives a simpler learning algorithm and cleaner results.

The generative process of the topic model component of the SCNTM is as follows. We first sample a root topic distribution $\mu$ with a GEM distribution to act as a base distribution for the author-topic distributions $\nu_{a}$ for each author $a$, and also for the category-topic distributions $\nu_{e}$ for each category $e$:

Here, $\mathcal{A}$ represents the set of all authors while $\mathcal{E}$ denotes the set of all categorical labels in the text corpus. Note we have used the same symbol ($\nu$) for both the author-topic distributions and the category-topic distributions.

We introduce a parameter $\eta$ called the author threshold which controls the level of supervision used by SCNTM. We say an author $a$ is significant if the author has produced more than or equal to $\eta$ publications, i.e.

Here, $a_{d}$ represents the author for document $d$, and $I(\triangle)$ is the indicator function that evaluates to $1$ if $\triangle$ is true, else .

Next, for each document $d$ in a publication collection of size $D$, we sample the document-topic prior $\theta^{\prime}_{d}$ from $\nu_{a_{d}}$ or $\nu_{e_{d}}$ depending on whether the author $a_{d}$ for the document is significant:

where $e_{d}$ is the categorical label associated with document $d$. For the sake of notational simplicity, we introduce a variable $b$ to capture both the author and the category. We let $b$ takes the value of $1,\dots,A$ for each author in $\mathcal{A}$, and let $b$ takes the value of $(A+1),\dots,B$ for the categories in $\mathcal{E}$. Note that $B=|\mathcal{A}|+|\mathcal{E}|$. Thus, we can also write the distribution of $\theta^{\prime}_{d}$ as

By modelling this way, we are able to handle missing authors and incorporate supervision into the SCNTM. For example, choosing $\eta=1$ allows us to make use of the categorical information for documents that have no valid author. Alternatively, we could select a higher $\eta$, this smooths out the document-topic distributions for documents that are written by authors who have authored only a small number of publications. This treatment leads to a better clustering result as these authors are usually not discriminative enough for prediction. On the extreme, we can set $\eta=\infty$ to achieve full supervision. We note that the SCNTM reverts to the CNTM when $\eta=0$, in this case the model is not supervised.

We then sample the document-topic distribution $\theta_{d}$ given $\theta^{\prime}_{d}$:

Note that instead of modelling a single document-topic distribution, we model a document-topic hierarchy with $\theta^{\prime}$ and $\theta$. The primed $\theta^{\prime}$ represents the topics of the document in the context of the citation network. The unprimed $\theta$ represents the topics of the text, naturally related to $\theta^{\prime}$ but not the same. Such modelling gives citation information a higher impact to take into account the relatively low amount of citations compared to the text. The technical details on the effect of such modelling is presented in LABEL:subsec:modelling_topic_hierarchy.

For the vocabulary side, we generate a background word distribution $\gamma$ given $H^{\gamma}$, a discrete uniform vector of length $|\mathcal{V}|$, i.e. $H^{\gamma}=(\cdots,\frac{1}{|\mathcal{V}|},\cdots)$. $\mathcal{V}$ is the set of distinct word tokens observed in a corpus. Then, we sample a topic-word distribution $\phi_{k}$ for each topic $k$, with $\gamma$ as the base distribution:

Modelling word burstiness (Buntine and Mishra, 2014) is important since words in a document are likely to repeat in the document. The same applies to publication abstract, as shown in Section 6. To address this property, we make the topics bursty so each document only focuses on a subset of words in the topic. This is achieved by defining the document-specific topic-word distribution $\phi^{\prime}_{dk}$ for each topic $k$ in document $d$ as:

Finally, for each word $w_{dn}$ in document $d$, we sample the corresponding topic assignment $z_{dn}$ from the document-topic distribution $\theta_{d}$; while the word $w_{dn}$ is sampled from the topic-word distribution $\phi^{\prime}_{d}$ given $z_{dn}$:

Note that $w$ includes words from the publications’ title and abstract, but not the full article. This is because title and abstract provide a good summary of a publication’s topics and thus more suited for topic modelling, while the full article contains too much technical detail that might not be too relevant.

In the next section, we describe the modelling of the citation network accompanying a publication collections. This completes the SCNTM.

2 Citation Network Poisson Model

To model the citation network between publications, we assume that the citations are generated conditioned on the topic distributions $\theta^{\prime}$ of the publications. Our approach is motivated by the degree-corrected variant of PMTLM (Zhu et al., 2013). Denoting $x_{ij}$ as the number of times document $i$ citing document $j$, we model $x_{ij}$ with a Poisson distribution with mean parameter $\lambda_{ij}$:

Here, $\lambda_{i}^{+}$ is the propensity of document $i$ to cite and $\lambda_{j}^{-}$ represents the popularity of cited document $j$, while $\lambda^{T}_{k}$ scales the $k$-th topic, effectively penalising common topics and strengthen rare topics. Hence, a citation from document $i$ to document $j$ is more likely when these documents are having relevant topics. Due to the limitation of the data, the $x_{ij}$ can only be or $1$, i.e. it is a Boolean variable. Nevertheless, the Poisson distribution is used instead of a Bernoulli distribution because it leads to dramatically reduced complexity in analysis (Zhu et al., 2013). Note that the Poisson distribution is similar to the Bernoulli distribution when the mean parameter is small. We present a list of variables associated with the SCNTM in Section 3.2.

Before presenting the posterior used to develop the MCMC sampler, we briefly review handling of the hierarchical PYP models in Section 4.1. We cannot provide an adequately detailed review in this paper, thus we present the main ideas.

The key to efficient sampling with PYPs is to marginalise out the probability vectors (e.g. topic distributions) in the model and record various associated counts instead, thus yielding a collapsed sampler. While a common approach here is to use the hierarchical Chinese Restaurant Process (CRP) of Teh and Jordan (2010), we use another representation that requires no dynamic memory and has better inference efficiency (Chen et al., 2011).

We denote $f^{*}(\mathcal{N})$ as the marginalised likelihood associated with the probability vector $\mathcal{N}$. Since the vector is marginalised out, the marginalised likelihood is in terms of — using the CRP terminology — the customer counts $c^{\mathcal{N}}=(\cdots,c_{k}^{\mathcal{N}},\cdots)$ and the table counts $t^{\mathcal{N}}=(\cdots,t_{k}^{\mathcal{N}},\cdots)$. The customer count $c_{k}^{\mathcal{N}}$ corresponds to the number of data points (e.g. words) assigned to group $k$ (e.g. topic) for variable $\mathcal{N}$. Here, the table counts $t^{\mathcal{N}}$ represent the subset of $c^{\mathcal{N}}$ that gets passed up the hierarchy (as customers for the parent probability vector of $\mathcal{N}$). Thus $t_{k}^{\mathcal{N}}\leq c_{k}^{\mathcal{N}}$, and $t_{k}^{\mathcal{N}}=0$ if and only if $c_{k}^{\mathcal{N}}=0$ since the counts are non-negative. We also denote $C^{\mathcal{N}}=\sum_{k}c_{k}^{\mathcal{N}}$ as the total customer counts for node $\mathcal{N}$, and similarly, ${T}^{\mathcal{N}}=\sum_{k}t_{k}^{\mathcal{N}}$ is the total table counts. The marginalised likelihood $f^{*}(\mathcal{N})$, in terms of $c^{\mathcal{N}}$ and $t^{\mathcal{N}}$, is given as

$S^{x}_{y,\alpha}$ is the generalised Stirling number that is easily tabulated; both $(x)_{C}$ and $(x|y)_{C}$ denote the Pochhammer symbol (rising factorial), see Buntine and Hutter (2012) for details. Note the GEM distribution behaves like a PYP in which the table count $t_{k}^{\mathcal{N}}$ is always $1$ for non-zero $c_{k}^{\mathcal{N}}$.

The innovation of Chen et al. (2011) was to notice that sampling with Equation 18 directly led to poor performance. The problem was that sampling an assignment to a latent variable, say moving a customer from group $k$ to $k^{\prime}$ (so $c_{k}^{\mathcal{N}}$ decreases by $1$ and $c_{k^{\prime}}^{\mathcal{N}}$ increases by $1$), the potential effect on $t_{k}^{\mathcal{N}}$ and $t_{k^{\prime}}^{\mathcal{N}}$ could not immediately be measured. Whereas, the hierarchical CRP automatically included table configurations in its sampling process and thus included the influence of the hierarchy in the sampling. Thus sampling directly with Equation 18 lead to comparatively poor mixing. As a solution, Chen et al. (2011) develop a collapsed version of the hierarchical CRP following the well known practice of Rao-Blackwellisation of sampling schemes (Casella and Robert, 1996), which, while not being as fast per step, it has two distinct advantages, (1) it requires no dynamic memory and (2) the sampling has significantly lower variance so converges much faster. This has empirically been shown to lead to better mixing of the samplers (Chen et al., 2011) and has been confirmed on different complex topic models (Buntine and Mishra, 2014).

The technique for collapsing the hierarchical CRP uses Equation 18 but the counts ($c^{\mathcal{N}},t^{\mathcal{N}}$) are now derived variables. They are derived from Boolean variables associated with each data point. The technique comprises the following conceptual steps: (1) add Boolean indicators $u_{dn}$ to the data $(z_{dn},w_{dn})$ from which the counts $c^{\mathcal{N}}$ and $t^{\mathcal{N}}$ can be derived, (2) modify the marginalised posterior accordingly, and (3) derive a sampler for the model.

We first consider $c_{k}^{\theta_{d}}$, which has a “+1” contributed to for every $z_{dn}=k$ in document $d$, hence $c_{k}^{\theta_{d}}=\sum_{n}I(z_{dn}=k)$. We now introduce a new Bernoulli indicator variable $u^{\theta_{d}}_{dn}$ associated with $z_{dn}$, which is “on” (or $1$) when the data $z_{dn}$ also contributed a “+1” to $t^{\theta_{d}}_{k}$. Note that $t_{k}^{\theta_{d}}\leq c_{k}^{\theta_{d}}$, so every data contributing a “+1” to $c_{k}^{\theta_{d}}$ may or may not contribute a “+1” to $t_{k}^{\theta_{d}}$. The result is that one derives $t_{k}^{\theta_{d}}=\sum_{n}I(z_{dn}=k)\,I(u^{\theta_{d}}_{dn}=1)$.

Now consider the parent of $\theta_{d}$, which is $\theta^{\prime}_{d}$. Its customer count is derived as $c_{k}^{\theta^{\prime}_{d}}=t_{k}^{\theta_{d}}$. Its table count $t_{k}^{\theta^{\prime}_{d}}$ can now be treated similarly. Those data $z_{dn}$ that contribute a “+1” to $t_{k}^{\theta_{d}}$ (and thus $c_{k}^{\theta^{\prime}_{d}}$) have a new Bernoulli indicator variable $u^{\theta^{\prime}_{d}}_{dn}$, which is used to derive $t_{k}^{\theta^{\prime}_{d}}=\sum_{n}I(z_{dn}=k)\,I(u^{\theta^{\prime}_{d}}_{dn}=1)$, similar as before. Note that if $u^{\theta^{\prime}_{d}}_{dn}=1$ then necessarily $u^{\theta_{d}}_{dn}=1$.

Similarly, one can define Boolean indicators for $\mu$, $\nu_{b}$, $\phi^{\prime}$, $\phi$, and $\gamma$ to have a full suite from which all the counts $c^{\mathcal{N}}$ and $t^{\mathcal{N}}$ are now derived. We denote $u_{dn}=\{u^{\theta_{d}}_{dn},u^{\theta^{\prime}_{d}}_{dn},u^{\nu_{b}}_{dn},u^{\mu}_{dn},u^{\phi^{\prime}_{d}}_{dn}$, $u^{\phi_{d}}_{dn},u^{\gamma}_{dn}\}$ as the collection of the Boolean indicators for data ($z_{dn}$, $w_{dn}$).

1.2 Probability of Boolean indicators

By symmetry, if there are $t_{k}^{\mathcal{N}}$ Boolean indicators “on” (out of $c_{k}^{\mathcal{N}}$), we are indifferent as to which is on. Thus the indicator variable $u^{\mathcal{N}}_{dn}$ is not stored, that is, we simply “forget” who contributed a table count and re-sample $u^{\mathcal{N}}_{dn}$ as needed:

Moreover, this means that the marginalised likelihood $f^{*}(\mathcal{N})$ of Equation 18 is extended to include the probability of $u^{\mathcal{N}}$, which is written in terms of $c^{\mathcal{N}}$, $t^{\mathcal{N}}$ and $u^{\mathcal{N}}$ as:

2 Likelihood for the Hierarchical PYP Topic Model

We use bold face capital letters to denote the set of all relevant lower case variables. For example, $\mathbf{Z}=\{z_{11},\cdots,z_{DN_{D}}\}$ denotes the set of all topic assignments. Variables $\mathbf{W}$, $\mathbf{T}$, $\mathbf{C}$ and $\mathbf{U}$ are similarly defined, that is, they denote the set of all words, table counts, customer counts, and Boolean indicators respectively. Additionally, we denote $\mathbf{\zeta}$ as the set of all hyperparameters (such as the $\alpha$’s). With the probability vectors replaced by the counts, the likelihood of the topic model can be written — in terms of $f(\cdot)$ as given in Equation 20 — as $p(\mathbf{Z},\mathbf{W},\mathbf{T},\mathbf{C},\mathbf{U}\,|\,\mathbf{\zeta})\propto$

Note that the last term in Equation 21 corresponds to the parent probability vector of $\gamma$ (see Section 3.1), and $v$ indexes the unique word tokens in vocabulary set $\mathcal{V}$. Note that the extra terms for $\mathbf{U}$ are simply derived using Equation 20 and not stored in the model. So in the discussions below we will usually represent $\mathbf{U}$ implicitly by $\mathbf{T}$ and $\mathbf{C}$, and introduce the $\mathbf{U}$ when explicitly needed.

Note that even though the probability vectors are integrated out and not explicitly stored, they can easily be estimated from the associated counts. The probability vector $\mathcal{N}$ can be estimated from its posterior mean given the counts and parent probability vector $\mathcal{P}$:

3 Likelihood for the Citation Network Poisson Model

For the citation network, the Poisson likelihood for each $x_{ij}$ is given as

Note that the term $x_{ij}!$ is dropped in Equation 23 due to the limitation of the data that $x_{ij}\in\{0,1\}$, thus $x_{ij}!$ is evaluated to $1$. With conditional independence of $x_{ij}$, the joint likelihood for the whole citation network $\mathbf{X}=\{x_{11},\cdots,x_{DD}\}$ can be written as $p(\mathbf{X}\,|\,\lambda,\theta^{\prime})=$

where $g^{+}_{i}$ is the number of citations for publication $i$, $g^{+}_{i}=\sum_{j}x_{ij}$, and $g^{-}_{i}$ is the number of times publication $i$ being cited, $g^{-}_{i}=\sum_{j}x_{ji}$. We also make a simplifying assumption that $x_{ii}=1$ for all documents $i$, that is, all publications are treated as self-cited. This assumption is important since defining $x_{ii}$ allows us to rewrite the joint likelihood into Equation 24, which leads to a cleaner learning algorithm that utilises an efficient caching. Note that if we do not define $x_{ii}$, we have to explicitly consider the case when $i=j$ in Equation 24 which results in messier summation and products.

Note the likelihood in Equation 24 contains the document-topic distribution $\theta^{\prime}$ in vector form. This is problematic as performing inference with the likelihood requires the probability vectors $\theta^{\prime}$, $\nu$ and $\mu$ to be stored explicitly (instead of counts as discussed in Section 4.1). To overcome this issue, we propose a novel representation that allows the probability vectors to remain integrated out. Such representation also leads to an efficient sampling algorithm for the citation network, as we will see in Section 5.

We introduce an auxiliary variable $y_{ij}$, named the citing topic, to denote the topic that prompts publication $i$ to cite publication $j$. To illustrate, for a biology publication that cites a machine learning publication for the learning technique, the citing topic would be ‘machine learning’ instead of ‘biology’. From Equation 17, we model the citing topic $y_{ij}$ as jointly Poisson with $x_{ij}$:

Incorporating $\mathbf{Y}$, the set of all $y_{ij}$, we rewrite the citation network likelihood as

where $h_{ik}=\sum_{j}x_{ij}I(y_{ij}=k)+\sum_{j}x_{ji}I(y_{ji}=k)$ is the number of connections publication $i$ made due to topic $k$.

where $g_{k}^{T}=\frac{1}{2}\sum_{i}h_{ik}$ . We note that $p(\mathbf{Z},\mathbf{W},\mathbf{T},\mathbf{C^{+}},\mathbf{U}\,|\,\mathbf{\zeta})$ is the same as Equation 21 but now with $\mathbf{C^{+}}$ instead of $\mathbf{C}$.

In the next section, we demonstrate that our model representation gives rise to an intuitive sampling algorithm for learning the model. We also show how the Poisson model integrates into the topic modelling framework.

Here, we derive the Markov chain Monte Carlo (MCMC) algorithms for learning the SCNTM. We first describe the sampler for the topic model and then for the citation network. The full inference procedure is performed by alternating between the two samplers. Finally, we outline the hyperparameter samplers that are used to estimate the hyperparameters automatically.

To sample the words’ topic $\mathbf{Z}$ and the associated counts $\mathbf{T}$ and $\mathbf{C}$ in the SCNTM, we design a Metropolis-Hastings (MH) algorithm based on the collapsed Gibbs sampler designed for the PYP (Chen et al., 2011). The concept of the MH sampler is analogous to LDA, which consists of (1) decrementing the counts associated with a word, (2) sampling the respective new topic assignment for the word, and (3) incrementing the associated counts. However, our sampler is more complicated than LDA. In particular, we have to consider the indicators $u^{\mathcal{N}}_{dn}$ described in Section 4.1 operating on the hierarchy of PYPs. Our MH sampler consists of two steps. First we sample the latent topic $z_{dn}$ associated with the word $w_{dn}$. We then sample the customer counts $\mathbf{C}$ and table counts $\mathbf{T}$.

The sampler proceeds by considering the latent variables associated with a given word $w_{dn}$. First, we decrement the counts associated with the word $w_{dn}$ and the latent topic $z_{dn}$. This is achieved by sampling the suite of indicators $u_{dn}$ according to Equation 19 and decrementing the relevant customer counts and table counts. For example, we decrement $c^{\theta_{d}}_{z_{dn}}$ by 1 if $u^{\theta_{d}}_{dn}=1$. After decrementing, we apply a Gibbs sampler to sample a new topic $z_{dn}$ from its conditional posterior distribution, given as

Note that the joint distribution in Equation 28 can be written as the ratio of the likelihood for the topic model (Equation 21):

Next, we proceed to sample the relevant customer counts and table counts given the new $z_{dn}=k$. We propose an MH algorithm for this. We define the proposal distribution for the new customer counts and table counts as

Thus we always accept the proposed sample.The algorithm is named MH algorithm instead of Gibbs sampling due to the fact that the sample space for the counts is restricted and thus we are not sampling from the posterior directly. Note that since $\mu$ is GEM distributed, incrementing $t^{\mu}_{k}$ is equivalent to sampling a new topic, i.e. the number of topics increases by $1$.

2 Sampling for the Citation Network

For the citation network, we propose another MH algorithm. The MH algorithm can be summarised in three steps: (1) estimate the document topic prior $\theta^{\prime}$, (2) propose a new citing topic $y_{ij}$, and (3) accept or reject the proposed $y_{ij}$ following an MH scheme. Note that the MH algorithm is similar to the sampler for the topic model, where we decrement the counts, sample a new state and update the counts. Since all probability vectors are represented as counts, we do not need to deal with their vector form. Additionally, our MH algorithm is intuitive and simple to implement. Like the words in a document, each citation is assigned a topic, hence the words and citations can be thought as voting to determine a documents’ topic.

We describe our MH algorithm for the citation network as follows. First, for each document $d$, we estimate the expected document-topic prior $\hat{\theta}^{\prime}_{d}$ from Equation 22. Then, for each document pair $(i,j)$ where $x_{ij}=1$, we decrement the network counts associated with $x_{ij}$, and re-sample $y_{ij}$ with a proposal distribution derived from Equation 25:

Note that we have abused the notations $i$ and $j$ in the above equation, where the $i$ and $j$ in the summation indexes all documents instead of pointing to particular document $i$ and document $j$. We decided against introducing additional variables to make things less confusing.

Finally, if the sample is accepted, we update $y_{ij}$ and the associated customer counts. Otherwise, we discard the sample and revert the changes.

3 Hyperparameter Sampling

Hyperparameter sampling for the priors are important (Wallach et al., 2009). In our inference algorithm, we sample the concentration parameters $\beta$ of all PYPs with an auxiliary variable sampler (Teh, 2006a), but leave the discount parameters $\alpha$ fixed. We do not sample the $\alpha$ due to the coupling of the parameter with the Stirling numbers cache.

Here we outline the procedure to sample the concentration parameter $\beta^{\mathcal{N}}$ of a PYP distributed variable $\mathcal{N}$, using an auxiliary variable sampler. Assuming each $\beta^{\mathcal{N}}$ has a Gamma distributed hyperprior with shape $\tau_{0}$ and rate $\tau_{1}$, we first sample the auxiliary variables $\xi$ and $\psi_{j}$ for $j\in\{0,T^{\mathcal{N}}-1\}$:

We then sample a new $\beta^{\prime}{{}^{\mathcal{N}}}$ from the following conditional posterior given the auxiliary variables:

In addition to the PYP hyperparameters, we also sample $\lambda^{+}$, $\lambda^{-}$ and $\lambda^{T}$ with a Gibbs sampler. We let the hyperpriors for $\lambda^{+}$, $\lambda^{-}$ and $\lambda^{T}$ to be Gamma distributed with shape $\epsilon_{0}$ and rate $\epsilon_{1}$. With the conjugate Gamma prior, the posteriors for $\lambda^{+}_{i}$, $\lambda^{-}_{i}$ and $\lambda^{T}_{k}$ are also Gamma distributed, so they can be sampled directly.

We apply vague priors to the hyperpriors by setting $\tau_{0}=\tau_{1}=\epsilon_{0}=\epsilon_{1}=1$.

Before we proceed with the next section on the datasets used in the paper, we summarise the full inference algorithm for the SCNTM in Algorithm 1.

We perform our experiments on subsets of CiteSeerX datahttp://citeseerx.ist.psu.edu/ which consists of scientific publications. Each publication from CiteSeerX is accompanied by title, abstract, keywords, authors, citations and other metadata. We prepare three publication datasets from CiteSeerX for evaluations. The first dataset corresponds to Machine Learning (ML) publications, which are queried from CiteSeerX using the keywords from Microsoft Academic Search.http://academic.research.microsoft.com/ The ML dataset contains 139,227 publications. Our second dataset corresponds to publications from ten distinct research areas. The query words for these ten disciplines are chosen such that the publications form distinct clusters. We name this dataset M10 (Multidisciplinary 10 classes), which is made of 10,310 publications. For the third dataset, we query publications from both arts and science disciplines. Arts publications are made of history and religion publications, while the science publications contain physics, chemistry and biology research. This dataset consists of 18,720 publications and is named AvS (Arts versus Science) in this paper. These queried datasets are made available online.http://karwai.weebly.com/publications.html

The keywords used to create the datasets are obtained from Microsoft Academic Search, and are listed in LABEL:appendix:keywords. For the clustering evaluation in LABEL:subsubsec:clustering, we treat the query categories as the ground truth. However, publications that span multiple disciplines can be problematic for clustering evaluation, hence we simply remove the publications that satisfy the queries from more than one discipline. Nonetheless, the labels are inherently noisy. The metadata for the publications can also be noisy, for instance, the authors field may sometimes display publication’s keywords instead of the authors, publication title is sometimes an URL, and table of contents can be mistakenly parsed as the abstract. We discuss our treatments to these issues in LABEL:subsec:preprocessing. We also note that non-English publications are discarded using langid.py (Lui and Baldwin, 2012).

In addition to the manually queried datasets, we also make use of existing datasets from LINQS (Sen et al., 2008)http://linqs.cs.umd.edu/projects/projects/lbc/ to facilitate comparison with existing work. In particular, we use their CiteSeer, Cora and PubMed datasets. Their CiteSeer data consists of Computer Science publications and hence we name the dataset CS to remove ambiguity. Although these datasets are small, they are fully labelled and thus useful for clustering evaluation. However, these three datasets do not come with additional metadata such as the authorship information. Note that the CS and Cora datasets are presented as Boolean matrices, i.e. the word counts information is lost and we assume that all words in a document occur only once. Additionally, the words have been converted to integer so they do not convey any semantics. Although this representation is less useful for topic modelling, we still use them for the sake of comparison. For the PubMed dataset, we recover the word counts from TF-IDF using a simple assumption (see Section C). We present a summary of the datasets in Table 2 and their respective categorical labels in Section 6.

Biology: enzyme, gene expression, amino acid, escherichia coli, transcription factor, nucleotides, dna sequence, saccharomyces cerevisiae, plasma membrane, embryonics.

Computer Science: neural network, genetic algorithm, machine learning, information retrieval, data mining, computer vision, artificial intelligent, optimization problem, support vector machine, feature selection.

Social Science: developing country, higher education, decision making, health care, high school, social capital, social science, public health, public policy, social support.

Financial Economics: stock returns, interest rate, stock market, stock price, exchange rate, asset prices, capital market, financial market, option pricing, cash flow.

Material Science: microstructures, mechanical property, grain boundary, transmission electron microscopy, composite material, materials science, titanium, silica, differential scanning calorimetry, tensile properties.

Physics: magnetic field, quantum mechanics, field theory, black hole, kinetics, string theory, elementary particles, quantum field theory, space time, star formation.

Petroleum Chemistry: fly ash, diesel fuel, methane, methyl ester, diesel engine, natural gas, pulverized coal, crude oil, fluidized bed, activated carbon.

Industrial Engineering: power system, construction industry, induction motor, power converter, control system, voltage source inverter, permanent magnet, digital signal processor, sensorless control, field oriented control.

Archaeology: radiocarbon dating, iron age, bronze age, late pleistocene, middle stone age, upper paleolithic, ancient dna, early holocene, human evolution, late holocene.

Agriculture: irrigation water, soil water, water stress, drip irrigation, grain yield, crop yield, growing season, soil profile, soil salinity, crop production

History: nineteeth century, cold war, south africa, foreign policy, civil war, world war ii, latin america, western europe, vietnam, middle east.

Religion: social support, foster care, child welfare, human nature, early intervention, gender difference, sexual abuse, young adult, self esteem, social services.

Physics: magnetic field, quantum mechanics, string theory, field theory, numerical simulation, black hole, thermodynamics, phase transition, electric field, gauge theory.

Chemistry: crystal structure, mass spectrometry, copper, aqueous solution, binding site, hydrogen bond, oxidant stress, free radical, liquid chromatography, organic compound.

Biology: genetics, enzyme, gene expression, polymorphism, nucleotides, dna sequence, saccharomyces cerevisiae, cell cycle, plasma membrane, embryonics.

The PubMed dataset (Sen et al., 2008) was preprocessed to TF-IDF (term frequency-inverse document frequency) format, i.e. the raw word count information is lost. Here, we describe how we recover the word count information, using a simple and reasonable assumption – that the least occurring words in a document only occur once.

We denote $t_{dw}$ as the TF-IDF for word $w$ in document $d$, $f_{dw}$ as the corresponding term frequency (TF), and $i_{w}$ as the inverse document frequency (IDF) for word $w$. Our aim is to recover the word counts $c_{dw}$ given the TF-IDF. TF-IDF is computedNote that there are multiple ways to define a TF-IDF in practice. The specific TF-IDF formula used by the PubMed dataset was determined via trial-and-error and elimination. as

where $I(\cdot)$ is the indicator function.

We note that $I(c_{dw}>0)=I(t_{dw}>0)$ since the TF-IDF for a word $w$ is positive if and only if the corresponding word count is positive. This allows us to compute the IDF $i_{w}$ easily from Equation 53. We can then determine the TF:

Now we are left with computing $c_{dw}$ given the $f_{dw}$, however, we can obtain infinitely many solutions since we can always multiply $c_{dw}$ by a constant and get the same $f_{dw}$. Fortunately, since we are working with natural language, it is reasonable to assume that the least occurring words in a document only occur once, or mathematically,

Thus we can work out the normaliser $\sum_{w}c_{dw}$ and recover the word counts for all words in all documents.

Below is a list of words we use to filter out invalid authors during preprocessing step:

society, university, universität, universitat, author, advisor, acknowledgement, video, mathematik, abstract, industrial, review, example, department, information, enterprises, informatik, laboratory, introduction, encyclopedia, algorithm, section, available

Here, we show how to integrate out probability distributions using the expectation of a PYP: