Topic Compositional Neural Language Model

Introduction

A language model is a fundamental component to natural language processing (NLP). It plays a key role in many traditional NLP tasks, ranging from speech recognition (Mikolov et al. 2010; Arisoy et al. 2012; Sriram et al. 2017), machine translation (Schwenk et al. 2012; Vaswani et al. 2013) to image captioning (Mao et al. 2014; Devlin et al. 2015). Training a good language model often improves the underlying metrics of these applications, e.g., word error rates for speech recognition and BLEU scores (Papineni et al. 2002) for machine translation. Hence, learning a powerful language model has become a central task in NLP. Typically, the primary goal of a language model is to predict distributions over words, which has to encode both the semantic knowledge and grammatical structure in the documents. RNN-based neural language models have yielded state-of-the-art performance (Jozefowicz et al. 2016; Shazeer et al. 2017). However, they are typically applied only at the sentence level, without access to the broad document context. Such models may consequently fail to capture long-term dependencies of a document (Dieng et al. 2016).

Fortunately, such broader context information is of a semantic nature, and can be captured by a topic model. Topic models have been studied for decades and have become a powerful tool for extracting high-level semantic structure of document collections, by inferring latent topics. The classical Latent Dirichlet Allocation (LDA) method (Blei et al. 2003) and its variants, including recent work on neural topic models (Wan et al. 2012; Cao et al. 2015; Miao et al. 2017), have been useful for a plethora of applications in NLP.

Although language models that leverage topics have shown promise, they also have several limitations. For example, some of the existing methods use only pre-trained topic models (Mikolov and Zweig 2012), without considering the word-sequence prediction task of interest. Another key limitation of the existing methods lies in the integration of the learned topics into the language model; e.g., either through concatenating the topic vector as an additional feature of RNNs (Mikolov and Zweig 2012; Lau et al. 2017), or re-scoring the predicted distribution over words using the topic vector (Dieng et al. 2016). The former requires a balance between the number of RNN hidden units and the number of topics, while the latter has to carefully design the vocabulary of the topic model.

Motivated by the aforementioned goals and limitations of existing approaches, we propose the Topic Compositional Neural Language Model (TCNLM), a new approach to simultaneously learn a neural topic model and a neural language model. As depicted in Figure 1, TCNLM learns the latent topics within a variational autoencoder (Kingma and Welling 2013) framework, and the designed latent code ${\boldsymbol{t}}$ quantifies the probability of topic usage within a document. Latent code ${\boldsymbol{t}}$ is further used in a Mixture-of-Experts model (Hu et al. 1997), where each latent topic has a corresponding language model (expert). A combination of these “experts,” weighted by the topic-usage probabilities, results in our prediction for the sentences. A matrix factorization approach is further utilized to reduce computational cost as well as prevent overfitting. The entire model is trained end-to-end by maximizing the variational lower bound. Through a comprehensive set of experiments, we demonstrate that the proposed model is able to significantly reduce the perplexity of a language model and effectively assemble the meaning of topics to generate meaningful sentences. Both quantitative and qualitative comparisons are provided to verify the superiority of our model.

Preliminaries

We briefly review RNN-based language models and traditional probabilistic topic models.

A language model aims to learn a probability distribution over a sequence of words in a pre-defined vocabulary. We denote $\mathcal{V}$ as the vocabulary set and $\{y_{1},...,y_{M}\}$ to be a sequence of words, with each $y_{m}\in\mathcal{V}$. A language model defines the likelihood of the sequence through a joint probability distribution

RNN-based language models define the conditional probabiltiy of each word $y_{m}$ given all the previous words $y_{1:m-1}$ through the hidden state ${\boldsymbol{h}}_{m}$:

The function $f(\cdot)$ is typically implemented as a basic RNN cell, a Long Short-Term Memory (LSTM) cell (Hochreiter and Schmidhuber 1997), or a Gated Recurrent Unit (GRU) cell (Cho et al. 2014). The input and output words are related via the relation $x_{m}=y_{m-1}$.

Topic Model

A topic model is a probabilistic graphical representation for uncovering the underlying semantic structure of a document collection. Latent Dirichlet Allocation (LDA) (Blei et al. 2003), for example, provides a robust and scalable approach for document modeling, by introducing latent variables for each token, indicating its topic assignment. Specifically, let ${\boldsymbol{t}}$ denote the topic proportion for document $d$, and $z_{n}$ represent the topic assignment for word $w_{n}$. The Dirichlet distribution is employed as the prior of ${\boldsymbol{t}}$. The generative process of LDA may be summarized as:

where ${\boldsymbol{\beta}}_{z_{n}}$ represents the distribution over words for topic $z_{n}$, $\alpha_{0}$ is the hyper-parameter of the Dirichlet prior, $n\in[1,N_{d}]$, and $N_{d}$ is the number of words in document $d$. The marginal likelihood for document $d$ can be expressed as

Topic Compositional Neural Language Model

We describe the proposed TCNLM, as illustrated in Figure 1. Our model consists of two key components: (i) a neural topic model (NTM), and (ii) a neural language model (NLM). The NTM aims to capture the long-range semantic meanings across the document, while the NLM is designed to learn the local semantic and syntactic relationships between words.

where $\mathcal{N}(\mu_{0},\sigma_{0}^{2})$ is an isotropic Gaussian distribution, with mean $\mu_{0}$ and variance $\sigma_{0}^{2}$ in each dimension; $g(\cdot)$ is a transformation function that maps sample ${\boldsymbol{\theta}}$ to the topic embedding ${\boldsymbol{t}}$, defined here as $g({\boldsymbol{\theta}})=\mbox{softmax}(\hat{{\bf W}}{\boldsymbol{\theta}}+\hat{{\boldsymbol{b}}})$, where $\hat{{\bf W}}$ and $\hat{{\boldsymbol{b}}}$ are trainable parameters.

The marginal likelihood for document ${\boldsymbol{d}}$ is:

The second equation in (5) holds because we can readily marginalized out the sampled topic words $z_{n}$ by

The re-parameterization trick (Kingma and Welling 2013) can be applied to build an unbiased and low-variance gradient estimator for the variational distribution. The parameter updates can still be derived directly from the variational lower bound, as discussed in Section 3.3.

Redundance in inferred topics is a common issue exisiting in general topic models. In order to address this issue, it is straightforward to regularize the row-wise distance between each paired topics to diversify the topics. Following Xie et al. 2015; Miao et al. 2017, we apply a topic diversity regularization while carrying out the inference.

Specifically, the distance between a pair of topics are measured by their cosine distance $a({\boldsymbol{\beta}}_{i},{\boldsymbol{\beta}}_{j})=\arccos\left(\frac{|{\boldsymbol{\beta}}_{i}\cdot{\boldsymbol{\beta}}_{j}|}{||{\boldsymbol{\beta}}_{i}||_{2}||{\boldsymbol{\beta}}_{j}||_{2}}\right)$. The mean angle of all pairs of $T$ topics is $\phi=\frac{1}{T^{2}}\sum_{i}\sum_{j}a({\boldsymbol{\beta}}_{i},{\boldsymbol{\beta}}_{j})$, and the variance is $\nu=\frac{1}{T^{2}}\sum_{i}\sum_{j}(a({\boldsymbol{\beta}}_{i},{\boldsymbol{\beta}}_{j})-\phi)^{2}$. Finally, the topic diversity regularization is defined as $R=\phi-\nu$.

2 Neural Language Model

We propose a Mixture-of-Experts (MoE) language model, which consists a set of “expert networks”, i.e., $E_{1},E_{2},...,E_{T}$. Each expert is itself an RNN with its own parameters corresponding to a latent topic.

where ${\boldsymbol{t}}_{k}$ is the usage of topic $k$ (component $k$ of ${\boldsymbol{t}}$), and $\sigma(\cdot)$ is a sigmoid function; ${\bf V}$ is the weight matrix connecting the RNN’s hidden state, used for computing a distribution over words. Bias terms are omitted for simplicity.

However, such an MoE module is computationally prohibitive and storage excessive. The training process is inefficient and even infeasible in practice. To remedy this, instead of ensembling the output of the $T$ experts as in (7), we extend the weight matrix of the RNN to be an ensemble of topic-dependent weight matrices. Specifically, the $T$ experts work together as follows:

In order to reduce the number of model parameters, motivated by Gan et al. 2016; Song et al. 2016, instead of implementing a tensor as in (11), we decompose ${\bf W}({\boldsymbol{t}})$ into a multiplication of three terms ${\bf W}_{a}\in R^{n_{h}\times n_{f}}$, ${\bf W}_{b}\in R^{n_{f}\times T}$ and ${\bf W}_{c}\in R^{n_{f}\times n_{x}}$, where $n_{f}$ is the number of factors. Specifically,

where $\odot$ represents the Hadamard operator. ${\bf W}_{a}$ and ${\bf W}_{c}$ are shared parameters across all topics, to capture the common linguistic patterns. ${\bf W}_{b}$ are the factors which are weighted by the learned topic embedding ${\boldsymbol{t}}$. The same factorization is also applied for ${\bf U}({\boldsymbol{t}})$. The topic distribution ${\boldsymbol{t}}$ affects RNN parameters associated with the document when predicting the succeeding words, which implicitly defines an ensemble of $T$ language models. In this factorized model, the RNN weight matrices that correspond to each topic share “structure”.

Now we generalize the above analysis by using LSTM units. Specifically, we summarize the new topic compositional LSTM cell as:

Compared with a standard LSTM cell, our LSTM unit has a total number of parameters in size of $4n_{f}\cdot(n_{x}+2T+3n_{h})$ and the additional computational cost comes from (14) and (15). Further, empirical comparison has been conducted in Section 5.6 to verify that our proposed model is superior than using the naive MoE implementation as in (7).

3 Model Inference

The proposed model (see Figure 1) follows the variational autoencoder (Kingma and Welling 2013) framework, which takes the bag-of-words as input and embeds a document into the topic vector. This vector is then used to reconstruct the bag-of-words input, and also to learn an ensemble of RNNs for predicting a sequence of words in the document.

The joint marginal likelihood can be written as:

Since the direct optimization of (16) is intractable, we employ variational inference (Jordan et al. 1999). We denote $q({\boldsymbol{t}}|{\boldsymbol{d}})$ to be the variational distribution for ${\boldsymbol{t}}$. Hence, we construct the variational objective function, also called the evidence lower bound (ELBO), as

More details can be found in the Supplementary Material. In experiments, we optimize the ELBO together with the diversity regularisation:

Related Work

Topic Model Topic models have been studied for a variety of applications in document modeling. Beyond LDA (Blei et al. 2003), significant extensions have been proposed, including capturing topic correlations (Blei and Lafferty 2007), modeling temporal dependencies (Blei and Lafferty 2006), discovering an unbounded number of topics (Teh et al. 2005), learning deep architectures (Henao et al. 2015; Zhou et al. 2015), among many others. Recently, neural topic models have attracted much attention, building upon the successful usage of restricted Boltzmann machines (Hinton and Salakhutdinov 2009), auto-regressive models (Larochelle and Lauly 2012), sigmoid belief networks (Gan et al. 2015), and variational autoencoders (Miao et al. 2016).

Variational inference has been successfully applied in a variety of applications (Pu et al. 2016; Wang et al. 2017; Chen et al. 2017). The recent work of Miao et al. 2017 employs variational inference to train topic models, and is closely related to our work. Their model follows the original LDA formulation and extends it by parameterizing the multinomial distribution with neural networks. In contrast, our model enforces the neural network not only modeling documents as bag-of-words, but also transfering the inferred topic knowledge to a language model for word-sequence generation.

Neural language models have recently achieved remarkable advances (Mikolov et al. 2010). The RNN-based language model (RNNLM) is superior for its ability to model longer-term temporal dependencies without imposing a strong conditional independence assumption; it has recently been shown to outperform carefully-tuned traditional n-gram-based language models (Jozefowicz et al. 2016).

An RNNLM can be further improved by utilizing the broad document context (Mikolov and Zweig 2012). Such models typically extract latent topics via a topic model, and then send the topic vector to a language model for sentence generation. Important work in this direction include Mikolov and Zweig 2012; Dieng et al. 2016; Lau et al. 2017; Ahn et al. 2016. The key differences of these methods is in either the topic model itself or the method of integrating the topic vector into the language model. In terms of the topic model, Mikolov and Zweig 2012 uses a pre-trained LDA model; Dieng et al. 2016 uses a variational autoencoder; Lau et al. 2017 introduces an attention-based convolutional neural network to extract semantic topics; and Ahn et al. 2016 utilizes the topic associated to the fact pairs derived from a knowledge graph (Vinyals and Le 2015).

Concerning the method of incorporating the topic vector into the language model, Mikolov and Zweig 2012 and Lau et al. 2017 extend the RNN cell with additional topic features. Dieng et al. 2016 and Ahn et al. 2016 use a hybrid model combining the predicted word distribution given by both a topic model and a standard RNNLM. Distinct from these approaches, our model learns the topic model and the language model jointly under the VAE framework, allowing an efficient end-to-end training process. Further, the topic information is used as guidance for a Mixture-of-Experts (MoE) model design. Under our factorization method, the model can yield boosted performance efficiently (as corroborated in the experiments).

Recently, Shazeer et al. 2017 proposes a MoE model for large-scale language modeling. Different from ours, they introduce a MoE layer, in which each expert stands for a small feed-forward neural network on the previous output of the LSTM layer. Therefore, it yields a significant quantity of additional parameters and computational cost, which is infeasible to train on a single GPU machine. Moreover, they provide no semantic meanings for each expert, and all experts are treated equally; the proposed model can generate meaningful sentences conditioned on given topics.

Our TCNLM is similar to Gan et al. 2016. However, Gan et al. 2016 uses a two-step pipline, first learning a multi-label classifier on a group of pre-defined image tags, and then generating image captions conditioned on them. In comparison, our model jointly learns a topic model and a language model, and focuses on the language modeling task.

Experiments

We present experimental results on three publicly available corpora: APNEWS, IMDB and BNC. APNEWS https://www.ap.org/en-gb/ is a collection of Associated Press news articles from 2009 to 2016. IMDB is a set of movie reviews collected by Maas et al. 2011, and BNC (BNC Consortium 2007) is the written portion of the British National Corpus, which contains excerpts from journals, books, letters, essays, memoranda, news and other types of text. These three datasets can be downloaded from GitHub https://github.com/jhlau/topically-driven-language-model.

We follow the preprocessing steps in Lau et al. 2017. Specifically, words and sentences are tokenized using Stanford CoreNLP (Manning et al. 2014). We lowercase all word tokens, and filter out word tokens that occur less than 10 times. For topic modeling, we additionally remove stopwords We use the following stopwords list: https://github. com/mimno/Mallet/blob/master/stoplists/en.txt in the documents and exclude the top $0.1\%$ most frequent words and also words that appear in less than 100 documents. All these datasets are divided into training, development and testing sets. A summary statistic of these datasets is provided in Table 1.

Setup

For the NTM part, we consider a 2-layer feed-forward neural network to model $q({\boldsymbol{t}}|{\boldsymbol{d}})$, with $256$ hidden units in each layer; ReLU (Nair and Hinton 2010) is used as the activation function. The hyper-parameter $\lambda$ for the diversity regularizer is fixed to 0.1 across all the experiments. All the sentences in a paragraph, excluding the one being predicted, are used to obtain the bag-of-words document representation ${\boldsymbol{d}}$. The maximum number of words in a paragraph is set to $300$.

In terms of the NLM part, we consider 2 settings: (i) a small 1-layer LSTM model with $600$ hidden units, and (ii) a large 2-layer LSTM model with $900$ hidden units in each layer. The sequence length is fixed to 30. In order to alleviate overfitting, dropout with a rate of $0.4$ is used in each LSTM layer. In addition, adaptive softmax (Grave et al. 2016) is used to speed up the training process.

During training, the NTM and NLM parameters are jointly learned using Adam (Kingma and Ba 2014). All the hyper-parameters are tuned based on the performance on the development set. We empirically find that the optimal settings are fairly robust across the 3 datasets. All the experiments were conducted using Tensorflow and trained on NVIDIA GTX TITAN X with 3072 cores and 12GB global memory.

1 Language Model Evaluation

Perplexity is used as the metric to evaluate the performance of the language model. In order to demonstrate the advantage of the proposed model, we compare TCNLM with the following baselines:

basic-LSTM: A baseline LSTM-based language model, using the same architecture and hyper-parameters as TCNLM wherever applicable.

LDA+LSTM: A topic-enrolled LSTM-based language model. We first pretrain an LDA model (Blei et al. 2003) to learn 50/100/150 topics for APNEWS, IMDB and BNC. Given a document, the LDA topic distribution is incorporated by concatenating with the output of the hidden states to predict the next word.

LCLM (Wang and Cho 2016): A context-based language model, which incorporates context information from preceding sentences. The preceding sentences are treated as bag-of-words, and an attention mechanism is used when predicting the next word.

TDLM (Lau et al. 2017): A convolutional topic model enrolled languge model. Its topic knowledge is utilized by concatenating to a dense layer of a recurrent language model.

Topic-RNN (Dieng et al. 2016): A joint learning framework that learns a topic model and a language model simutaneously. The topic information is incorporated through a linear transformation to rescore the prediction of the next word.

Topic-RNN (Dieng et al. 2016) is implemented by ourselves and other comparisons are copied from (Lau et al. 2017). Results are presented in Table 2. We highlight some observations. (i) All the topic-enrolled methods outperform the basic-LSTM model, indicating the effectiveness of incorporating global semantic topic information. (ii) Our TCNLM performs the best across all datasets, and the trend keeps improving with the increase of topic numbers. (iii) The improved performance of TCNLM over LCLM implies that encoding the document context into meaningful topics provides a better way to improve the language model compared with using the extra context words directly. (iv) The margin between LDA+LSTM/Topic-RNN and our TCNLM indicates that our model supplies a more efficient way to utilize the topic information through the joint variational learning framework to implicitly train an ensemble model.

2 Topic Model Evaluation

We evaluate the topic model by inspecting the coherence of inferred topics (Chang et al. 2009; Newman et al. 2010; Mimno et al. 2011). Following Lau et al. 2014, we compute topic coherence using normalized PMI (NPMI). Given the top $n$ words of a topic, the coherence is calculated based on the sum of pairwise NPMI scores between topic words, where the word probabilities used in the NPMI calculation are based on co-occurrence statistics mined from English Wikipedia with a sliding window. In practice, we average topic coherence over the top $5/10/15/20$ topic words. To aggregate topic coherence score for a trained model, we then further average the coherence scores over topics. For comparison, we use the following baseline topic models:

LDA: LDA (Blei et al. 2003) is used as a baseline topic model. We use LDA to learn the topic distributions for LDA+LSTM.

NTM: We evaluate the neural topic model proposed in Cao et al. 2015. The document-topic and topic-words multinomials are expressed using neural networks. N-grams embeddings are incorporated as inputs of the model.

TDLM (Lau et al. 2017): The same model as used in the language model evaluation.

Topic-RNN (Dieng et al. 2016): The same model as used in the language model evaluation.

Results are summarized in Table 4. Our TCNLM achieves promising results. Specifically, (i) we achieve the best coherence performance over APNEWS and IMDB, and are relatively competitive with LDA on BNC. (ii) We also observe that a larger model may result in a slightly worse coherence performance. One possible explanation is that a larger language model may have more impact on the topic model, and the inherited stronger sequential information may be harmful to the coherence measurement. (iii) Additionally, the advantage of our TCNLM over Topic-RNN indicates that our TCNLM supplies a more powerful topic guidance.

In order to better understand the topic model, we provide the top 5 words for 10 randomly chosen topics on each dataset (the boldface word is the topic name summarized by us), as shown in Table 3. These results correspond to the small network with 100 neurons. We also present some inferred topic distributions for several documents from our TCNLM in Figure 2. The topic usage for a specific document is sparse, demonstrating the effectiveness of our NTM. More inferred topic distribution examples are provided in the Supplementary Material.

3 Sentence Generation

Another advantage of our TCNLM is its capacity to generate meaningful sentences conditioned on given topics. Given topic i, we construct an LSTM generator by using only the $i$-th factor of ${\bf W}_{b}$ and ${\bf U}_{b}$. Then we start from a zero hidden state, and greedily sample words until an end token occurs. Table 5 shows the generated sentences from our TCNLM learned with 50 topics using the small network. Most of the sentences are strongly correlated with the given topics. More interestingly, we can also generate reasonable sentences conditioned on a mixed combination of topics, even if the topic pairs are divergent, e.g., “animal” and “lottory” for APNEWS. More examples are provided in the Supplementary Material. It shows that our TCNLM is able to generate topic-related sentences, providing an interpretable way to understand the topic model and the language model simulaneously. These qualitative analysis further demonstrate that our model effectively assembles the meaning of topics to generate sentences.

4 Empirical Comparison with Naive MoE

We explore the usage of a naive MoE language model as in (7). In order to fit the model on a single GPU machine, we train a NTM with $30$ topics and each NLM of the MoE is a 1-layer LSTM with $100$ hidden units. Results are summarized in Table 6. Both the naive MoE and our TCNLM provide better performance than the basic LSTM. Interestingly, though requiring less computational cost and storage usage, our TCNLM outperforms the naive MoE by a non-trivial margin. We attribute this boosted performance to the “structure” design of our matrix factorization method. The inherent topic-guided factor control significantly prevents overfitting, and yields efficient training, demonstrating the advantage of our model for transferring semantic knowledge learned from the topic model to the language model.

Conclusion

We have presented Topic Compositional Neural Language Model (TCNLM), a new method to learn a topic model and a language model simultaneously. The topic model part captures the global semantic meaning in a document, while the language model part learns the local semantic and syntactic relationships between words. The inferred topic information is incorporated into the language model through a Mixture-of-Experts model design. Experiments conducted on three corpora validate the superiority of the proposed approach. Further, our model infers sensible topics, and has the capacity to generate meaningful sentences conditioned on given topics. One possible future direction is to extend the TCNLM to a conditional model and apply it for the machine translation task.

References

Appendix A Detailed model inference

We provide the detailed derivation for the model inference. Start from (16), we have

Appendix B Documents used to infer topic distributions

The documents used to infer the topic distributions ploted in Figure 2 are provided below.

: colombia ’s police director says six police officers have been killed and a seventh wounded in ambush in a rural southwestern area where leftist rebels operate . gen. jose roberto leon tells the associated press that the officers were riding on four motorcycles when they were attacked with gunfire monday afternoon on a rural stret ch of highway in the cauca state town of padilla . he said a front of the revolutionary armed forces of colombia , or farc , operates in the area . if the farc is r esponsible , the deaths would bring to 15 the number of security force members killed since the government and rebels formally opened peace talks in norway on oct. 18 . the talks to end a nearly five-decade-old conflict are set to begin in earnest in cuba on nov. 15 .

IMDB

: having just watched this movie for a second time , some years after my initial viewing , my feelings remain unchanged . this is a solid sci-fi drama that i enjoy very much . what sci-fi elements there are , are primarily of added interest rather than the main substance of the film . what this movie is really about is wartime confl ict , but in a sci-fi setting . it has a solid cast , from the ever reliable david warner to the up and coming freddie prinze jr , also including many british tv regu lars ( that obviously add a touch of class :) , not forgetting the superb tcheky karyo . i feel this is more of an ensemble piece than a starring vehicle . reminisc ent of wwii films based around submarine combat and air-combat ( the fighters seem like adaptations of wwii corsairs in their design , evoking a retro feel ) this is one of few american films that i felt was not overwhelmed by sentiment or saccharine . the sets and special effects are all well done , never detracting form the bel ievability of the story , although the kilrathi themselves are rather under developed and one dimensional . this is a film more about humanity in conflict rather than a film about exploring a new and original alien race or high-brow sci-fi concepts . forget that it ’s sci-fi , just watch and enjoy .

BNC: an army and civilian exercise went ahead in secret yesterday a casualty of the general election . the simulated disaster in exercise gryphon ’s lift was a midair coll ision between military and civilian planes over catterick garrison . hamish lumsden , the ministry of defence ’s chief press officer who arrived from london , said : ’ there ’s an absolute ban on proactive pr during an election . ’ journalists who reported to gaza barracks at 7.15 am as instructed were told they would not be all owed to witness the exercise , which involved 24 airmobile brigade , north yorkshire police , fire and ambulance services , the county emergency planning department a nd ’ casualties ’ from longlands college , middlesbrough . the aim of the gryphon lift was to test army support for civil emergencies . brief details supplied to th e press outlined the disaster . a fully loaded civilian plane crashes in mid-air with an armed military plane over catterick garrison . the 1st battalion the green ho wards and a bomb disposal squad cordon and clear an area scattered with armaments . 24 airmobile field ambulance , which served in the gulf war , tends a burning , pa cked youth hostel hit by pieces of aircraft . 38 engineer regiment from claro barracks , ripon , search a lake where a light aircraft crashed when hit by flying wreck age . civilian emergency services , including the police underwater team , were due to work alongside military personnel under the overall co-ordination of the police . mr lumsden said : ’ there is a very very strict rule that during a general election nothing that the government does can intrude on the election process . ’

Appendix C More inferred topic distribution examples

We present the inferred topic distributions for the first 5 documents in the development set over each dataset in Figure 3.

Appendix D More generated sentences

We present generated sentences using the topics listed in Table 3 for each dataset. The generated sentences for a single topic are provided in Table 7, 8, 9; the generated sentences for a mixed combination of topics are provided in Table 10.