A Latent Variable Recurrent Neural Network for Discourse Relation Language Models

Introduction

Natural language processing (NLP) has recently experienced a neural network “tsunami” [Manning, 2016]. A key advantage of these neural architectures is that they employ discriminatively-trained distributed representations, which can capture the meaning of linguistic phenomena ranging from individual words [Turian et al., 2010] to longer-range linguistic contexts at the sentence level [Socher et al., 2013] and beyond [Le and Mikolov, 2014]. Because they are discriminatively trained, these methods can learn representations that yield very accurate predictive models (e.g., Dyer et al, 2015).

However, in comparison with the probabilistic graphical models that were previously the dominant machine learning approach for NLP, neural architectures lack flexibility. By treating linguistic annotations as random variables, probabilistic graphical models can marginalize over annotations that are unavailable at test or training time, elegantly modeling multiple linguistic phenomena in a joint framework [Finkel et al., 2006]. But because these graphical models represent uncertainty for every element in the model, adding too many layers of latent variables makes them difficult to train.

In this paper, we present a hybrid architecture that combines a recurrent neural network language model with a latent variable model over shallow discourse structure. In this way, the model learns a discriminatively-trained distributed representation of the local contextual features that drive word choice at the intra-sentence level, using techniques that are now state-of-the-art in language modeling [Mikolov et al., 2010]. However, the model treats shallow discourse structure — specifically, the relationships between pairs of adjacent sentences — as a latent variable. As a result, the model can act as both a discourse relation classifier and a language model. Specifically:

If trained to maximize the conditional likelihood of the discourse relations, it outperforms state-of-the-art methods for both implicit discourse relation classification in the Penn Discourse Treebank [Rutherford and Xue, 2015] and dialog act classification in Switchboard [Kalchbrenner and Blunsom, 2013]. The model learns from both the discourse annotations as well as the language modeling objective, unlike previous recursive neural architectures that learn only from annotated discourse relations [Ji and Eisenstein, 2015].

If the model is trained to maximize the joint likelihood of the discourse relations and the text, it is possible to marginalize over discourse relations at test time, outperforming language models that do not account for discourse structure.

In contrast to recent work on continuous latent variables in recurrent neural networks [Chung et al., 2015], which require complex variational autoencoders to represent uncertainty over the latent variables, our model is simple to implement and train, requiring only minimal modifications to existing recurrent neural network architectures that are implemented in commonly-used toolkits such as Theano, Torch, and CNN.

We focus on a class of shallow discourse relations, which hold between pairs of adjacent sentences (or utterances). These relations describe how the adjacent sentences are related: for example, they may be in contrast, or the latter sentence may offer an answer to a question posed by the previous sentence. Shallow relations do not capture the full range of discourse phenomena [Webber et al., 2012], but they account for two well-known problems: implicit discourse relation classification in the Penn Discourse Treebank, which was the 2015 CoNLL shared task [Xue et al., 2015]; and dialog act classification, which characterizes the structure of interpersonal communication in the Switchboard corpus [Stolcke et al., 2000], and is a key component of contemporary dialog systems [Williams and Young, 2007]. Our model outperforms state-of-the-art alternatives for implicit discourse relation classification in the Penn Discourse Treebank, and for dialog act classification in the Switchboard corpus.

Background

Our model scaffolds on recurrent neural network (RNN) language models [Mikolov et al., 2010], and recent variants that exploit multiple levels of linguistic detail [Ji et al., 2015, Lin et al., 2015].

Let us denote token $n$ in a sentence $t$ by ${y}_{t,n}\in\{1\ldots V\}$, and write $\bm{y}_{t}=\{y_{t,n}\}_{n\in\{1\ldots N_{t}\}}$ to indicate the sequence of words in sentence $t$. In an RNN language model, the probability of the sentence is decomposed as,

The model as described thus far is identical to the recurrent neural network language model (RNNLM) of ?). In this paper, we replace the above simple hidden state units with the more complex Long Short-Term Memory units [Hochreiter and Schmidhuber, 1997], which have consistently been shown to yield much stronger performance in language modeling [Pham et al., 2014]. For simplicity, we still use the term RNNLM in referring to this model.

Document Context Language Model

One drawback of the RNNLM is that it cannot propagate long-range information between the sentences. Even if we remove sentence boundaries, long-range information will be attenuated by repeated application of the non-linear transition function. ?) propose the Document Context Language Model (DCLM) to address this issue. The core idea is to represent context with two vectors: $\bm{h}_{t,n}$, representing intra-sentence word-level context, and $\bm{c}_{t}$, representing inter-sentence context. These two vectors are then linearly combined in the generation function for word ${y}_{t,n}$,

where $\bm{c}_{t-1}$ is set to the last hidden state of the previous sentence. ?) show that this model can improve language model perplexity.

Discourse Relation Language Models

We now present a probabilistic neural model over sequences of words and shallow discourse relations. Discourse relations $z_{t}$ are treated as latent variables, which are linked with a recurrent neural network over words in a latent variable recurrent neural network [Chung et al., 2015].

Our model (see Figure 1) is formulated as a two-step generative story. In the first step, context information from the sentence $(t-1)$ is used to generate the discourse relation between sentences $(t-1)$ and $t$,

where ${z}_{t}$ is a random variable capturing the discourse relation between the two sentences, and $\bm{c}_{t-1}$ is a vector summary of the contextual information from sentence $(t-1)$, just as in the DCLM (Equation 5). The model maintains a default context vector $\bm{c}_{0}$ for the first sentences of documents, and treats it as a parameter learned with other model parameters during training.

In the second step, the sentence $\bm{y}_{t}$ is generated, conditioning on the preceding sentence $\bm{y}_{t-1}$ and the discourse relation ${z}_{t}$:

The generative probability for the sentence $\bm{y}_{t}$ decomposes across tokens as usual (Equation 7). The per-token probabilities are shown in Equation 4, in Figure 2. Discourse relations are incorporated by parameterizing the output matrices $\mathbf{W}_{o}^{({z}_{t})}$ and $\mathbf{W}_{c}^{({z}_{t})}$; depending on the discourse relation that holds between $(t-1)$ and $t$, these matrices will favor different parts of the embedding space. The bias term $\bm{b}_{o}^{({z}_{t})}$ is also parametrized by the discourse relation, so that each relation can favor specific words.

Overall, the joint probability of the text and discourse relations is,

If the discourse relations $z_{t}$ are not observed, then our model is a form of latent variable recurrent neural network (LVRNN). Connections to recent work on LVRNNs are discussed in § 6; the key difference is that the latent variables here correspond to linguistically meaningful elements, which we may wish to predict or marginalize, depending on the situation.

As proposed, the Discourse Relation Language Model has a large number of parameters. Let $K$, $H$ and $V$ be the input dimension, hidden dimension and the size of vocabulary in language modeling. The size of each prediction matrix $\mathbf{W}_{o}^{({z})}$ and $\mathbf{W}_{c}^{({z})}$ is $V\times H$; there are two such matrices for each possible discourse relation. We reduce the number of parameters by factoring each of these matrices into two components:

where $\mathbf{V}^{({z})}$ and $\mathbf{M}^{({z})}$ are relation-specific components for intra-sentential and inter-sentential contexts; the size of these matrices is $H\times H$, with $H\ll V$. The larger $V\times H$ matrices $\mathbf{W}_{o}$ and $\mathbf{W}_{c}$ are shared across all relations.

2 Inference

There are two possible inference scenarios: inference over discourse relations, conditioning on words; and inference over words, marginalizing over discourse relations.

The probability of discourse relations given the sentences $p(\bm{z}_{1:T}\mid\bm{y}_{1:T})$ is decomposed into the product of probabilities of individual discourse relations conditioned on the adjacent sentences $\prod_{t}p({z}_{t}\mid\bm{y}_{t},\bm{y}_{t-1})$. These probabilities are computed by Bayes’ rule:

The terms in each product are given in Equations 6 and 7. Normalizing involves only a sum over a small finite number of discourse relations. Note that inference is easy in our case because all words are observed and there is no probabilistic coupling of the discourse relations.

Inference over Words

In discourse-informed language modeling, we marginalize over discourse relations to compute the probability of a sequence of sentence $\bm{y}_{1:T}$, which can be written as,

because the word sequences are observed, decoupling each ${z}_{t}$ from its neighbors ${z}_{t+1}$ and ${z}_{t-1}$. This decoupling ensures that we can compute the overall marginal likelihood as a product over local marginals.

3 Learning

The model can be trained in two ways: to maximize the joint probability $p(\bm{y}_{1:T},\bm{z}_{1:T})$, or to maximize the conditional probability $p(\bm{z}_{1:T}\mid\bm{y}_{1:T})$. The joint training objective is more suitable for language modeling scenarios, and the conditional objective is better for discourse relation prediction. We now describe each objective in detail.

The joint likelihood objective function is directly adopted from the joint probability defined in Equation 8. The objective function for a single document with $T$ sentences or utterances is,

where $\bm{\theta}$ represents the collection of all model parameters, including the parameters in the LSTM units and the word embeddings.

Conditional objective

This training objective is specific to the discourse relation prediction task, and based on Equation 10 can be written as:

4 Modeling limitations

The discourse relation language model is carefully designed to decouple the discourse relations from each other, after conditioning on the words. It is clear that text documents and spoken dialogues have sequential discourse structures, and it seems likely that modeling this structure could improve performance. In a traditional hidden Markov model (HMM) generative approach [Stolcke et al., 2000], modeling sequential dependencies is not difficult, because training reduces to relative frequency estimation. However, in the hybrid probabilistic-neural architecture proposed here, training is already expensive, due to the large number of parameters that must be estimated. Adding probabilistic couplings between adjacent discourse relations $\langle z_{t-1},z_{t}\rangle$ would require the use of dynamic programming for both training and inference, increasing time complexity by a factor that is quadratic in the number of discourse relations. We did not attempt this in this paper; we do compare against a conventional HMM on the dialogue act prediction task in § 5.

?) propose an alternative form of the document context language model, in which the contextual information $\bm{c}_{t}$ impacts the hidden state $\bm{h}_{t+1}$, rather than going directly to the outputs $\bm{y}_{t+1}$. They obtain slightly better perplexity with this approach, which has fewer trainable parameters. However, this model would couple $z_{t}$ with all subsequent sentences $\bm{y}_{>t}$, making prediction and marginalization of discourse relations considerably more challenging. Sequential Monte Carlo algorithms offer a possible solution [de Freitas et al.,, Gu et al., 2015], which may be considered in future work.

Data and Implementation

We evaluate our model on two benchmark datasets: (1) the Penn Discourse Treebank [Prasad et al., 2008, PDTB], which is annotated on a corpus of Wall Street Journal acticles; (2) the Switchboard dialogue act corpus [Stolcke et al., 2000, SWDA], which is annotated on a collections of phone conversations. Both corpora contain annotations of discourse relations and dialogue relations that hold between adjacent spans of text.

provides a low-level discourse annotation on written texts. In the PDTB, each discourse relation is annotated between two argument spans, Arg1 and Arg2. There are two types of relations: explicit and implicit. Explicit relations are signalled by discourse markers (e.g., “however”, “moreover”), and the span of Arg1 is almost totally unconstrained: it can range from a single clause to an entire paragraph, and need not be adjacent to either Arg2 nor the discourse marker. However, automatically classifying these relations is considered to be relatively easy, due to the constraints from the discourse marker itself [Pitler et al., 2008]. In addition, explicit relations are difficult to incorporate into language models which must generate each word exactly once. On the contrary, implicit discourse relations are annotated only between adjacent sentences, based on a semantic understanding of the discourse arguments. Automatically classifying these discourse relations is a challenging task [Lin et al., 2009, Pitler et al., 2009, Rutherford and Xue, 2015, Ji and Eisenstein, 2015]. We therefore focus on implicit discourse relations, leaving to the future work the question of how to apply our modeling framework to explicit discourse relations. During training, we collapse all relation types other than implicit (explicit, EntRel, and NoRel) into a single dummy relation type, which holds between all adjacent sentence pairs that do not share an implicit relation.

As in the prior work on first-level discourse relation identification (e.g., Park and Cardie, 2012), we use sections 2-20 of the PDTB as the training set, sections 0-1 as the development set for parameter tuning, and sections 21-22 for testing. For preprocessing, we lower-cased all tokens, and substituted all numbers with a special token “NUM”. To build the vocabulary, we kept the 10,000 most frequent words from the training set, and replaced low-frequency words with a special token “UNK”. In prior work that focuses on detecting individual relations, balanced training sets are constructed so that there are an equal number of instances with and without each relation type [Park and Cardie,, Biran and McKeown, 2013, Rutherford and Xue, 2014]. In this paper, we target the more challenging multi-way classification problem, so this strategy is not applicable; in any case, since our method deals with entire documents, it is not possible to balance the training set in this way.

The Switchboard Dialog Act Corpus (SWDA)

is annotated on the Switchboard Corpus of human-human conversational telephone speech [Godfrey et al., 1992]. The annotations label each utterance with one of 42 possible speech acts, such as agree, hedge, and wh-question. Because these speech acts form the structure of the dialogue, most of them pertain to both the preceding and succeeding utterances (e.g., agree). The SWDA corpus includes 1155 five-minute conversations. We adopted the standard split from ?), using 1,115 conversations for training and nineteen conversations for test. For parameter tuning, we randomly select nineteen conversations from the training set as the development set. After parameter tuning, we train the model on the full training set with the selected configuration. We use the same preprocessing techniques here as in the PDTB.

1 Implementation

We use a single-layer LSTM to build the recurrent architecture of our models, which we implement in the cnn package.https://github.com/clab/cnn Our implementation is available on https://github.com/jiyfeng/drlm. Some additional details follow.

Following prior work on RNN initialization [Bengio, 2012], all parameters except the relation prediction parameters $\mathbf{U}$ and $\bm{b}$ are initialized with random values drawn from the range $[-\sqrt{6/(d_{1}+d_{2})},\sqrt{6/(d_{1}+d_{2})}]$, where $d_{1}$ and $d_{2}$ are the input and output dimensions of the parameter matrix respectively. The matrix $\mathbf{U}$ is initialized with random numbers from $[-10^{-5},10^{-5}]$ and $\bm{b}$ is initialized to $\bm{0}$.

Learning

Online learning was performed using AdaGrad [Duchi et al., 2011] with initial learning rate $\lambda=0.1$. To avoid the exploding gradient problem, we used norm clipping trick with a threshold of $\tau=5.0$ [Pascanu et al., 2012]. In addition, we used value dropout [Srivastava et al., 2014] with rate $0.5$, on the input $\mathbf{X}$, context vector $\bm{c}$ and hidden state $\bm{h}$, similar to the architecture proposed by ?). The training procedure is monitored by the performance on the development set. In our experiments, 4 to 5 epochs were enough.

Hyper-parameters

Our model includes two tunable hyper-parameters: the dimension of word representation $K$, the hidden dimension of LSTM unit $H$. We consider the values $\{32,48,64,96,128\}$ for both $K$ and $H$. For each corpus in experiments, the best combination of $K$ and $H$ is selected via grid search on the development set.

Experiments

Our main evaluation is discourse relation prediction, using the PDTB and SWDA corpora. We also evaluate on language modeling, to determine whether incorporating discourse annotations at training time and then marginalizing them at test time can improve performance.

We first evaluate our model with implicit discourse relation prediction on the PDTB dataset. Most of the prior work on first-level discourse relation prediction focuses on the “one-versus-all” binary classification setting, but we attack the more general four-way classification problem, as performed by ?). We compare against the following methods:

build a set of feature-rich classifiers on the PDTB, and then augment these classifiers with additional automatically-labeled training instances. We compare against their published results, which are state-of-the-art.

employ a recursive neural network architecture. Their experimental setting is different, so we re-run their system using the same setting as described in § 4.

As shown in Table 1, the conditionally-trained discourse relation language models (DrLM) outperforms all alternatives, on both metrics. While the jointly-trained DrLM is at the same level as the previous state-of-the-art, conditional training on the same model provides a significant additional advantage, indicated by a binomial test.

2 Dialogue Act tagging

Dialogue act tagging has been widely studied in both NLP and speech communities. We follow the setup used by ?) to conduct experiments, and adopt the following systems for comparison:

employ a hidden Markov model, with each HMM state corresponding to a dialogue act.

employ a complex neural architecture, with a convolutional network at each utterance and a recurrent network over the length of the dialog. To our knowledge, this model attains state-of-the-art accuracy on this task, outperforming other prior work such as [Webb et al., 2005, Milajevs and Purver, 2014].

As shown in Table 2, the conditionally-trained discourse relation language model (DrLM) outperforms all competitive systems on this task. A binomial test shows the result in line 6 is significantly better than the previous state-of-the-art (line 4). All comparisons are against published results, and Macro-$F_{1}$ scores are not available. Accuracy is more reliable on this evaluation, since no single class dominates, unlike the PDTB task.

3 Discourse-aware language modeling

As a joint model for discourse and language modeling, DrLM can also function as a language model, assigning probabilities to sequences of words while marginalizing over discourse relations. To determine whether discourse-aware language modeling can improve performance, we compare against the following systems:

This is the same basic architecture as the RNNLM proposed by [Mikolov et al., 2010], which was shown to outperform a Kneser-Ney smoothed 5-gram model on modeling Wall Street Journal text. Following ?), we replace the Sigmoid nonlinearity with a long short-term memory (LSTM).

We compare against the Document Context Language Model (DCLM) of ?). We use the “context-to-output” variant, which is identical to the current modeling approach, except that it is not parametrized by discourse relations. This model achieves strong results on language modeling for small and medium-sized corpora, outperforming RNNLM+LSTM.

The perplexities of language modeling on the PDTB and the SWDA are summarized in Table 3. The comparison between line 1 and line 2 shows the benefit of considering multi-sentence context information on language modeling. Line 3 shows that adding discourse relation information yields further improvements for both datasets. We emphasize that discourse relations in the test documents are marginalized out, so no annotations are required for the test set; the improvements are due to the disambiguating power of discourse relations in the training set.

Because our training procedure requires discourse annotations, this approach does not scale to the large datasets typically used in language modeling. As a consequence, the results obtained here are somewhat academic, from the perspective of practical language modeling. Nonetheless, the positive results here motivate the investigation of training procedures that are also capable of marginalizing over discourse relations at training time.

Related Work

This paper draws on previous work in both discourse modeling and language modeling.

Early work on discourse relation classification utilizes rich, hand-crafted feature sets [Joty et al., 2012, Lin et al., 2009, Sagae, 2009]. Recent representation learning approaches attempt to learn good representations jointly with discourse relation classifiers and discourse parsers [Ji and Eisenstein, 2014, Li et al., 2014]. Of particular relevance are applications of neural architectures to PDTB implicit discourse relation classification [Ji and Eisenstein, 2015, Zhang et al., 2015, Braud and Denis, 2015]. All of these approaches are essentially classifiers, and take supervision only from the 16,000 annotated discourse relations in the PDTB training set. In contrast, our approach is a probabilistic model over the entire text.

Probabilistic models are frequently used in dialog act tagging, where hidden Markov models have been a dominant approach [Stolcke et al., 2000]. In this work, the emission distribution is an $n$-gram language model for each dialogue act; we use a conditionally-trained recurrent neural network language model. An alternative neural approach for dialogue act tagging is the combined convolutional-recurrent architecture of ?). Our modeling framework is simpler, relying on a latent variable parametrization of a purely recurrent architecture.

Language modeling

There are an increasing number of attempts to incorporate document-level context information into language modeling. For example, ?) introduce LDA-style topics into RNN based language modeling. ?) use a convolutional structure to summarize the context from previous two utterances as context vector for RNN based language modeling. Our models in this paper provide a unified framework to model the context and current sentence. ?) and ?) construct bag-of-words representations of previous sentences, which are then used to inform the RNN language model that generates the current sentence. The most relevant work is the Document Context Language Model [Ji et al., 2015, DCLM]; we describe the connection to this model in § 2. By adding discourse information as a latent variable, we attain better perplexity on held-out data.

Latent variable neural networks

Introducing latent variables to a neural network model increases its representational capacity, which is the main goal of prior efforts in this space [Kingma and Welling, 2014, Chung et al., 2015]. From this perspective, our model with discourse relations as latent variables shares the same merit. Unlike this prior work, in our approach, the latent variables carry a linguistic interpretation, and are at least partially observed. Also, these prior models employ continuous latent variables, requiring complex inference techniques such as variational autoencoders [Kingma and Welling, 2014, Burda et al., 2016, Chung et al., 2015]. In contrast, the discrete latent variables in our model are easy to sum and maximize over.

Conclusion

We have presented a probabilistic neural model over sequences of words and shallow discourse relations between adjacent sequences. This model combines positive aspects of neural network architectures with probabilistic graphical models: it can learn discriminatively-trained vector representations, while maintaining a probabilistic representation of the targeted linguistic element: in this case, shallow discourse relations. This method outperforms state-of-the-art systems in two discourse relation detection tasks, and can also be applied as a language model, marginalizing over discourse relations on the test data. Future work will investigate the possibility of learning from partially-labeled training data, which would have at least two potential advantages. First, it would enable the model to scale up to the large datasets needed for competitive language modeling. Second, by training on more data, the resulting vector representations might support even more accurate discourse relation prediction.

Acknowledgments

Thanks to Trevor Cohn, Chris Dyer, Lingpeng Kong, and Quoc V. Le for helpful discussions, and to the anonymous reviewers for their feedback. This work was supported by a Google Faculty Research award to the third author. It was partially performed during the 2015 Jelinek Memorial Summer Workshop on Speech and Language Technologies at the University of Washington, Seattle, and was supported by Johns Hopkins University via NSF Grant No IIS 1005411, DARPA LORELEI Contract No HR0011-15-2-0027, and gifts from Google, Microsoft Research, Amazon and Mitsubishi Electric Research Laboratory.

References