Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning

Introduction

Modern artificial neural network approaches to natural language understanding tasks like translation (Sutskever et al., 2014; Cho et al., 2014), summarization (Rush et al., 2015), and classification (Yang et al., 2016) depend crucially on subsystems called sentence encoders that construct distributed representations for sentences. These encoders are typically implemented as convolutional (Kim, 2014), recursive (Socher et al., 2013), or recurrent neural networks (Mikolov et al., 2010) operating over a sentence’s words or characters (Zhang et al., 2015; Kim et al., 2016).

Most of the early successes with sentence encoder-based models have been on tasks with ample training data, where it has been possible to train the encoders in a fully-supervised end-to-end setting. However, recent work has shown some success in using unsupervised pretraining with unlabeled data to both improve the performance of these methods and extend them to lower-resource settings (Dai and Le, 2015; Kiros et al., 2015; Bajgar et al., 2016).

This paper presents a set of methods for unsupervised pretraining that train sentence encoders to recognize discourse coherence. When reading text, human readers have an expectation of coherence from one sentence to the next. In most cases, for example, each sentence in a text should be both interpretable in context and relevant to the topic under discussion. Both of these properties depend on an understanding of the local context, which includes both relatively knowledge about the state of the world and the specific meanings of previous sentences in the text. Thus, a model that is successfully trained to recognize discourse coherence must be able to understand the meanings of sentences as well as relate them to key pieces of knowledge about the world.

Hobbs (1979) presents a formal treatment of this phenomenon. He argues that for a discourse (here, a text) to be interpreted as coherent, any two adjacent sentences must be related by one of a few set kinds of coherence relations. For example, a sentence might be followed by another that elaborates on it, parallels it, or contrasts with it. While this treatment may not be adequate to cover the full complexity of language understanding, it allows Hobbs to show how identifying such relations depends upon sentence understanding, coreference resolution, and commonsense reasoning.

Recently proposed techniques (Kiros et al., 2015; Ramachandran et al., 2016) succeed in exploiting discourse coherence information of this kind to train sentence encoders, but rely on generative objectives which require models to compute the likelihood of each word in a sentence at training time. In this setting, a single epoch of training on a typical (76M sentence) text corpus can take weeks, making further research difficult, and making it nearly impossible to scale these methods to the full volume of available unlabeled English text. In this work, we propose alternative objectives which exploit much of the same coherence information at greatly reduced cost.

In particular, we propose three fast coherence-based pretraining tasks, show that they can be used together effectively in multitask training (Figure 1), and evaluate models trained in this setting on the training tasks themselves and on standard text classification tasks.All code, resources, and models involved in these experiments will be made available upon publication. We find that our approach makes it possible to learn to extract broadly useful sentence representations in hours.

Related Work

This work is inspired most directly by the Skip Thought approach of Kiros et al. (2015), which introduces the use of paragraph-level discourse information for the unsupervised pretraining of sentence encoders. Since that work, three other papers have presented improvements to this method (the SDAE of Hill et al. 2016, also Gan et al. 2016; Ramachandran et al. 2016). These improved methods are based on techniques and goals that are similar to ours, but all three involve models that explicitly generate full sentences during training time at considerable computational cost.

In closely related work, Logeswaran et al. (2016) present a model that learns to order the sentences of a paragraph. While they focus on learning to assess coherence, they show positive results on measuring sentence similarity using their trained encoder. Alternately, the FastSent model of Hill et al. (2016) is designed to work dramatically more quickly than systems like Skip Thought, but in service of this goal the standard sentence encoder RNN is replaced with a low-capacity CBOW model. Their method does well on existing semantic textual similarity benchmarks, but its insensitivity to order places an upper bound on its performance in more intensive extrinsic language understanding tasks.

Looking beyond work on unsupervised pretraining: Li and Hovy (2014) and Li and Jurafsky (2016) use representation learning systems to directly model the problem of sentence order recovery, but focus primarily on intrinsic evaluation rather than transfer. Wang and Cho (2016) train sentence representations for use as context in language modeling. In addition, Ji et al. (2016) treat discourse relations between sentences as latent variables and show that this yields improvements in language modeling in an extension of the document-context model of Ji et al. (2015).

Outside the context of representation learning, there has been a good deal of work in NLP on discourse coherence, and on the particular tasks of sentence ordering and coherence scoring. Barzilay and Lapata (2008) provide thorough coverage of this work.

Discourse Inspired Objectives

In this work, we propose three objective functions for use over paragraphs extracted from unlabeled text. Each captures a different aspect of discourse coherence and together the three can be used to train a single encoder to extract broadly useful sentence representations.

Many coherence relations have an inherent direction. For example, if $S_{1}$ is an elaboration of $S_{0}$, $S_{0}$ is not generally an elaboration of $S_{1}$. Thus, being able to identify these coherence relations implies an ability to recover the original order of the sentences. Our first task, which we call order, consists in taking pairs of adjacent sentences from text data, switching their order with probability 0.5, and training a model to decide whether they have been switched. Table 1 provides some examples of this task, along with the kind of coherence relation that we assume to be involved. It should be noted that since some of these relations are unordered, it is not always possible to recover the original order based on discourse coherence alone (see e.g. the flowers / birds example).

Next Sentence

Many coherence relations are transitive by nature, so that any two sentences from the same paragraph will exhibit some coherence. However, two adjacent sentences will generally be more coherent than two more distant ones. This leads us to formulate the next task: given the first three sentences of a paragraph and a set of five candidate sentences from later in the paragraph, the model must decide which candidate immediately follows the initial three in the source text. Table 2 presents an example of such a task: candidates 2 and 3 are coherent with the third sentence of the paragraph, but the elaboration (3) takes precedence over the progression (2).

Conjunction Prediction

Finally, information about the coherence relation between two sentences is sometimes apparent in the text (Miltsakaki et al., 2004): this is the case whenever the second sentence starts with a conjunction phrase. To form the conjunction objective, we create a list of conjunction phrases and group them into nine categories (see supplementary material). We then extract from our source text all pairs of sentences where the second starts with one of the listed conjunctions, give the system the pair without the phrase, and train it to recover the conjunction category. Table 3 provides examples.

Experiments

In this section, we introduce our training data and methods, present qualitative results and comparisons among our three objectives, and close with quantitative comparisons with related work.

We train our models on a combination of data from BookCorpus (Zhu et al., 2015), the Gutenberg project (Stroube, 2003), and Wikipedia. After sentence and word tokenization (with NLTK; Bird, 2006) and lower-casing, we identify all paragraphs longer than 8 sentences and extract a next example from each, as well as pairs of sentences for the order and conjunction tasks. This gives us 40M examples for order, 1.4M for conjunction, and 4.1M for next.

Despite having recently become a standard dataset for unsupervised learning, BookCorpus does not exhibit sufficiently rich discourse structure to allow our model to fully succeed—in particular, some of the conjunction categories are severely under-represented. Because of this, we choose to train our models on text from all three sources. While this precludes a strict apples-to-apples comparison with other published results, our goal in extrinsic evaluation is simply to show that our method makes it possible to learn useful representations quickly, rather than to demonstrate the superiority of our learning technique given fixed data and unlimited time.

We consider three sentence encoding models: a simple 1024D sum-of-Words (CBOW) encoding, a 1024D GRU recurrent neural network (Cho et al., 2014), and a 512D bidirectional GRU RNN (BiGRU). All three use FastText (Joulin et al., 2016) pre-trained word embeddingshttps://github.com/facebookresearch/fastText/ blob/master/pretrained-vectors.md to which we apply a Highway transformation (Srivastava et al., 2015). The encoders are trained jointly with three bilinear classifiers for the three objectives (for the next examples, the three context sentences are encoded separately and their representations are concatenated). We perform stochastic gradient descent with AdaGrad (Duchi et al., 2011), subsampling conjunction and next by a factor of 4 and 6 respectively (chosen using held-out accuracy averaged over all three tasks on held out data after training on 1M examples). In this setting, the BiGRU model takes 8 hours to see all of the examples from the BookCorpus dataset at least once. For ease of comparison, we train all three models for exactly 8 hours.

Intrinsic and Qualitative Evaluation

Table 4 compares the performance of different training regimes along two axes: encoder architecture and whether we train one model per task or one joint model. As expected, the more complex bidirectional GRU architecture is required to capture the appropriate sentence properties, although CBOW still manages to beat the simple GRU (the slowest model), likely by virtue of its substantially faster speed, and correspondingly greater number of training epochs. Joint training does appear to be effective, as both the order and next tasks benefit from the information provided by conjunction. Early experiments on the external evaluation also show that the joint BiGRU model substantially outperforms each single model.

Table 5 and the supplementary material show nearest neighbors in the trained BiGRU’s representation space for a random set of seed sentences. We select neighbors from among 400k held-out sentences. The encoder appears to be especially sensitive to high-level syntactic structure.

Extrinsic Evaluation

We evaluate the quality of the encoder learned by our system, which we call DiscSent, by using the sentence representations it produces in a variety of sentence classification tasks. We follow the settings of Kiros et al. (2015) on paraphrase detection (MSRP; Dolan et al., 2004), subjectivity evaluation (SUBJ; Pang and Lee, 2004) and question classification (TREC; Voorhees, 2001).

Overall, our system performs comparably with the SDAE and Skip Thought approaches with a drastically shorter training time. Our system also compares favorably to the similar discourse-inspired method of Logeswaran et al. (2016), achieving similar results on MSRP in a sixth of their training time.

Conclusion

In this work, we introduce three new training objectives for unsupervised sentence representation learning inspired by the notion of discourse coherence, and use them to train a sentence representation system in competitive time, from 6 to over 40 times shorter than comparable methods, while obtaining comparable results on external evaluations tasks. We hope that the tasks that we introduce in this paper will prompt further research into discourse understanding with neural networks, as well as into strategies for unsupervised learning that will make it possible to use unlabeled data to train and refine a broader range of models for language understanding tasks.

References

Supplement to: Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning

Table 7 lists the conjunction phrases and groupings used. Table 8 (next page) shows the Euclidean nearest neighbors of a sample of sentences in our representation space.