Learning Distributed Representations of Sentences from Unlabelled Data

Introduction

Distributed representations - dense real-valued vectors that encode the semantics of linguistic units - are ubiquitous in today’s NLP research. For single-words or word-like entities, there are established ways to acquire such representations from naturally occurring (unlabelled) training data based on comparatively task-agnostic objectives (such as predicting adjacent words). These methods are well understood empirically [Baroni et al., 2014b] and theoretically [Levy and Goldberg, 2014]. The best word representation spaces reflect consistently-observed aspects of human conceptual organisation [Hill et al., 2015b], and can be added as features to improve the performance of numerous language processing systems [Collobert et al., 2011].

By contrast, there is comparatively little consensus on the best ways to learn distributed representations of phrases or sentences.See the contrasting conclusions in [Mitchell and Lapata, 2008, Clark and Pulman, 2007, Baroni et al., 2014a, Milajevs et al., 2014] among others. With the advent of deeper language processing techniques, it is relatively common for models to represent phrases or sentences as continuous-valued vectors. Examples include machine translation [Sutskever et al., 2014], image captioning [Mao et al., 2015] and dialogue systems [Serban et al., 2015]. While it has been observed informally that the internal sentence representations of such models can reflect semantic intuitions [Cho et al., 2014], it is not known which architectures or objectives yield the ‘best’ or most useful representations. Resolving this question could ultimately have a significant impact on language processing systems. Indeed, it is phrases and sentences, rather than individual words, that encode the human-like general world knowledge (or ‘common sense’) [Norman, 1972] that is a critical missing part of most current language understanding systems.

We address this issue with a systematic comparison of cutting-edge methods for learning distributed representations of sentences. We constrain our comparison to methods that do not require labelled data gathered for the purpose of training models, since such methods are more cost-effective and applicable across languages and domains. We also propose two new phrase or sentence representation learning objectives - Sequential Denoising Autoencoders (SDAEs) and FastSent, a sentence-level log-linear bag-of-words model. We compare all methods on two types of task - supervised and unsupervised evaluations - reflecting different ways in which representations are ultimately to be used. In the former setting, a classifier or regression model is applied to representations and trained with task-specific labelled data, while in the latter, representation spaces are directly queried using cosine distance.

We observe notable differences in approaches depending on the nature of the evaluation metric. In particular, deeper or more complex models (which require greater time and resources to train) generally perform best in the supervised setting, whereas shallow log-linear models work best on unsupervised benchmarks. Specifically, SkipThought Vectors [Kiros et al., 2015] perform best on the majority of supervised evaluations, but SDAEs are the top performer on paraphrase identification. In contrast, on the (unsupervised) SICK sentence relatedness benchmark, FastSent, a simple, log-linear variant of the SkipThought objective, performs better than all other models. Interestingly, the method that exhibits strongest performance across both supervised and unsupervised benchmarks is a bag-of-words model trained to compose word embeddings using dictionary definitions [Hill et al., 2015a]. Taken together, these findings constitute valuable guidelines for the application of phrasal or sentential representation-learning to language understanding systems.

Distributed Sentence Representations

To constrain the analysis, we compare neural language models that compute sentence representations from unlabelled, naturally-ocurring data, as with the predominant methods for word representations.This excludes innovative supervised sentence-level architectures including [Socher et al., 2011, Kalchbrenner et al., 2014] and many others. Likewise, we do not focus on ‘bottom up’ models where phrase or sentence representations are built from fixed mathematical operations on word vectors (although we do consider a canonical case - see CBOW below); these were already compared by ?). Most space is devoted to our novel approaches, and we refer the reader to the original papers for more details of existing models.

SkipThought Vectors For consecutive sentences $S_{i-1},S_{i},S_{i+1}$ in some document, the SkipThought model [Kiros et al., 2015] is trained to predict target sentences $S_{i-1}$ and $S_{i+1}$ given source sentence $S_{i}$. As with all sequence-to-sequence models, in training the source sentence is ‘encoded’ by a Recurrent Neural Network (RNN) (with Gated Recurrent uUnits [Cho et al., 2014]) and then ‘decoded’ into the two target sentences in turn. Importantly, because RNNs employ a single set of update weights at each time-step, both the encoder and decoder are sensitive to the order of words in the source sentence.

For each position in a target sentence $S_{t}$, the decoder computes a softmax distribution over the model’s vocabulary. The cost of a training example is the sum of the negative log-likelihood of each correct word in the target sentences $S_{i-1}$ and $S_{i+1}$. This cost is backpropagated to train the encoder (and decoder), which, when trained, can map sequences of words to a single vector.

ParagraphVector ?) proposed two log-linear models of sentence representation. The DBOW model learns a vector $\mathbf{s}$ for every sentence $S$ in the training corpus which, together with word embeddings $v_{w}$, define a softmax distribution optimised to predict words $w\in S$ given $S$. The $v_{w}$ are shared across all sentences in the corpus. In the DM model, $k$-grams of consecutive words $\{w_{i}\dots w_{i+k}\in S\}$ are selected and $\mathbf{s}$ is combined with $\{v_{w_{i}}\dots v_{w_{i+k}}\}$ to make a softmax prediction (parameterised by additional weights) of $w_{i+k+1}$.

We used the Gensim implementation,https://radimrehurek.com/gensim/ treating each sentence in the training data as a ‘paragraph’ as suggested by the authors. During training, both DM and DBOW models store representations for every sentence (as well as word) in the training corpus. Even on large servers it was therefore only possible to train models with representation size $200$, and DM models whose combination operation was averaging (rather than concatenation).

Bottom-Up Methods We train CBOW and SkipGram word embeddings [Mikolov et al., 2013b] on the Books corpus, and compose by elementwise addition as proposed by ?).We also tried multiplication but this gave very poor results.

We also compare to C-PHRASE [Pham et al., 2015], an approach that exploits a (supervised) parser to infer distributed semantic representations based on a syntactic parse of sentences. C-PHRASE achieves state-of-the-art results for distributed representations on several evaluations used in this study.Since code for C-PHRASE is not publicly-available we use the available pre-trained model (http://clic.cimec.unitn.it/composes/cphrase-vectors.html). Note this model is trained on $3\times$ more text than others in this study.

Non-Distributed Baseline We implement a TFIDF BOW model in which the representation of sentence $S$ encodes the count in $S$ of a set of feature-words weighted by their tfidf in $C$, the corpus. The feature-words are the 200,000 most common words in $C$.

2 Models Trained on Structured Resources

The following models rely on (freely-available) data that has more structure than raw text.

DictRep ?) trained neural language models to map dictionary definitions to pre-trained word embeddings of the words defined by those definitions. They experimented with BOW and RNN (with LSTM) encoding architectures and variants in which the input word embeddings were either learned or pre-trained (+embs.) to match the target word embeddings. We implement their models using the available code and training data.https://www.cl.cam.ac.uk/~fh295/. Definitions from the training data matching those in the WordNet STS 2014 evaluation (used in this study) were excluded.

CaptionRep Using the same overall architecture, we trained (BOW and RNN) models to map captions in the COCO dataset [Chen et al., 2015] to pre-trained vector representations of images. The image representations were encoded by a deep convolutional network [Szegedy et al., 2014] trained on the ILSVRC 2014 object recognition task [Russakovsky et al., 2014]. Multi-modal distributed representations can be encoded by feeding test sentences forward through the trained model.

NMT We consider the sentence representations learned by neural MT models. These models have identical architecture to SkipThought, but are trained on sentence-aligned translated texts. We used a standard architecture [Cho et al., 2014] on all available En-Fr and En-De data from the 2015 Workshop on Statistical MT (WMT).www.statmt.org/wmt15/translation-task.html

3 Novel Text-Based Models

We introduce two new approaches designed to address certain limitations with the existing models.

Sequential (Denoising) Autoencoders The SkipThought objective requires training text with a coherent inter-sentence narrative, making it problematic to port to domains such as social media or artificial language generated from symbolic knowledge. To avoid this restriction, we experiment with a representation-learning objective based on denoising autoencoders (DAEs). In a DAE, high-dimensional input data is corrupted according to some noise function, and the model is trained to recover the original data from the corrupted version. As a result of this process, DAEs learn to represent the data in terms of features that explain its important factors of variation [Vincent et al., 2008]. Transforming data into DAE representations (as a ‘pre-training’ or initialisation step) gives more robust (supervised) classification performance in deep feedforward networks [Vincent et al., 2010].

The original DAEs were feedforward nets applied to (image) data of fixed size. Here, we adapt the approach to variable-length sentences by means of a noise function $N(S|p_{o},p_{x})$, determined by free parameters $p_{o},p_{x}\in$. First, for each word $w$ in $S$, $N$ deletes $w$ with (independent) probability $p_{o}$. Then, for each non-overlapping bigram $w_{i}w_{i+1}$ in $S$, $N$ swaps $w_{i}$ and $w_{i+1}$ with probability $p_{x}$. We then train the same LSTM-based encoder-decoder architecture as NMT, but with the denoising objective to predict (as target) the original source sentence $S$ given a corrupted version $N(S|p_{o},p_{x})$ (as source). The trained model can then encode novel word sequences into distributed representations. We call this model the Sequential Denoising Autoencoder (SDAE). Note that, unlike SkipThought, SDAEs can be trained on sets of sentences in arbitrary order.

We label the case with no noise (i.e. $p_{o}=p_{x}=0$ and $N\equiv id$) SAE. This setting matches the method applied to text classification tasks by ?). The ‘word dropout’ effect when $p_{o}\geq 0$ has also been used as a regulariser for deep nets in supervised language tasks [Iyyer et al., 2015], and for large $p_{x}$ the objective is similar to word-level ‘debagging’ [Sutskever et al., 2011]. For the SDAE, we tuned $p_{o}$, $p_{x}$ on the validation set (see Section 3.2).We searched $p_{o},p_{x}\in\{0.1,0.2,0.3\}$ and observed best results with $p_{o}=p_{x}=0.1$. We also tried a variant (+embs) in which words are represented by (fixed) pre-trained embeddings.

FastSent The performance of SkipThought vectors shows that rich sentence semantics can be inferred from the content of adjacent sentences. The model could be said to exploit a type of sentence-level Distributional Hypothesis [Harris, 1954, Polajnar et al., 2015]. Nevertheless, like many deep neural language models, SkipThought is very slow to train (see Table 1). FastSent is a simple additive (log-linear) sentence model designed to exploit the same signal, but at much lower computational expense. Given a BOW representation of some sentence in context, the model simply predicts adjacent sentences (also represented as BOW) .

More formally, FastSent learns a source $u_{w}$ and target $v_{w}$ embedding for each word in the model vocabulary. For a training example $S_{i-1},S_{i},S_{i+1}$ of consecutive sentences, $S_{i}$ is represented as the sum of its source embeddings $\mathbf{s_{i}}=\sum_{w\in S_{i}}u_{w}$. The cost of the example is then simply:

where $\phi(v_{1},v_{2})$ is the softmax function.

We also experiment with a variant (+AE) in which the encoded (source) representation must predict its own words as target in addition to those of adjacent sentences. Thus in FastSent+AE, (1) becomes

At test time the trained model (very quickly) encodes unseen word sequences into distributed representations with $\mathbf{s}=\sum_{w\in S}u_{w}$.

4 Training and Model Selection

Unless stated above, all models were trained on the Toronto Books Corpus,http://www.cs.toronto.edu/~mbweb/ which has the inter-sentential coherence required for SkipThought and FastSent. The corpus consists of 70m ordered sentences from over 7,000 books.

Specifications of the models are shown in Table 1. The log-linear models (SkipGram, CBOW, ParagraphVec and FastSent) were trained for one epoch on one CPU core. The representation dimension $d$ for these models was found after tuning $d\in\{100,200,300,400,500\}$ on the validation set.For ParagraphVec only $d\in\{100,200\}$ was possible due to the high memory footprint. All other models were trained on one GPU. The S(D)AE models were trained for one epoch ($\approx 8$ days). The SkipThought model was trained for two weeks, covering just under one epoch.Downloaded from https://github.com/ryankiros/skip-thoughts For CaptionRep and DictRep, performance was monitored on held-out training data and training was stopped after 24 hours after a plateau in cost. The NMT models were trained for 72 hours.

Evaluating Sentence Representations

In previous work, distributed representations of language were evaluated either by measuring the effect of adding representations as features in some classification task - supervised evaluation [Collobert et al., 2011, Mikolov et al., 2013a, Kiros et al., 2015] - or by comparing with human relatedness judgements - unspervised evaluation [Hill et al., 2015a, Baroni et al., 2014b, Levy et al., 2015]. The former setting reflects a scenario in which representations are used to inject general knowledge (sometimes considered as pre-training) into a supervised model. The latter pertains to applications in which the sentence representation space is used for direct comparisons, lookup or retrieval. Here, we apply and compare both evaluation paradigms.

Representations are applied to 6 sentence classification tasks: paraphrase identification (MSRP) [Dolan et al., 2004], movie review sentiment (MR) [Pang and Lee, 2005], product reviews (CR) [Hu and Liu, 2004], subjectivity classification (SUBJ) [Pang and Lee, 2004], opinion polarity (MPQA) [Wiebe et al., 2005] and question type classification (TREC) [Voorhees, 2002]. We follow the procedure (and code) of ?): a logistic regression classifier is trained on top of sentence representations, with 10-fold cross-validation used when a train-test split is not pre-defined.

2 Unsupervised Evaluations

We also measure how well representation spaces reflect human intuitions of the semantic sentence relatedness, by computing the cosine distance between vectors for the two sentences in each test pair, and correlating these distances with gold-standard human judgements. The SICK dataset [Marelli et al., 2014] consists of 10,000 pairs of sentences and relatedness judgements. The STS 2014 dataset [Agirre et al., 2014] consists of 3,750 pairs and ratings from six linguistic domains. Example ratings are shown in Table 2. All available pairs are used for testing apart from the 500 SICK ‘trial’ pairs, which are held-out for tuning hyperparameters (representation size of log-linear models, and noise parameters in SDAE). The optimal settings on this task are then applied to both supervised and unsupervised evaluations.

Results

Performance of the models on the supervised evaluations (grouped according to the data required by their objective) is shown in Table 3. Overall, SkipThought vectors perform best on three of the six evaluations, the BOW DictRep model with pre-trained word embeddings performs best on two, and the SDAE on one. SDAEs perform notably well on the paraphrasing task, going beyond SkipThought by three percentage points and approaching state-of-the-art performance of models designed specifically for the task [Ji and Eisenstein, 2013]. SDAE is also consistently better than SAE, which aligns with other findings that adding noise to AEs produces richer representations [Vincent et al., 2008].

Results on the unsupervised evaluations are shown in Table 4. The same DictRep model performs best on four of the six STS categories (and overall) and is joint-top performer on SICK. Of the models trained on raw text, simply adding CBOW word vectors works best on STS. The best performing raw text model on SICK is FastSent, which achieves almost identical performance to C-PHRASE’s state-of-the-art performance for a distributed model [Pham et al., 2015]. Further, it uses less than a third of the training text and does not require access to (supervised) syntactic representations for training. Together, the results of FastSent on the unsupervised evaluations and SkipThought on the supervised benchmarks provide strong support for the sentence-level distributional hypothesis: the context in which a sentence occurs provides valuable information about its semantics.

Across both unsupervised and supervised evaluations, the BOW DictRep with pre-trained word embeddings exhibits by some margin the most consistent performance. Ths robust performance suggests that DictRep representations may be particularly valuable when the ultimate application is non-specific or unknown, and confirms that dictionary definitions (where available) can be a powerful resource for representation learning.

Discussion

Many additional conclusions can be drawn from the results in Tables 3 and 4.

Different objectives yield different representations It may seem obvious, but the results confirm that different learning methods are preferable for different intended applications (and this variation appears greater than for word representations). For instance, it is perhaps unsurprising that SkipThought performs best on TREC because the labels in this dataset are determined by the language immediately following the represented question (i.e. the answer) [Voorhees, 2002]. Paraphrase detection, on the other hand, may be better served by a model that focused entirely on the content within a sentence, such as SDAEs. Similar variation can be observed in the unsupervised evaluations. For instance, the (multimodal) representations produced by the CaptionRep model do not perform particularly well apart from on the Image category of STS where they beat all other models, demonstrating a clear effect of the well-studied modality differences in representation learning [Bruni et al., 2014].

The nearest neighbours in Table 5 give a more concrete sense of the representation spaces. One notable difference is between (AE-style) models whose semantics come from within-sentence relationships (CBOW, SDAE, DictRep, ParagraphVec) and SkipThought/FastSent, which exploit the context around sentences. In the former case, nearby sentences generally have a high proportion of words in common, whereas for the latter it is the general concepts and/or function of the sentence that is similar, and word overlap is often minimal. Indeed, this may be a more important trait of FastSent than the marginal improvement on the SICK task. Readers can compare the CBOW and FastSent spaces at http://45.55.60.98/.

Differences between supervised and unsupervised performance Many of the best performing models on the supervised evaluations do not perform well in the unsupervised setting. In the SkipThought, S(D)AE and NMT models, the cost is computed based on a non-linear decoding of the internal sentence representations, so, as also observed by [Almahairi et al., 2015], the informative geometry of the representation space may not be reflected in a simple cosine distance. The log-linear models generally perform better in this unsupervised setting.

Differences in resource requirements As shown in Table 1, different models require different resources to train and use. This can limit their possible applications. For instance, while it was easy to make an online demo for fast querying of near neighbours in the CBOW and FastSent spaces, it was not practical for other models owing to memory footprint, encoding time and representation dimension.

The role of word order is unclear The average scores of models that are sensitive to word order (76.3) and of those that are not (76.6) are approximately the same across supervised evaluations. Across the unsupervised evaluations, however, BOW models score 0.55 on average compared with 0.42 for RNN-based (order sensitive) models. This seems at odds with the widely held view that word order plays an important role in determining the meaning of English sentences. One possibility is that order-critical sentences that cannot be disambiguated by a robust conceptual semantics (that could be encoded in distributed lexical representations) are in fact relatively rare. However, it is also plausible that current available evaluations do not adequately reflect order-dependent aspects of meaning (see below). This latter conjecture is supported by the comparatively strong performance of TFIDF BOW vectors, in which the effective lexical semantics are limited to simple relative frequencies.

The evaluations have limitations The internal consistency (Chronbach’s $\alpha$) of all evaluations considered together is $0.81$ (just above ‘acceptable’).wikipedia.org/wiki/Cronbach’s_alpha Table 6 shows that consistency is far higher (‘excellent’) when considering the supervised or unsupervised tasks as independent cohorts. This indicates that, with respect to common characteristics of sentence representations, the supervised and unsupervised benchmarks do indeed prioritise different properties. It is also interesting that, by this metric, the properties measured by MSRP and image-caption relatedness are the furthest removed from other evaluations in their respective cohorts.

While these consistency scores are a promising sign, they could also be symptomatic of a set of evaluations that are all limited in the same way. The inter-rater agreement is only reported for one of the 8 evaluations considered (MPQA, $0.72$ [Wiebe et al., 2005]), and for MR, SUBJ and TREC, each item is only rated by one or two annotators to maximise coverage. Table 2 illustrates why this may be an issue for the unsupervised evaluations; the notion of sentential ’relatedness’ seems very subjective. It should be emphasised, however, that the tasks considered in this study are all frequently used for evaluation, and, to our knowledge, there are no existing benchmarks that overcome these limitations.

Conclusion

Advances in deep learning algorithms, software and hardware mean that many architectures and objectives for learning distributed sentence representations from unlabelled data are now available to NLP researchers. We have presented the first (to our knowledge) systematic comparison of these methods. We showed notable variation in the performance of approaches across a range of evaluations. Among other conclusions, we found that the optimal approach depends critically on whether representations will be applied in supervised or unsupervised settings - in the latter case, fast, shallow BOW models can still achieve the best performance. Further, we proposed two new objectives, FastSent and Sequential Denoising Autoencoders, which perform particularly well on specific tasks (MSRP and SICK sentence relatedness respectively).We make all code for training and evaluating these new models publicly available, together with pre-trained models and an online demo of the FastSent sentence space. If the application is unknown, however, the best all round choice may be DictRep: learning a mapping of pre-trained word embeddings from the word-phrase signal in dictionary definitions. While we have focused on models using naturally-occurring training data, in future work we will also consider supervised architectures (including convolutional, recursive and character-level models), potentially training them on multiple supervised tasks as an alternative way to induce the ’general knowledge’ needed to give language technology the elusive human touch.

Acknowledgments

This work was supported by a Google Faculty Award to AK and FH and a Google European Doctoral Fellowship to FH. Thanks also to Marek Rei, Tamara Polajnar, Laural Rimell, Jamie Ryan Kiros and Piotr Bojanowski for helpful discussion and comments.

References