Modelling Sentence Pairs with Tree-structured Attentive Encoder

Introduction

Modelling a sentence pair is to score two pieces of sentences in terms of their semantic relationship. The applications include measuring the semantic relatedness of two sentences [Marelli et al., 2014], recognizing the textual entailment [Bowman et al., 2015] between the premise and hypothesis sentences, paraphrase identification [He et al., 2015], answer selection and query ranking [Yin et al., 2015] etc.

The approach of modelling a sentence pair based on neural networks usually consist of two steps. First, a sentence encoder transforms each sentence into a vector representation. Second, a classifier receives two sentence representations as features to make the classification. The sentence encoder can be regarded as a semantic compositional function which maps a sequence of word vectors to a sentence vector. This compositional function takes a range of different forms, including (but not limited to) sequential recurrent neural networks (Seq-RNNs) [Mikolov, 2012], tree-structured recursive neural networks (Tree-RNNs) [Socher et al., 2014, Tai et al., 2015] and convolutional neural networks (CNNs) [Kim, 2014].

We introduce an approach that combines recursive neural networks and recurrent neural networks with the attention mechanism, which has been widely used in the sequence to sequence learning (seq2seq) framework whose applications ranges from machine translation [Bahdanau et al., 2015, Luong et al., 2015], text summarization [Rush et al., 2015] to natural language conversation [Shang et al., 2015] and other NLP tasks such as question answering [Sukhbaatar et al., 2015, Hermann et al., 2015], classification [Rocktäschel et al., 2016, Shimaoka et al., 2016]. In the machine translation, the attention mechanism is used to learn the alignments between source words and target words in the decoding phase. More generally, we consider that the motivation of attention mechanism is to allow the model to attend over a set of elements with the intention of attaching different emphases to each element. We argue that the attention mechanism used in a tree-structured model is different from a sequential model. Our idea is inspired by Rocktäschel et al. [Rocktäschel et al., 2016] and Hermann et al. [Hermann et al., 2015]. In this paper, we utilise the attention mechanism to select semantically more relevant child by the representation of one sentence learned by a Seq-RNNs, when constructing the head representation of the other sentence in the pair on a dependency tree. Since our model adopts the attention in the sentence encoding phase, we refer to it as an attentive encoder. In this work, we implement this attentive encoder with two architectures: tree-structured LSTM and tree-structured GRU.

We evaluate the proposed encoder on three sentence pair modelling tasks: semantic similarity on the SICK dataset, paraphrase identification on the MSRP dataset and true-false question selection on the AI2-8grade science questions dataset. Experimental results demonstrate that our attentive encoder is able to outperform all non-attentional counterparts and achieves the state-of-the-art performance on the SICK dataset and AI2-8grade dataset.

Models

Let’s begin with a high-level discussion of our tree-structured attentive encoder. As shown in Figure 1, given a sentence pair ($S^{a}$, $S^{b}$), our goal is to score this sentence pair. Our tree-structured attentive model has two components. In the first component, a pair of sentences is fed to a Seq-RNNs, which encodes each sentence and results in a pair of sentence representations. In second component, the Attentive Tree-RNNs encodes a sentence again, aimed by the representation of the other sentence generated by the first component. Compared with the existing approaches of modelling sentence pairs, our attentive encoder consider not only the sentence itself but also the other sentence in the pair. Finally, the two sentence vectors produced by the second component are fed to the multilayer perceptron network to produce a distribution over possible values. These components will be detailed in the following sections.

We first describe the RNN composer, which is the basic unit of Seq-RNNs. Given an input sequence of arbitrary length, an RNN composer iteratively computes a hidden state $h_{t}$ using the input vector $x_{t}$ and its previous hidden state $h_{t-1}$. In this paper, the input vector $x_{t}$ is a word vector of the $t$-th word in a sentence. The hidden state $h_{t}$ can be interpreted as a distributed representation of the sequence of tokens observed up to time $t$. Commonly, the RNN transition function is the following:

We refer to the model that recursively apply the RNN composer to a sequence as the Seq-RNNs. Unfortunately, standard Seq-RNNs suffers from the problem that the gradients of the hidden states of earlier part of the sequence vanishes in long sequences [Hochreiter, 1998]. Long Short-term Memory (LSTM) [Hochreiter and Schmidhuber, 1997] and Gated Recurrent Unit (GRU) [Chung et al., 2014] are two powerful and popular architectures that address this problem by introducing gates and memory. In this paper, we only show the illustrations of LSTM (Figure 2(a)) and GRU (Figure 2(d)). The implementations of standard LSTM and GRU in this paper are same as [Luong et al., 2015] and [Chung et al., 2014]. When we replace the standard RNN composer with LSTM or GRU, the Seq-RNNs becomes Seq-LSTMs or Seq-GRUs.

2 Standard Tree-RNNs

Compared with standard RNN composer, which computes its hidden state from the input at the current time step and the hidden state of previous time step, the Tree-RNN composer computes its hidden state from an input and the hidden states of arbitrarily many child units. We now describe the Child-Sum Tree-LSTM and Child-Sum Tree-GRU architectures which are formed by applying the Child-Sum algorithm to LSTM and GRU respectively.

Child-Sum Tree-GRU.

where $\sigma$ denotes the sigmoid function and $\odot$ denotes element-wise multiplication.

We can easily apply the Child-Sum Tree-RNN to the dependency trees that have branching factors of arbitrary number and order-insensitive nodes. We refer to the model adopting the Tree-LSTM and Tree-GRU composer to the dependency tree as the Dependency Tree-LSTMs and Dependency Tree-GRUs. For simplicity, we also omit the prefix “Dependency” in the following sections.

3 Attentive Tree-RNNs

We now details how we extend the standard Tree-RNN. The idea that we incorporate the attention into the standard Tree-RNN comes from: (1) there will be semantic relevance between two sentences in the sentence pair modelling tasks; (2) the effect of semantic relevance could be implemented in the process of constructing the sentence representation by Tree-RNN where each child should be assigned a different weight; and (3) the attention mechanism is well suited for learning weights on a contextual collection where a guided vector is attending over.

In this work, the attention mechanism is implemented by a soft attention layer $A$. Given a collection of hidden states $h_{1},h_{2},\dots,h_{n}$ and an external vector $s$, the soft attention layer produce a weight $\alpha_{k}$ for each hidden state as well as a weighted vector $g$ via the Equations 13:

Attentive Tree-LSTM and -GRU

4 MLP

The multilayer perceptron network (MLP) receives a pair of vectors produced by the sentence encoder to compute a multinomial distribution over possible values. Given two sentence representations $h_{L}$ and $h_{R}$, we compute their componentwise product $h_{L}\odot h_{R}$ and their absolute difference $|h_{L}-h_{R}|$. These features are also used by Tai et al.[Tai et al., 2015]. We then compress these features into a low dimensional vector $h_{s}$, which is used to compute the probability distribution $\hat{p}_{\theta}$. The equations are the following:

Experiments and Results

In order to make a meaningful comparison between the sequential models, tree-structured models and attentive models, we present four baselines. They are: (i) Seq-LSTMs, learning two sentence representations by the sequential LSTMs; (ii) Seq-GRUs, like Seq-LSTMs but using GRU composer; (iii) Tree-LSTMs, learning two sentence representations by the Dependency Tree-LSTMs; and (iv) Tree-GRUs, like Tree-LSTMs but using Child-Sum Tree-GRU composer. The two sentence representations are fed to the MLP to produce a probability distribution.

1 Task 1: Semantic Similarity

First we conduct our semantic similarity experiment on the Sentences Involving Compositional Knowledge(SICK) dataset [Marelli et al., 2014]Dependency trees are parsed by the Stanford Parser package, http://nlp.stanford.edu/software/lex-parser.htmlGlove vectors are available at http://nlp.stanford.edu/projects/glove/. This task is to predict a similarity score of a pair of sentences, based on human generated scores. The SICK dataset consists of 9927 sentence pairs with the split of 4500 training pairs, 500 development pairs and 4927 testing pairs. Each sentence pair is annotated with a similarity score ranging from 1 to 5. A high score indicates that the sentence pair is highly related. All sentences are derived from existing image and video annotation dataset. The evaluation metrics are Pearson’s $r$, Spearman’s $\rho$ and mean squared error (MSE).

Recall that the output of MLP (Section 2.4) is a probability distribution $\hat{p}_{\theta}$. Our goal in this task is to predict a similarity score of two sentences. Let $r^{\intercal}=[1,\dots,5]$ be an integer vector, the similarity score $\hat{y}$ is computed by $\hat{y}=r^{\intercal}\hat{p}_{\theta}$. We take the same setup as [Tai et al., 2015] that computes a target distribution $p$ as a function of prediction score $y$ given by:

The loss function of semantic similarity is the KL-divergence that measures the continuous distance between the predicted distribution $\hat{p}_{\theta}$ and the distribution of the ground truth $p$:

The results are summarized in Table 2. We first compare our results against the previous results. ECNU [Zhao et al., 2014], the best result of SemEval 2014 submissions, achieves a 0.8414 $r$ score by a heavily feature-engineered approach. Kiros et al. [Kiros et al., 2015] presents an unsupervised approach to learn the universal sentence vectors without depending on a specific task. Their Combine–skip+COCO model improve the Pearson’s $r$ to 0.8655, but a weakness is that their sentence vectors are high-dimensional vectors (2400D). Training the skip-thoughts vectors needs a lot of time and space. He et al. [He et al., 2015] show the effectiveness of convolutional nets with the similarity measurement layer for modelling sentence similarity. Their ConvNet outperforms ECNU with +0.027 Pearson’s $r$. We can observe that dependency Tree-LSTM, combine-skip+COCO and ConvNet almost achieve the same performance and our Attentive Tree-LSTMs outperforms these three methods around +0.005 points. Comparison to ECNU, our Attentive Tree-LSTMs gains an improvement of +0.032 and achieves the state-of-the-art performance. We find a phenomenon also appeared in [Tai et al., 2015] that tree-structured models can outperform sequential counterparts. Comparison to the non-attentional baselines (such as Tree-LSTMs), the attention mechanism (such as Attentive Tree-LSTMs) gives us a boost of around +0.007. All results highlight that our attentive Tree-RNNs are well suited for the semantic similarity task.

2 Task 2: Paraphrase Identification

The next task we evaluate is paraphrase identification on the Microsoft Research Paraphrase Corpus (MSRP) [Dolan et al., 2004]. Given two sentences, this task is to predict whether or not they are paraphrases. The dataset is collected from news sources and contains 5801 pairs of sentences, with 4076 for training and the remaining 1725 for testing. We randomly select 10% of training set and use them as our dev set. This task is a binary classification task, therefore we report the accuracy and F1 score.

Since that the $\hat{p}_{\theta}$ indicates the distribution over the possible labels, we take $argmax(\hat{p}_{\theta})$ as the predicted label in the testing phase. The loss function for the binary classification is the binary cross-entropy:

Table 3 (left) presents our results on the MSRP dataset. The previous approaches are: (1) Baseline, cosine similarity with tf-idf weighting; (2) RAE, recursive autoencoder with dynamic pooling; (3) combine-skip+feats, skip-thought vectors with features; (4) ABCNN-3, attention-based convolutional nets; and (5) TF-KLD, matrix factorization with supervised reweighting. First, all our models are able to outperform the baseline. We only compare our models with the neural networks-based approaches, including RAE and ABCNN-3 for a fair comparison. We find that our models do not prove to be very competitive. After a careful analysis, we conclude that the reasons are (1) our models are pure neural networks-based, we don’t add any features to identify paraphrases while the other methods have used additional features; (2) The MLP is not very suitable in this task. We attempt to replace the MLP with the cosine distance and euclidean distance in our future work. Although our models have not yet matched the SOTA performance, we obtain an improvement of +2.3 accuracy by Attentive Tree-LSTMs when we incorporate the attention into the standard Tree-LSTM.

3 Task 3: True-False Question Selection

We last consider a challenging task: selecting true or false given a scientific question and its evidence. In this task, we use the AI2-8grade dataset built by [Baudis et al., 2016]. This dataset is derived from the AI2 Elementary School Science Questions released by Allen Institute. Each sentence pair consists of a hypothesis sentence processed by substituting the $wh$-word in the question by answer and its evidence sentence extracted from a collection of CK12 textbooks. The number of sample pairs in the training, development, and test set are 12689, 2483 and 11359 respectively. This dataset contains 626 words not appearing in Glove vectors, most of which are named entities and scientific jargons.

The loss function is the same as the paraphrase identification since this task is also a binary classification task. We reports the accuracy on development set and test set shown in Table 3 (right). Since this dataset is a fresh and uncompleted dataset, we only compare our models with Baudis et al. [Baudis et al., 2016] who have evaluated several models on it. Comparison to [Baudis et al., 2016], all of our models gain a significant improvement. Specially, our best result achieved by the Attentive Tree-LSTMs is higher than the best of [Baudis et al., 2016] by +28 percents. It is observed that tree-structured models are more competitive than the sequential counterparts. As we expected, the attentive models can outperform all non-attentional counterparts.

Quantitative Analysis

Table 4 presents example predictions that are produced by our Attentive Tree-LSTMs. The first group shows that our model is able to predict semantic similarity score nearly perfectly on the SICK dataset. We argue the reason is that the sentences of SICK dataset are image and video descriptions whose sentence structure is relatively simple and there are less uncommon words and named entities in the vocabulary. The second group gives us three examples on the MSRP test set. We find that our model can identify whether two fact statements are paraphrases, but fails to recognize the numbers (in group 2, line 3). We presents the examples on AI2-8grade dataset in the last group. We can observe that our model is efficient to select the false questions, while our model is difficult to select the true answers, unless the evidence of question is very strong.

Effect of Sentence Length

In order to analyse the effect of mean sentence length on the SICK dataset, we draw the Figure 3(a). We observe that the Pearson score become lower as sentence become longer. Compared with the Seq-RNNs, the Tree-RNNs obtain a little improvements. Specially, the Attentive Tree-GRUs proves to be more effective than Tree-GRUs when the mean sentence length reaches to 20.

Effect of N𝑁N-grams

In the MSR paraphrase corpus, a hypothesis is that two sentence tend to be paraphrases when the value of their $n$-gram overlap is high. As a result we present the Figure 3(b), x-axis is the normalized $n$-grams overlap whose value is computed by $c*\frac{(unigram+bigram+trigram)}{mean\_sent\_length}$, where $c$ equals to 50, and y-axis is the accuracy. We can observe that the Attentive Tree-GRUs are more effective than Tree-GRUs when the value of normalized $n$-grams overalp is less than 40. The results suggest that our attentive models are more general.

Attention Visualization

It is instructive to analyse which child the attentive model is attending over when constructing the head representation. We visualize the heatmaps of attention weights shown in Figure 4. The words at x-axis are modified by the words at y-axis with a weight (greater than zero). For example in Figure 4, the 5th word at x-axis is “playing” whose children are “boy”, “outdoors”, “and” and “is”. We can observe that the word “boy” holds a higher weight among all the modifiers. It means that the branch rooted with “boy” contributes more when constructing the representation of subtree whose root node is “playing”. This phenomenon is very reasonable because the sentence is describing a image of “a boy is playing something”.

Conclusion

In this paper, we introduced a way of incorporating attention into the Child-Sum Tree-LSTM and Tree-GRU that can be applied to the dependency tree. We evaluate the proposed models on three sentence pair modelling tasks and achieve state-of-the-art performance on two of them. Experiment results show that our attentive models are effective for modelling sentence pairs and can outperform all non-attentional counterparts. In the future, we will evaluate our models on the other sentence pair modelling tasks (such as RTE) and extend them to the $seq2seq$ learning framework.

Acknowledgements

This work was funded in part by the National Key Research and Development Program of China (2016YFB0201900), the National Science Foundation of China (grant 61472459, 61370021, U1401256, 61472453), Natural Science Foundation of Guangdong Province under Grant S2013010011905.

References