Diversity driven Attention Model for Query-based Abstractive Summarization

Introduction

Over the past few years neural models based on the encode-attend-decode Bahdanau et al. (2014) paradigm have shown great success in various natural language generation (NLG) tasks such as machine translation Bahdanau et al. (2014), abstractive summarization (Rush et al. (2015),Nallapati et al. (2016)) dialog Li et al. (2016), etc. One such NLG problem which has not received enough attention in the past is query based abstractive text summarization where the aim is to generate the summary of a document in the context of a query. In general, abstractive summarization, aims to cover all the salient points of a document in a compact and coherent fashion. On the other hand, query focused summarization highlights those points that are relevant in the context of the query. Thus given a document on “the super bowl”, the query “How was the half-time show?”, would result in a summary that would not cover the actual game itself.

Note that there has been some work on query based extractive summarization in the past where the aim is to simply extract the most salient sentence(s) from a document and treat these as a summary. There is no natural language generation involved. Since, we were interested in abstractive (as opposed to extractive) summarization we created a new dataset based on debatepedia. This dataset contains triplets of the form (query, document, summary). Further, each summary is abstractive and not extractive in the sense that the summary does not necessarily comprise of a sentence which is simply copied from the original document.

Using this dataset as a testbed, we focus on a recurring problem in models based on the encode-attend-decode paradigm. Specifically, it is observed that the summaries produced by such models contain repeated phrases. Table 1 shows a few such examples of summaries generated by such a model when trained on this new dataset. This problem has also been reported by Chen et al. (2016) in the context of summarization and by Sankaran et al. (2016) in the context of machine translation.

We first provide an intuitive explanation for this problem and then propose a solution for alleviating it. A typical encode-attend-decode model first computes a vectorial representation for the document and the query and then produces a contextual summary one word at a time. Each word is produced by feeding a new context vector to the decoder at each time step by attending to different parts of the document and query. If the decoder produces the same word or phrase repeatedly then it could mean that the context vectors fed to the decoder at these time steps are very similar.

We propose a model which explicitly prevents this by ensuring that successive context vectors are orthogonal to each other. Specifically, we subtract out any component that the current context vector has in the direction of the previous context vector. Notice that, we do not require the current context vector to be orthogonal to all previous context vectors but just its immediate predecessor. This enables the model to attend to words repeatedly if required later in the process. To account for the complete history (or all previous context vectors) we also propose an extension of this idea where we pass the sequence of context vectors through a LSTM Hochreiter and Schmidhuber (1997) and ensure that the current state produced by the LSTM is orthogonal to the history. At each time step, the state of the LSTM is then fed to the decoder to produce one word in the summary.

Our contributions can be summarized as follows: (i) We propose a new dataset for query based abstractive summarization and evaluate encode-attend-decode models on this dataset (ii) We study the problem of repeating phrases in NLG in the context of this dataset and propose two solutions for countering this problem. We show that our method outperforms a vanilla encoder-decoder model with a gain of 28% (absolute) in ROUGE-L score (iii) We also demonstrate that our method clearly outperforms a recent state of the art method proposed for handling the problem of repeating phrases with a gain of 7% (absolute) in ROUGE-L scores (iv) We do a qualitative analysis of the results and show that our model indeed produces outputs with fewer repetitions.

Related Work

Summarization has been studied in the context of text (Mani (2001), Das and Martins (2007), Nenkova and McKeown (2012)) as well as speech (Zhu and Penn (2006), Zhu et al. (2009)). A vast majority of this work has focused on extractive summarization where the idea is to construct a summary by selecting the most relevant sentences from the document (Neto et al. (2002), Erkan and Radev (2004), Filippova and Altun (2013), Colmenares et al. (2015), Riedhammer et al. (2010), Ribeiro et al. (2013)). There has been some work on abstractive summarization in the context of DUC-2003 and DUC-2004 contests Zajic et al. . We refer the reader to Das and Martins (2007) and Nenkova and McKeown (2012) for an excellent survey of the field.

Recent research in abstractive summarization has focused on data driven neural models based on the encode-attend-decode paradigm Bahdanau et al. (2014). For example, Rush et al. (2015), report state of the art results on the GigaWord and DUC corpus using such a model. Similarly, the work of Lopyrev (2015) uses neural networks to generate news headline from short news stories. Chopra et al. (2016) extend the work of Rush et al. (2015) and report further improvements on the two datasets. Hu et al. (2015) introduced a dataset for Chinese short text summarization and evaluated a similar RNN encoder-decoder model on it.

One recurring problem in encoder-decoder models for NLG is that they often repeat the same phrase/word multiple times in the summary (at the cost of both coherency and fluency). Sankaran et al. (2016) study this problem in the context of MT and propose a temporal attention model which enforces the attention weights for successive time steps to be different from each other. Similarly, and more relevant to this work, Chen et al. (2016) propose a distraction based attention model which maintains a history of attention vectors and context vectors. It then subtracts this history from the current attention and context vector. When evaluated on our dataset their method performs poorly. This could be because their method is very aggressive in dealing with the history (as explained later in the Experiments section). On the other hand, our method has a better way of handling history (by passing context vectors through an LSTM recurrent network) which gives us the flexibility to forget/retain some portions of the history and at the same time produce diverse context vectors at successive time steps.

We evaluate our method in the context of query based abstractive summarization - a problem which has received almost no attention in the past due to unavailability of datasets. We create a new dataset for this task and show that our method indeed produces better output by reducing the number of repeated phrases produced by encoder decoder models.

Dataset

As mentioned earlier, there are no existing datasets for query based abstractive summarization. We create such a dataset from Debatepedia an encyclopedia of pro and con arguments and quotes on critical debate topics. There are 663 debates in the corpus (we have considered only those debates which have at least one query with one document). These 663 debates belong to 53 overlapping categories such as Politics, Law, Crime, Environment, Health, Morality, Religion, etc. A given topic can belong to more than one category. For example, the topic “Eye for an Eye philosophy” belongs to both “Law” as well as “Morality”. The average number of queries per debate is 5 and the average number of documents per query is 4. Please refer to the dataset urlhttps://github.com/PrekshaNema25/DiverstiyBasedAttentionMechanism for more details about number of debates per category.

For example, Figure 2 shows the queries associated with the topic “Algae Biofuel”. It also lists the set of documents and an abstractive summary associated with each query. As is obvious from the example, the summary is an abstractive summary and not extracted directly from the document. We crawled 12695 such {query, document, summary} triples from debatepedia (these were all the triples that were available). Table 2 reports the average length of the query, summary and documents in this dataset.

We used 10 fold cross validation for all our experiments. Each fold uses 80% of the documents for training, 10% for validation and 10% for testing.

Proposed model

Given a query $\mathbf{q}=q_{1},q_{2},...,q_{k}$ containing $k$ words, a document $\mathbf{d}=d_{1},d_{2},...,d_{n}$ containing $n$ words, the task is to generate a contextual summary $\mathbf{y}=y_{1},y_{2},...,y_{m}$ containing $m$ words. This can be modeled as the problem of finding a $\mathbf{y}^{*}$ that maximizes the probability $p(\mathbf{y}|\mathbf{q},\mathbf{d})$ which can be further decomposed as:

We now describe a way of modeling $p(y_{t}|y_{1},...,y_{t-1},\mathbf{q},\mathbf{d})$ using the neural encoder-attention-decoder paradigm. The proposed model contains the following components: (i) an encoder RNN for the query (ii) an encoder RNN for the document (iii) attention mechanism for the query (iv) attention mechanism for the document and (v) a decoder RNN. All the RNNs use a GRU cell.

Encoder for the query: We use a recurrent neural network with Gated Recurrent Units (GRU) for encoding the query. It reads the query $\mathbf{q}=q_{1},q_{2},...,q_{k}$ from left to right and computes a hidden representation for each time-step as:

Encoder for the document: This is similar to the query encoder and reads the document $\mathbf{d}=d_{1},d_{2},...,d_{n}$ from left to right and computes a hidden representation for each time-step as:

Attention mechanism for the query : At each time step, the decoder produces an output word by focusing on different portions of the query (document) with the help of a query (document) attention model. We first describe the query attention model which assigns weights $\alpha_{t,i}^{q}$ to each word in the query at each decoder timestep using the following equations.

Attention mechanism for the document : We now describe the document attention model which assigns weights to each word in the document using the following equations.

Note that $d_{t}$ now encodes the relevant information from the document as well as the query (see Equation (7)) at time step $t$. We refer to this as the context vector for the decoder.

Decoder: The hidden state of the decoder $s_{t}$ at each time $t$ is again computed using a GRU as follows:

where, $y_{t-1}$ gives a distribution over the vocabulary words at timestep $t-1$ and is computed as:

The model as described above is an instantiation of the encoder-attention-decoder idea applied to query based abstractive summarization. As mentioned earlier (and demonstrated later through experiments), this model suffers from the problem of repeating the same phrase/word in the output. We now propose a new attention model which we refer to as diversity based attention model to address this problem.

As hypothesized earlier, if the decoder produces the same phrase/word multiple times then it is possible that the context vectors being fed to the decoder at consecutive time steps are very similar. We propose four models ($\mathbf{D_{1}}$, $\mathbf{D_{2}}$, $\mathbf{SD_{1}}$, $\mathbf{SD_{2}}$) to directly address this problem.

$\mathbf{D_{1}}$: In this model, after computing $d_{t}$ as described in Equation (8), we make it orthogonal to the context vector at time $t-1$:

$\mathbf{SD_{1}}$: The above model imposes a hard orthogonality constraint on the context vector($d_{t}^{{}^{\prime}}$). We also propose a relaxed version of the above model which uses a gating parameter. This gating parameter decides what fraction of the previous context vector should be subtracted from the current context vector using the following equations:

$\mathbf{D_{2}}$: The above model only ensures that the current context vector is diverse w.r.t the previous context vector. It ignores all history before time step $t-1$. To account for the history, we treat successive context vectors as a sequence and use a modified LSTM cell to compute the new state at each time step. Specifically, we use the following set of equations to compute a diverse context at time $t$:

$\mathbf{SD_{2}}$: This model again uses a relaxed version of the orthogonality constraint used in $\mathbf{D_{2}}$. Specifically, we define a gating parameter $g_{t}$ and replace (4.1) above by (4.1) as define below:

Baseline Methods

We compare with two recently proposed baseline diversity methods Chen et al. (2016) as described below. Note that these methods were proposed in the context of abstractive summarization (not query based abstractive summarization) and we adapt them for the task of query based abstractive summarization. Below we just highlight the key differences from our model in computing the context vector $d_{t}^{{}^{\prime}}$ passed to the decoder.

M1: This model accumulates all the previous context vectors as $\sum_{j=1}^{t-1}d_{j}^{{}^{\prime}}$ and incorporates this history while computing a diverse context vector:

M2: In this model, in addition to computing a diverse context as described in Equation (15), the attention weights at each time step are also forced to be diverse from the attention weights at the previous time step.

Experimental Setup

We evaluate our models on the dataset described in section 3. Note that there are no prior baselines on query based abstractive summarization so we could only compare with different variations of the encoder decoder models as described above. Further, we compare our diversity based attention models with existing models for diversity by suitably adapting them to this problem as described earlier. Specifically, we compare the performance of the following models:

Vanilla e-a-d: This is the vanilla encoder-attention-decoder model adapted to the problem of abstractive summarization. It contains the following components (i) document encoder (ii) document attention model (iii) decoder. It does not contain an encoder or attention model for the query. This helps us understand the importance of the query.

$\mathbf{Query}_{enc}$: This model contains the query encoder in addition to the three components used in the vanilla model above. It does not contain any attention model for the query.

$\mathbf{Query}_{att}$: This model contains the query attention model in addition to all the components in $Query_{enc}$.

$\mathbf{D}_{1}$: The diversity attention model as described in Section 4.1.

$\mathbf{D}_{2}$: The LSTM based diversity attention model as described in Section 4.1.

$\mathbf{SD}_{1}$: The soft diversity attention model as described in Section 4.1

$\mathbf{SD}_{2}$: The soft LSTM based diversity attention model as described in Section 4.1

$\mathbf{B}_{1}$: Diversity cell in Figure3 is replaced by the basic LSTM cell (i.e. $c_{t}^{diverse}=c_{t}$ instead of using Equation (4.1). This helps us understand whether simply using an LSTM to track the history of context vectors (without imposing a diversity constraint) is sufficient.

$\mathbf{M}_{1}$: The baseline model which operates on the context vector as described in Section 5.

$\mathbf{M}_{2}$: The baseline model which operates on the attention weights in addition to the context vector as described in Section 5.

We used 80% of the data for training, 10% for validation and 10% for testing. We create 10 such folds and report the average Rouge-1, Rouge-2, Rouge-L scores across the 10 folds. The hyperparameters (batch size and GRU cell sizes) of all the models are tuned on the validation set. We tried the following batch sizes : 32, 64 and the following GRU cell sizes 200, 300, 400. We used Adam Kingma and Ba (2014) as the optimization algorithm with the initial learning rate set to 0.0004, $\beta_{1}=0.9$, $\beta_{2}=0.999$. We used pre-trained publicly available Glove word embeddingshttp://nlp.stanford.edu/projects/glove/ and fine-tuned them during training. The same word embeddings are used for the query words and the document words.

Table 3 summarizes the results of our experiments.

Discussions

In this section, we discuss the results of the experiments reported in Table 3. 1. Effect of Query: Comparing rows 1 and 2 we observe that adding an encoder for the query and allowing it to influence the outputs of the decoder indeed improves the performance. This is expected as the query contains some keywords which could help in sharpening the focus of the summary.

2. Effect of Query attention model: Comparing rows 2 and 3 we observe that using an attention model to dynamically compute the query representation at each time step improves the results. This suggests that the attention model indeed learns to focus on relevant portions of the query at different time steps.

3. Effect of Diversity models: All the diversity models introduced in the paper (rows 7, 8, 9, 10) give significant improvement over the non-diversity models. In particular, the modified LSTM based diversity model gives the best results. This is indeed very encouraging and Table 4 shows some sample summaries comparing the performance of different models.

4. Comparison with baseline diversity models: The baseline diversity model M1 performs at par with our models D1 and SD1 but not as good as D2 and SD2. However, the model M2 performs very poorly. We believe that simultaneously adding a constraint on the context vectors as well as attention weights (as is indeed the case with M2) is a bit too aggressive and leads to poor performance (although this needs further investigation).

5. Quantitative Analysis: In addition to the qualitative analysis reported in Table 4 we also did a quantitative analysis by counting the number of sentences containing repeated words generated by different models. Specifically for the 1268 test instances we counted the number of sentences containing repeated words as generated by different modes. Table 5 summarizes this analysis.

Conclusion

In this work we proposed a query-based summarization method. The unique feature of the model is a novel diversification mechanism based on successive orthogonalization. This gives us the flexibility to: (i) provide diverse context vectors at successive time steps and (ii) pay attention to words repeatedly if need be later in the summary (as opposed to existing models which aggressively delete the history). We also introduced a new data set and empirically verified we perform significantly better (gain of 28% (absolute) in ROUGE-L score) than applying a plain encode-attend-decode mechanism to this problem. We observe that adding an attention mechanism on the query string gives significant improvements. We also compare with a state of the art diversity model and outperform it by a good margin (gain of 7% (absolute) in ROUGE-L score). The diversification model proposed is general enough to apply to other NLG tasks with suitable modifications and we are currently working on extending this to dialog systems and general summarization.

References