Query-Based Abstractive Summarization Using Neural Networks

Introduction

Creating short summaries of documents with respect to a query has applications in for example search engines, where it may help inform users of the most relevant results. However, constructing such a summary automatically is a difficult problem yet to be fully solved. In this paper, a neural network model for this task is presented. More specifically, the model is designed for brief, commonly single-sentence, summaries. A situation where this may be useful is when a user has performed a search in a search engine and a set of documents have been returned. Concise summaries could then be displayed along with the search results, giving a quick overview of how the document is related to the search query. What is commonly done in search engines today is that text surrounding an occurrence of a search query in the document is displayed as a summary. This is an example of extractive summarization, which produces a summary that only contains parts of the original document. A significant difference in the model we present is that it generates an abstractive summary. This type of summary allows for rephrasing and using words not necessarily present in the original document, comparable to a human-written summary. This has the potential of summarizing documents in a more concise way than what is possible with an extractive summary, i.e. making it easier for a reader to understand the relationship between a document and a query.

Automatic text summarization has been a research topic for many years. In general, the goal is to concisely represent the most important information in documents. Much previous work in summarization has been using extractive methods Nenkova and McKeown (2012); Mogren et al. (2015). Commonly, individual sentences are extracted and composed together to form a summary. This gives sentences that are as grammatically correct as the source document. They are however inherently limited, and cannot reproduce human-written summaries in general. Abstractive summarization in particular is closely related to natural language generation, and it would be desirable to reach human-level performance in writing summaries. It may however require human-level understanding of the context of documents to produce results comparable to human-written ones. An important progress in using neural network models for generating text is sequence-to-sequence, used by Sutskever et al. (2014) for machine translation. It is a way of mapping a varying-length input text to a varying-length output text, and it is applicable to machine translation as well as summarization. In recent years, progress has been made on using neural network models for text summarization and similar problems. Some examples are sequence-to-sequence models for non-query-based abstractive summarization by Rush et al. (2015) and Nallapati et al. (2016). Neural network models have additionally been used for generating image captions Karpathy and Fei-Fei (2015), which is a form of summary, and for question answering problems, such as by Hermann et al. (2015) and Tan et al. (2015). Inspired by this progress, we designed a model for query-based summarization using neural networks.

The main contributions of this work includes: (1) A model for query-based abstractive summarization, presented in Section 3. (2) A dataset for query-based abstractive summarization, created by adapting an existing dataset originally used for question answering, described further in Section 4. (3) A quantitative evaluation of the performance of the proposed model compared with an extractive baseline and an uninformed abstractive model, presented in Section 6. (4) A qualitative analysis of the generated summaries.

An early work evaluating several methods for extractive query-based summarization is presented by Goldstein et al. (1999). Besides "full queries", they use "short queries", which on average are 3.9 words. These are similar in length to the types of queries used in the experiments of this thesis work. Besides the work by Otterbacher et al. (2009), recent work in query-based summarization has been done by Wang et al. (2013), using parse trees and sentence compression. It is described as not "pure extractive summarization". During the later stages of this thesis work, Nema et al. (2017) propose a neural network model for query-based abstractive summarization, which has some similarities to the model we present. However, the dataset they use is smaller in both average document length and number of documents. Additionally, the types of queries used are different, in that they use complete questions as opposed to our single-entity queries.

The task of question answering is to produce an answer to a question posed in natural language. The task is very general and many other problems can be expressed as a question-answering problem. Summarizing with respect to a query may for instance be expressed as "What is a summary of the document with respect to the query X?", for the query X. If the answer to a question is a single complete sentence, then it is especially close to the types of query-based summaries considered in this thesis. Otterbacher et al. (2009) present a model, Biased LexRank, which they use for a form of question answering as well as extractive query-based summarization. The answers they generate are full sentences, which makes it similar to our task of query-based summarization. Hermann et al. (2015) present neural network models for question answering. For training these, they create a large dataset from CNN/Daily Mail news articles. We adapt this dataset for query-based summarization, as detailed in Chapter 4. Kumar et al. (2016) introduce Dynamic Memory Networks, which they show reached state-of-the-art performance in a variety of NLP tasks. We draw inspiration from their use of a question module when we incorporate query information in our model.

General abstractive summarization differs from query-based summarization in that a document is summarized without respect to a query. Nallapati et al. (2016) build upon a machine translation model by Bahdanau et al. (2015) and generate general abstractive summaries on multiple datasets, including the CNN/Daily Mail dataset by Hermann et al. (2015). Additions they make for their model include a pointer-generator mechanism Gülçehre et al. (2016) that allows the model to copy words from the source document. See et al. (2017) propose a similar model, using a similar pointer-generator mechanism, that outperforms Nallapati et al. (2016) on a slightly different version of the CNN/Daily Mail dataset (making the result not "strictly comparable"). They also incorporate what they call coverage for avoiding repetitions in the output.

Background

In the following sections, various terms and concepts used throughout the paper are explained.

Information extraction is a class of tasks that involve extracting structured information from documents. An example of such a task is named entity recognition, which is the classification of parts of text into different categories, such as persons or locations, or no category. An example from the sentence "The mathematician Jeff Paris visited the city of Paris." is that "Jeff Paris" should be annotated as a person, and the last "Paris" as a location.

2 Gated Recurrent Units

The gated recurrent unit (GRU) is a type of recurrent neural network (RNN) that is designed to alleviate the vanishing/exploding gradient problem Hochreiter (1991); Bengio et al. (1994) which hinders the original RNN from capturing long term dependencies. GRU is similar to the popular long short-term memory (LSTM) model but is simpler and less computationally intensive, while still achieving comparable results on many tasks Chung et al. (2014); Kumar et al. (2016). The entire GRU architecture can be described by the formulas

The vectors $x_{t}$ is the input at time step $t$, and $h_{t}$ is the output, while $r_{t}$ and $z_{t}$ are scaling vectors, intended to regulate what information is let through. These can be described as gates. They have elements in $$. The vector$h^{\prime}_{t}$is rather intended to carry data. Its elements are in$$, generated from a network with a$\tanh$ activation function. We denote an entire GRU update step as

3 Word Embeddings

Given a vocabulary $V$, we can encode each word uniquely using a one-hot encoding. This gives a vector of length $|V|$ where every word in the vocabulary is mapped uniquely to some dimension, which a value of 1, while the other dimensions are 0. This vector can be transformed to an embedding for the word by multiplying it by an embedding matrix $W_{\text{emb}}$ of dimensionality $d_{\text{emb}}\times|V|$, where $d_{\text{emb}}$ is the word embedding dimensionality, commonly a hyperparameter in neural network models. The intention is that the embeddings capture some characteristics of words, giving useful vector representations. For instance, two related words such as football and soccer may be expected to be close to each other in the vector space. Two methods for generating word embeddings are word2vec Mikolov et al. (2013) and GloVe Pennington et al. (2014).

4 Attention

For many problems, it has been found to be beneficial to use more of the RNN states than the final fixed-size hidden state. Attention is a mechanism for allowing the model to access more information in the decoding process, by letting it identify relevant parts of the input and use the encoder hidden state at these locations. This technique has been used successfully for machine translation Bahdanau et al. (2015) and image captioning Xu et al. (2015).

Model

The document encoder processes an input document, generating a state for each input word. To get a representation of the context around a word, we use a bidirectional RNN Schuster and Paliwal (1997) encoder, so both the context before and after contribute to the representation. This is used by Bahdanau et al. (2015) amongst others, achieving good results on a similar task related to text comprehension.

The combined RNN hidden state at time step $i$, $h_{i}$, and the intermediate states, $\stackrel{{\scriptstyle\rightarrow}}{{h}}_{i}$ and $\stackrel{{\scriptstyle\leftarrow}}{{h}}_{i}$, from the forward reader and backward reader respectively, are computed as

where $w_{i}\in V$, for the vocabulary $V$, is word $i$ in the input document; $\stackrel{{\scriptstyle\leftarrow}}{{w}}_{i}$ is word $i$ in the reversed input; and $E(w_{i})$ is the word embedding of $w_{i}$. The initial states $\stackrel{{\scriptstyle\rightarrow}}{{h}}_{0}$ and $\stackrel{{\scriptstyle\leftarrow}}{{h}}_{0}$ are zero vectors. Due to the concatenation, the combined state $h_{i}$ has twice the dimensionality of the state of each unidirectional encoder. The document encoder state dimensionality is denoted $d_{\text{doc}}$ and the word embedding dimensionality $d_{\text{emb}}$.

2 Query Encoder

The query encoder is responsible for creating a fixed-size internal representation of the input query. Unlike the document encoder, the query encoder is a unidirectional RNN encoder since queries are relatively short compared to documents and we only use the final state to represent the whole query. The RNN state $h_{i}^{Q}$ at query word $i$, is updated according to $h^{Q}_{i}=\operatorname{GRU}_{\text{que}}(h^{Q}_{i-1},E(w^{Q}_{i})),q=h^{Q}_{N_{Q}}$, where $w^{Q}$ is the input query and $N_{Q}$ is the length of the query. The initial state $h^{Q}_{0}$ is the zero vector. The query encoder state dimensionality is denoted $d_{\text{que}}$.

3 Decoder

The decoder is a unidirectional RNN for constructing a summary of the input document by depending on the final state of the input encoder, the query. It utilizes soft attention, in combination with a pointer mechanism, as well as a generator part similar to Bahdanau et al. (2015). The query embedding $q$ is fed as input at each decoder time step. This is similar to the answering module in a question answering model presented by Kumar et al. (2016), who use an RNN-encoded question representation as input at each decoder time step. In our model, the RNN state is updated according to $s_{t}=\operatorname{GRU}_{\text{dec}}(s_{t-1},[c_{t},q,E(y_{t-1})])$, where $s_{0}=h_{N_{D}}$, the final document encoder state, $N_{D}$ being the number of input words; $y_{0}$ corresponds to a special token, used at the initial time step when no previous word has been predicted; $c_{t}$ is the context vector at time step $t$ from the attention mechanism, defined subsequently; and $y_{t-1}\in V$ is the predicted output word at time step $t-1$. This is either from the generator mechanism, or the pointer mechanism, also defined subsequently. The word embeddings are the same as are used in the encoder.

The intention of the inclusion of $q$ to the input of $\operatorname{GRU}_{\text{dec}}$ is to give the decoder the ability to tune the structure of the output sequence to eventually output something concerning the query. For example, if the query is a location, the decoder can output words leading up to an appropriate inclusion of the location.

The model has a soft attention mechanism, based on one used by Bahdanau et al. (2015) for machine translation. The result of the attention mechanism is a context vector $c_{t}$ produced at each time step $t$, computed as

4 Pointer Mechanism

A general issue is that with a generator mechanism limited to frequent words, infrequent words cannot be generated. Further, if the model needs to learn to output names, and there are many different ones and few occurrences of each in the training data, training a model to generate them correctly is problematic. A way to solve these issues is to allow the model to directly copy a word in the input document to the output summary, or point to it. This may additionally be viewed as using the input text as a secondary output vocabulary, in addition to $V_{\text{gen}}$.

If $p^{\text{ptr}}_{t}>0.5$, a word is copied from the input, otherwise the generator output is used. What is copied from the input for the $t$th decoder word is determined by the attention distribution. Specifically, at time step $t$, we select the word at index $i^{\prime}_{t}=\underset{i}{\operatorname{arg}\,\operatorname{max}}\;\alpha_{ti}$ in the document, where the attention is highest, as $y^{\text{ptr}}_{t}=w_{\left(i^{\prime}_{t}\right)}$. The final output word can then be defined as

5 Training Loss

The model is trained in when to use the pointer mechanism in a supervised manner. We define an additional training input $x^{\text{ptr}}_{t}$ that is either 1 if the pointer mechanism is set to be used for the $t$th word in the summary, or 0 otherwise. For training this, we define a loss function $L_{\text{ptr}}=\sum_{t=1}^{N_{S}}(x^{\text{ptr}}_{t}(-\log{p^{\text{ptr}}_{t}})+(1-x^{\text{ptr}}_{t})(-\log({1-p^{\text{ptr}}_{t}})))$.

For training the generator mechanism, we define a loss over the generator softmax layer as $L_{\text{gen}}=\sum_{t=1}^{N_{S}}(1-x^{\text{ptr}}_{t})(-\log{P^{\text{gen}}_{t}(w^{*})})$, where $N_{S}$ is the length of the target summary, $w^{*}\in V_{\text{gen}}$ is the the $t$th word in the target summary. Multiplying by $(1-x^{\text{ptr}}_{t})$ excludes any addition to the loss when the pointer mechanism is set to be used.

We introduce a form of supervised attention for when the pointer mechanism is set to be used for an output word by introducing a loss function $L_{\text{att}}=\sum_{t=1}^{N_{S}}x^{\text{ptr}}_{t}(-\log{\alpha_{ti^{*}}})$, where $i^{*}$ is the index in the input document to point to.

The final loss function is the sum of the different losses, normalized by the length, computed as $L=\frac{1}{N_{S}}(L_{\text{gen}}+L_{\text{att}}+L_{\text{ptr}})$.

6 Generating Summaries

Summaries are considered complete when a special token has been generated, or after a maximum output length is reached. Potential summaries are explored using beam search. However, for time steps where the pointer mechanism is used, the partial summaries are prioritized by probabilities as if the generator had been used instead, so $k$ partial summaries with different probabilities are created for the word chosen by the pointer mechanism. This is difficult to justify, but we hope that this should give a reasonable probability at time steps when the pointer mechanism is used, preventing summaries using the pointer mechanism more to be prioritized.

A slight deviation from what is presented in Section 3.4 is that when the pointer mechanism is used and the attended word was not in $V$, we do not output , which it is otherwise interpreted as in the model, but rather the actual input word before it being converted to an index in the vocabulary. This may be viewed as a post-processing step.

Dataset

The dataset constructed for this paper is based Hermann et al. (2015) and consist of document–query–answer triples from CNN and Daily Mail news articles. Included with each published news article, there are a number of human-written highlights, which summarize different aspects of the article. Table 1 shows some example highlights for a single article. They construct a document–query–answer by considering a named entity in a highlight to be unknown, making the highlight into a Cloze-style question Taylor (1953), whose answer is the entity made unknown. An example document and a Cloze-style question and its answer can be seen in Table A.1.

We propose using the CNN/Daily Mail dataset for query-based abstractive summarization by regarding each highlight as a summary of its document, and entities in the highlight as queries. For every occurrence of an entity in a highlight, we construct a document-query-summary triple for query-based summarization. Table A.1 shows for a sample document a Cloze-style question compared and the corresponding query-summary pair constructed by us. If an entity is mentioned in multiple highlights, we consider there being multiple target references for the document-query pair. In contrast to Hermann et al. (2015), we do not translate entities into identifiers but use only minimal preprocessing in the form of tokenization and lowercasing. Further, we mix articles from DNN and Daily mail while Hermann et al. (2015) keeps them separate. We decided to train our model on a mix of CNN and Daily Mail articles, with a proportion of them being reserved for validation and test sets. Which articles are included for the validation and test set is determined randomly with equal probability for every article.

Some statistics of the resulting dataset can be seen in Table 2.

The dataset can be reproduced using a script made available on GitHubhttps://github.com/helmertz/querysum-data.

Experiments

Two experiments were conducted. The first to measure if the model uses the information in the query, Section 5.1, and the second compares the model to an extractive baseline, Section 5.2. A beam width of $k=5$ and a maximum output length of $32$ was used.

To determine whether incorporating a query benefits our model, we compare our proposed model to one where the query is corrupted. Instead of evaluating the generated summary for a document and a query with ID $n$ against the reference summaries for that query, we evaluate it against the reference summaries for query $n+1$, i.e. the query ID has been offset. For the query with the highest ID, the reference summaries for the first query are used. The idea is that if the score is lower than for the normal evaluation, then the model has made use of the additional information in the query. Table 3 shows for an example document, 1, what the generated summaries are evaluated against during the query-dependence evaluation.

It is worth to mention that two reference summaries for different queries may be the same, as the same original highlight may be used as a reference summary for multiple queries. In these cases, the query will be appropriate for the summary and the model may have benefited from the query even in the query-offset evaluation.

2 Extractive Baseline

As a baseline, we compare the results to a simple extractive summary, designed specifically for the dataset used in this thesis work. The baseline summary is constructed by selecting the first sentence in the document containing the query, without restricting the length of the document. If no such sentence is found, i.e. the document does not contain the query, the first sentence of the document is used instead. This does occur in the dataset, but not frequently.

We additionally observe that the average length of baseline sentences using the CNN/Daily Mail dataset is commonly greater than for the reference summaries. The average number of words is 30.56 for the baseline summaries, while it is 14.44 for the reference summaries. It may be possible to gain a higher ROUGE score if a fewer number of words around the query occurrence is selected, but it might not form a complete sentence.

3 Evaluation Metric

Our results are evaluated using four different metrics provided by ROUGE (Recall-Oriented Understudy for Gisting Evaluation) Lin (2004), the defacto standard evaluation method for automatic summarization. ROUGE-1, ROUGE-2, ROUGE-L, and ROUGE-SU4. ROUGE-1 and ROUGE-2 are the scores for 1-grams and 2-grams respectively. ROUGE-L and ROUGE-SU4 are more complex metrics, detailed by Lin (2004).

4 Training Details

The vocabulary $V$ used for the input text contains the 150,000 most frequent words in the training set while the generator vocabulary $V_{\text{gen}}$ consist of the 20,000 most frequent words. The smaller vocabulary of the generator is due to the pointer mechanism.

Word embeddings for the vocabulary words are initialized with 100-dimensional GloVe embeddingsDownloadable as ”glove.6B.zip” at: https://nlp.stanford.edu/projects/glove/, trained on "Wikipedia 2014 + Gigaword 5". If the word does not have a GloVe embedding, we initialize the word embedding by sampling the per-dimension univariate normal distributions with means and standard deviations of the entire collection of GloVe embeddings.

Both during training and test time, we limit the document length to the first 800 words, to reduce computation time.

The loss $L$ is minimized using the SGD-based Adam optimizer Kingma and Ba (2015). We used mini-batches of 30 samples, with an averaged loss over all the samples in the batch. The mini-batches remained the same over epochs, but the order in which they were trained on was randomized between every epoch.

Experiments have been run on a single Nvidia Tesla K80, with 12 GB of memory and took about 54 hours to train. The model is implemented using TensorFlow Abadi et al. (2015), and the complete source has been made available onlinehttps://github.com/helmertz/querysum.

The hyperparameters used for the experiments is reported in Table 4. No extensive hyperparameter tuning has been performed, but instead examined hyperparameters used for similar models, such as Nallapati et al. (2016) and See et al. (2017).

Results

The results from our experiments are summarised in Table 5.

From the result of the query dependence evaluation ("offset queries"), described in Section 5.1, we can see that the ROUGE scores goes down, with statistical significance according to the ROUGE-reported 95% confidence intervals, when the queries are offset. This indicates that the model benefits from the information provided by queries.

Further, we observe that our model score lower than the baseline model which we denote the first query sentence described in Section 5.2. However, it should be noted that this baseline is expected to be strong given the nature of this dataset.

We observe that the attention at a time step appears to often be highly focused on only a few words in the document. An example of an output summary can be seen in Table 6, and Figure A.3 shows the attention distribution over time for the same generated summary.

Another observation we make is that the attention often is focused at the beginning of the documents. However, there are certainly instances when entities are selected from far back in documents. This bias may partly be due to our decision to point out the first occurrences of entities. Although, it has been noted by Goldstein et al. (1999) that the beginning of news articles often summarizes the article quite well.

From examining some of the output summaries from our model, we see that they often strongly match the topic of the input documents, but they rarely succeed in generating summaries rephrasing something actually stated in the article. Table 7 shows an example output that is fairly grammatically correct, but not truthful with respect to the article.

We observe that the model manages to learn some of the dataset samples which are not actual summaries, described in Section 4, such as notices repeated over several articles. The generated summary shown in Table 8 is an example of this. Interestingly, the model manages to literally repeat the reference summary, up to the maximum output length limit.

We can frequently see repetitions of the same phrases; an extreme example can be seen in Figure A.3. The model appears to get stuck trying to begin a summary. Additionally, we observe that the repetition can be observed in the attention distribution as well. The same problem has been seen by Nallapati et al. (2016), who make an addition, temporal attention Sankaran et al. (2016), to their model for alleviating the issue of repetitions. See et al. (2017) propose using coverage to solve the same issue.

Before running experiments, we suspected that it may be difficult for the pointer mechanism to sequentially point out words that make up longer entities. However, we see that this is done successfully quite often. For an example summary, the certainty of selecting a sequence of entity words can be seen in Figure A.3.

Compared to the reference summaries, the output is generally shorter. The average number of words in output summaries is 11.27, while the dataset average is 14.44. As is noted by Wu et al. (2016), beam search commonly favors shorter summaries. They propose an addition of length normalization, for reducing this tendency. Implementing such a measure may improve the results of our model as well.

In comparison to Nallapati et al. (2016) and See et al. (2017), our ROUGE scores are low. They use a different version of the dataset where all highlights are combined to form a single, often multi-sentence, summary. With similar models, they get ROUGE-1 results of around 35 on the general summarization task. However, while they always train the model to output the same summary for the same document, we often have completely different target summaries for different queries, where the queries make up a much smaller part of the input.

Conclusion

We have designed a model for query-based abstractive summarization and evaluated it on an adapted QA dataset, redesigned for query-based summarization. While the overall performance of the model is not enough to outperform our extractive baseline, we have shown that it can incorporate a query and utilize the information to create more focused summaries.

References

Appendix A Supplemental Material

An example of a record in the dataset is shown in Table A.1.

We organize the dataset triples hierarchically, first by document, then query, then reference. The documents and queries are numbered numerically starting with 1, while the references are numbered alphabetically starting with A. Document 1 may have queries 1.1 and 1.2, and reference summaries A.1.1, B.1.1 and A.1.2Selected for matching the format expected by pyrouge. The order is shuffled amongst document, query and reference IDs.