A Divide-and-Conquer Approach to the Summarization of Long Documents

Introduction

Summarization is closely related to data compression and information understanding, both of which are key to information science and retrieval. Being able to produce informative and well-written document summaries has the potential to greatly improve the success of both information discovery systems and human readers that are trying to quickly skim large numbers of documents for important information. Indeed, automatic summarization has been recently recognized as one of the most important natural language processing (NLP) tasks, yet one of the least solved ones .

This work is concerned with the neural summarization of long documents, such as academic articles and financial reports. In previous years, neural summarization approaches have mainly focused on short pieces of text that typically come from news articles . This is also reflected in the amount of datasets that exist for this particular problem .

Summarizing long documents is a very different problem to newswire summarization. In academic articles for example, the input text can range from 2,000 to 7,000 words, while in the case of newswire articles it rarely exceeds 700 words . Similarly, the expected summary of a news article is less than 100 words long, while the abstract of an academic article can easily exceed 200 words.

The increased input and output length lead neural summarization methods to a much higher computational complexity, making it extremely hard to train models that have enough capacity to perform this task. This is more prominent with abstractive summarization models where the complexity of text generation becomes prohibitive for very long sequences. Most importantly, long documents introduce a lot of noise to the summarization process. Indeed, one of the major difficulties in summarizing a long document is that large parts of the document are not really key to its narrative and thus should be ignored. Finally, long summaries typically contain a number of diverse key information points from a document, which are more difficult to produce, compared to the more focused information contained in short summaries.

Certain methods have tried to address these problems by limiting the size of the input document, either by selecting specific sections that are more informative , or by first employing a more efficient extractive model that learns to identify and select the most important parts of the input . While this reduces the noise and the computational cost in processing a long document, there remain the computational cost and information diversity issues in producing a long summary. In the context of Transformer models with self-attention, sparse attention mechanisms such as Big Bird manage to increase the input length by a large amount but they still cannot scale to very long summaries.

In contrast to the above methods that aim to produce a complete summary at once, we propose a novel divide-and-conquer approach that first breaks both the document and its target summary into multiple smaller source-target pairs, then trains a neural model that learns to summarize these smaller document parts, and finally during inference aggregates the partial summaries in order to produce a final complete summary. By decomposing the problem of long document summarization into smaller summarization problems, our approach reduces the computational complexity of the summarization task. At the same time, our approach increases the number and, more importantly, the quality of the training examples by having source and target summary pairs that are focused on a specific aspect of the text, which results in better alignment between them and less noise. This leads to a decomposition of the summarization problem into simpler summarization problems that are easier to learn. Empirical results on two publicly available datasets of academic articles, show that our approach can enhance the ability of summarization models and lead to overall improved results. We show that using a 3 years-old sequence-to-sequence model , our approach manages to achieve surprisingly good results, surpassing recent more advanced models . In addition, when paired with a very strong Transformer model such as PEGASUS our method produces results that are on par with the state-of-the-art on both datasets.

This paper is based on past work that assumed the existence of structured summaries, such as those available for some of the biomedical articles indexed in PubMed. Here we lift this assumption by using sentence level Rouge similarities in order to match sentences of the summary with parts of the document and automatically create source-target pairs for training. This is a key advancement, since the vast majority of academic documents are not accompanied by structured abstracts. Also, such an approach makes this work applicable to any type of document, from academic articles to blog posts and financial documents. Ultimately, our proposed method allows advanced summarization methods to be used in a number of different applications that previously might have been infeasible.

The rest of this work is structured as follows. Section 2 gives a brief overview of the related work. Section 3 describes in detail the proposed method. Section 4 presents the experimental setup and Section 5 discusses the results of our experiments. Finally, Section 6 concludes this works and points to future work directions.

Related work

A variety of solutions have been proposed to the problem of automatic summarization. These include simple unsupervised methods , graph-based methods that involve arranging the input text in a graph and then using ranking or graph traversal algorithms in order to construct the summary and neural methods, which are discussed in more detail in the following subsection. Subsequently we review related work on long document summarization and summarization of academic articles, which are the most common type of long documents in the summarization literature. Finally, we provide and overview of summarization datasets with emphasis on academic article summarization.

Closely following the advances in neural machine translation and language modeling and, fueled by the increased availability of computational resources as well as large annotated datasets , neural summarization is nowadays achieving state-of-the-art results.

Extractive methods aim to select the salient sentences from the input and combine them, typically by concatenation, in order to generate a summary. This is usually approached as a binary classification problem, where for each sentence the model decides whether it should be included in the summary or not . On the other hand, abstractive methods try to encode the input into a hidden representation and then use a decoder conditioned on that representation to generate the summary . In addition to these two main categories, there also exist hybrid approaches that combine both extractive and abstractive methods either by using pointer-generators or by fusing extractive and abstractive models .

In order to encode the input text, different methods are using different variations of encoders based on RNNs or convolutional networks . One notable addition here is which introduces the Rotational Unit of Memory (RUM). RUM is a different type of RNN unit that can be superior to conventional LSTMs in some summarization scenarios. Finally, given the increased popularity and success of large pre-trained Transformers in various NLP tasks, many recent methods employ Transformer models . Towards that direction a variety of pre-training objectives have been suggested that are better suited for the task of abstractive summarization .

In an effort to enhance performance and address some common shortcomings of neural summarization models, policy learning has been proposed to further improve summarization performance.

2 Long document summarization

Most of the aforementioned approaches are mainly focused on summarizing short documents (e.g. news articles), in order to produce short summaries (e.g. headlines).

In cases where the input and target sequences are longer, for example academic articles, the complexity of models that process the complete article at once increases dramatically making such methods infeasible. Different approaches attempt to solve this problem by exploiting the structure of a document. For example, makes use of multiple “encoder agents” where each one processes a different paragraph of the input. A decoder is based on the hidden states of all agents in order to generate the final summary. The model is trained end-to-end using a combination of Maximum Likelihood Estimation (MLE) and Reinforcement Learning (RL) objectives. This approach exploits the structure of an article to shorten the input sequences of each encoding agent. On the other hand, the dependency between encoder agents makes it hard to parallelize, while the single decoder still experiences the same difficulties with long output sequences.

Also, uses a hybrid model with an “extractor agent” that selects salient sentences and an “abstractor agent” that re-writes each of the extracted sentences separately. Each submodel is trained separately with MLE and then the full end-to-end model is trained with RL. This model achieves significant improvements in performance as well as speed during training and decoding. However, while this method, effectively reduces the complexity of the abstractive model that works on single sentences, the extractive model still has to process the whole document. Although extractive models are much more efficient when processing long sequences, there is a limit to the amount of information they can process at once.

Lastly, Big Bird tries to deal with the problem of long document summarization by replacing the full self-attention of Transformer models with a sparse attention mechanism that can scale to inputs that are many times longer. This helps the model use a lot more context when summarizing a document and scale to a lot longer sequences without losing the advantages of full attention. Nevertheless, this method might struggle to scale to documents of arbitrary length and does little to exploit the underlying structure of documents.

In contrast, we treat each section of the text as a separate summarization instance and as a result our method is easily parallelizable. Furthermore, each summarization instance has to deal with significantly shorter input and output sequences than each of these methods. By exploiting the structure of a document it is possible to scale to documents of arbitrary length such as review papers or financial reports. On the downside, the lack of communication during the summarization of different sections may lead to section-level repetitions.

3 Summarizing academic articles

Existing approaches for summarizing academic articles include extractive models that perform sentence selection and hybrid models that first select and then re-write sentences from the full text . In addition, the Pre-training with Extracted Gap-sentences for Abstractive SUmmarization Sequence-to-sequence (PEGASUS) model is a Transformer encoder-decoder pre-trained on massive corpora of documents (Web and news articles) that has demonstrated great potential on various summarization benchmarks, including academic articles. The optimization objective of PEGASUS is called Gap Sentence Generation (GSG), where whole sentences of the input are masked and the model attempts to generate these gap-sentences from the rest of the input. This objective was proposed by the authors of the PEGASUS paper, because it is better aligned with the summarization task and allows for better adaptation of the model during fine-tuning and overall improved performance. The performance and scaling capabilities of PEGASUS can be further improved with the addition of the sparse attention mechanism of Big Bird.

4 Summarization datasets

A number of publicly available datasets of short articles, such as the New York Times , Gigaword , CNN/Daily Mail and Newsroom are commonly used as a benchmark for many of the earlier summarization methods .

When focusing on the task of academic article summarization, several large scale datasets have been introduced. The arXiv and PubMed datasets were created using open access articles from the corresponding popular repositories. PMC-SA is a dataset of open access articles from PubMed Central, where the abstract of each article is structured into sections similar to the full text. Finally, the Science Daily dataset was created by crawling stories from the Science Daily web sitehttps://www.sciencedaily.com/. Each story is about a recent scientific paper and is also accompanied by a short summary that is used as target for training and evaluation.

In addition to the datasets mentioned above, there is also the TAC2014 biomedical summarization datasethttp://www.nist.gov/tac/2014. TAC2014 contains 20 topics, each consisting of one reference article and several articles citing it. Additionally, each reference article is accompanied by four scientific summaries that are written by domain experts. This dataset has been used in the earlier literature , but since it is rather small, it is not suitable for the training of neural summarization models. Another more recent dataset that is focused on scientific articles from the Computational Linguistics domain is the dataset of the CL-SciSumm 2016 shared task . It is composed of 30 annotated sets of open access citing and reference papers accompanied by hand-written summaries. This is also a rather small dataset that is not suitable for neural summarization approaches. Finally, ScisummNet is a medium scale dataset of 1,000 articles from the Computational Linguistics domain that are manually-annotated for summarization.

Our Approach

We propose a divide-and-conquer approach for the summarization of long documents. In this section, we present a training algorithm for the partial summarization systems as well as the methodology we are following at prediction time. Finally, we discuss different model variants that can be combined with this approach.

We argue that a very efficient way of dealing with long documents is to train a summarization model that learns to summarize separately the different sections of the document. Our approach assumes that long documents are structured into discrete sections and exploits this discourse structure by working on each section separately. Each section of the document is treated as a different example during the training of the model by pairing it with a distinct summarization target.

A first idea for achieving this pairing would be to use the whole summary of the document as target for each different section. However, this approach would be problematic for a couple of reasons. First of all, having very long target sequences is very demanding in terms of computational resources. This problem would be even more apparent if instead of an RNN model we decided to use a Transformer-based model, since the computational complexity and memory requirements of full attention Transformers explode for very long sequences. Secondly, the summary will most likely include information that is irrelevant to some sections of the document. For example, information about the conclusions of an academic article in its abstract will most likely be irrelevant to the section describing the methods. As a result, it would be impossible for the model to generate these parts of the target sequence and this may result in poor performance.

We introduce Divide-ANd-ConquER (DANCER) summarization, a method that automatically splits the summary of a document into sections and pairs each of these sections to the appropriate section of the document, in order to create distinct target summaries. Splitting a summary into sections is not straightforward, apart from the limited case of structured abstracts of academic articles . In DANCER we employ ROUGE metrics in order to match each part of the summary with a section of the document. Similar to , a summary is represented as a list of $M$ sentences $A=(a_{1},\ldots,a_{M})$. In addition, each document is represented as a list of $K$ sections $(s^{1},\ldots,s^{K})$ and each section $s^{k}$ of the document as a list of $N$ sentences $s^{k}=(s^{k}_{1},\ldots,s^{k}_{N})$. We compute the ROUGE-L precision, between each sentence of the summary $a_{m}$ and each sentence of the document $s^{k}_{n}$. Given two word sequences $x=(x_{1},\ldots,x_{I})$ and $y=(y_{1},\ldots,y_{L})$ with lengths $I$ and $L$ respectively, the longest common sub-sequence (LCS) is the common sub-sequence with the maximum length. If $LCS(x,y)$ is the length of the longest common sub-sequence of $x$ and $y$, then ROUGE-L precision between $x$ and $y$, $P_{LCS}(x,y)$ is computed as follows:

In more detail, once we have computed the ROUGE-L precision between the summary sentence $a_{m}$ and all the sentences of the document, we find the full text sentence $s^{k_{max}}_{n_{max}}$ with the highest ROUGE-L precision score and we assign $a_{m}$ to be part of the summary of section $k_{max}$. We repeat this process until all sentences of the summary have been assigned to one document section. Then we group all summary sentences by section and concatenate the sentences corresponding to the same section in order to create the target summary for that section.

This approach is mainly inspired by the input-target sentence alignment method that is commonly used to create sentence level targets for extractive summarization . In the extractive summarization context, ROUGE metrics are used in order to match each target sentence with the most similar input sentence. We extend this idea and use the most similar input sentence as an indicator to find the most relevant section of the input for each target sentence and then group target sentences based on their corresponding sections. Other sentence similarity metrics such as BLEU could also be explored in this setup but we leave this for future work.

During training, each section of the document is used as input text and the corresponding part of the summary is the target summary. The training itself is performed with simple teacher forcing , where we are minimizing the negative log likelihood of the target summary sequence $y=(y^{1},\ldots,y^{N})$ given the input sequence $x$.

We have found that this training strategy has several advantages over other methods proposed in the literature. Firstly, by breaking down the problem into multiple smaller problems we greatly reduce the complexity and make it much easier to solve. We believe that this is a very efficient way to approach the summarization of long documents, since it greatly reduces the length of both the input and more importantly, the output sequences. Also, since the target summaries for each section are selected based on the ROUGE-L scores of each sentence, we create a better and more focused matching between the source and the target sequences and avoid having parts of the target summary that are irrelevant to the input sequence. This property prevents us from penalizing the model for not predicting information that was absent in the input text.

Secondly, by splitting each training document into multiple input-target pairs we create a lot more training examples. This is especially beneficial for neural summarization models because by splitting each document into multiple examples we can effectively make use of more training content. This becomes clearer if we think of a neural summarization decoder as a conditional language model that cannot process an unlimited amount of text from each training example. The way that we approach the training allows us to effectively distribute the source and target texts into more training examples and thus enable us to train our model on a larger amount of textual content which leads to improved output quality.

Finally, the method itself is simple and model agnostic and can employ different summarization models, from encoder-decoder RNNs to Transformers. It can also be combined with other more sophisticated methods that perform sentence extraction before the main summarization process, since it has been observed that pointer neural networks sometimes struggle at selecting relevant parts of the input.

2 Section selection

When working with long structured documents it is usually the case that not all sections of the document are key to the document. If we take as an example an academic article, sections like literature review or background are not essential when trying to summarize the main points of the article. On the other hand, sections like the introduction and conclusion usually include quite a lot of the important information that we want to include in the summary. Another similar example would be financial reports that are also structured in sections. Some of those sections, usually referred to as “front-end” sections, include key information and reviews that are core to the narrative, while others consist mostly of financial statements and are less useful for producing a summary .

What’s more, by trying to include sections that are not really important to the overall summary we can possibly end up adding a lot of noise and overall reducing the quality of the generated summary. With that in mind we decided that by selecting specific section types and only including those into the summary we can improve the overall quality of the summarization results.

We are following the same approach described in in order to select the sections we want to use for summarization. First we classify each section into different section types like introduction, methods and conclusion based on a heuristic keyword matching of some common keywords in the section header. The specific keywords used for the classification are presented in Table I.

Based on experiments on the arXiv and PubMed datasets, we have found that for each document the abstract has on average $\sim 6.5$ and $\sim 6.3$ sentences respectively. After classifying the article sections and pairing them with the target summaries created by DANCER we end up having the distribution of target sentences per section type shown in Figure 1.

From this distribution we observe that the majority of summary sentences, especially for the arXiv dataset, are assigned to the introduction section followed by the methods and conclusion sections. The results section is paired with significantly fewer sentences while the literature section is almost never matched with any summary sentences. Based on that observation, when generating the summary we select and use only the sections of the full text that are classified introduction, methods, results and conclusion ignoring the literature section.

This simple method very effectively allows us to filter out parts of the article that are less important for the summary, like the literature review, and leads to summaries that are more focused.

One of the obvious weaknesses of this method is that in some articles the section headers cannot be matched by the heuristic rules and as a result they will be discarded by the heuristic method. Exploring more sophisticated methods that use machine learning to identify the type of each section should be explored in future work. Although these section categories are meaningful when working on academic articles, if the proposed method is extended to different domains (e.g. financial documents), then a new categorization of sections would be required. Towards that direction, a sound idea would be to use machine learning in order to do the section selection. In that scenario a machine learning model can be used in order to make the decision if a given section should be included in the summary. This direction that closely resembles hybrid extractive-abstractive summarization models (although it works on a section level instead of a sentence level) also requires further exploration in future work.

3 Model variants

Here we will describe the different summarization models that we combined with DANCER for our experiments. The first model is an RNN based Pointer-Generator model similar to in two different variants. The second is the PEGASUS model which is based on Transformers.

The Pointer-Generator model is based on the sequence-to-sequence RNN paradigm that has been widely adopted in the pre-Transformer literature. The sequence-to-sequence architecture includes an encoder of bidirectional LSTM units that encodes the input in it’s hidden state and a unidirectional LSTM decoder that autoregressively generates the output one word at a time. Given an input sequence $x=(x^{1},\ldots,x^{T})$ the encoder produces a sequence of hidden states $h$. On each time step $t$ the decoder takes as input the encoder state $h$, the previous word and has a hidden state $s^{t}$.

This model is also equipped with an attention mechanism similar to that generates an attention distribution at each decoder step as in equations 3 and 4 where $v$, $W_{h}$, $W_{s}$ and $b_{attn}$ are learned during training.

From the attention distribution we produce a context vector $h^{*t}$ as shown in equation 5. The context vector is a sum of the encoder hidden states weighted by the attention distribution $\alpha^{t}$.

The context vector is concatenated with the decoder state $s^{t}$ and fed through two linear layers to produce the vocabulary distribution $P_{vocab}$ as shown in equation 6. This is essentially the probability distribution over all words in the vocabulary given the input sequence $x$ and the sequence $y$ generated so far. Again here $V^{\prime}$, $V$, $b^{\prime}$ and $b$ are learnable parameters.

Finally, the model also uses a copying mechanism that has the ability to copy a specific token directly from the input based on a switch mechanism. The token generated at each time step is determined by the vocabulary distribution $P_{vocab}$ and the pointer generator probability $p_{gen}$ as shown in equations 7 and 8. Vectors $w^{T}_{h*}$, $w_{s}$, $w^{t}_{x}$ and scalar $b_{ptr}$ are learnable parameters, $\sigma$ is the sigmoid function and $P_{final}(w)$ is the probability of generating word $w$.

This specific model architecture was proposed by and its variants have been adopted in various other works . Figure 2 better illustrates the full Pointer-Generator model.

One of the advantages of this model is the ability to both extract tokens from the input and generate new tokens with a language model. The language model has the ability to rewrite parts of the text and improve the fluency of the generated text. The copying mechanism is especially important in the case of scientific articles because they include a lot of out-of-vocabulary technical terms as well as symbols. Those cannot possibly be covered by a fixed vocabulary since this will lead to a huge vocabulary and thus make the computational cost of the embedding and softmax layers prohibitive.

3.2 Rotational Unit of Memory

Incorporating rotational units of memory (RUM) into a sequence-to-sequence model can lead to improved summarization results . In particular, including RUM units in the model results in larger gradients during training thus leading to a more stable training and better convergence. In contrast, the gates of LSTM units typically have tanh activation functions and as a result the gradients very quickly become small despite using gradient clipping. We created a variant of our model, where we replaced the LSTM units of the decoder with RUM units. We decided to keep the LSTM units for the encoder, since it has been shown that a mixture of both unit types is usually advantageous .

3.3 PEGASUS

The PEGASUS model is a Transformer based sequence-to-sequence model that is pre-trained on massive corpora of unsupervised data (Web and news articles). The model itself is a standard Transformer encoder-decoder similar to and . The pre-trained model can be further fine-tuned for summarization tasks and is selected here because it has demonstrated great potential on various summarization benchmarks, including CNN/Daily Mail , Gigaword , NEWSROOM , arXiv and PubMed .

What makes the PEGASUS model a promising approach for the summarization task is its pre-training strategy. Gap Sentence Generation (GSG) is a self-supervised objective engineered specifically for abstractive summarization. By masking whole sentences from a document and generating these gap-sentences from the rest of the document encourages the model to understand the whole-document and generate sentences in a summary-like fashion. In addition, they propose a strategy that aims to choose important sentences for masking rather than randomly selected ones.

One key difference of the PEGASUS model with the Pointer-Generator model is that it operates at the level of subword tokens instead of word tokens. This is a common practice for many Transformer models and enables the model to learn and use a wide variety of words with only a limited vocabulary. In particular, the pre-trained version of PEGASUS uses a vocabulary built with the SentencePiece Unigram algorithm although the authors of the paper also experimented with Byte Pair Encoding (BPE) . The use of subword vocabularies is in fact so effective that there is no need to employ copying mechanisms in the context of this model.

4 Compiling the article summary

When we are generating the summary of an article, the following steps are taken. We split the article in sections and select the appropriate sections to use. Then we autoregressively generate a summary for each section of the input text using simple beam search decoding . Finally, we compose the complete summary by concatenating the individual summaries.

Since the summarization of each section is independent of the other sections, our approach is highly parallelizable. At test time, we can very easily process all sections of the document in parallel and thus make the summary generation a lot faster. This can be ideal for systems that are trying to offer summarization as an online service, where the efficiency of the model is an important factor.

One common problem with this type of generative models is that parts of the input might be attended multiple times resulting in repetitions and, in certain situations, the whole decoded sequence may end up in a degenerate repetitive text. This behavior is especially prominent in RNN models. In order to deal with this issue, multiple different approaches have been proposed in the literature. We avoided using the coverage mechanism proposed in , since this approach modifies the training strategy and adds more complexity to the model. Instead, for our Pointer-Generator model, we opted for a simpler yet effective approach that tries to deal with repetition at the decoding phase and was proposed by . During beam search decoding we prevent the decoder from outputting the same trigram multiple times. In order to do this we set the output probability $p(y^{t})=0$, when outputting $y^{t}$ would create a trigram already existing in the generated hypothesis of the current beam.

For the PEGASUS model, based on our experimental results there was a minimal number of repetitions within the section summaries generated by the model. This means that we did not need to use a repetition avoidance mechanism.

Although the aforementioned methods can effectively deal with word and sentence level repetitions, they cannot deal with section level repetitions. Since each individual summary does not have access to the summaries of other sections it is possible that certain information might be repeated in multiple section summaries. The exploration of different strategies that can address this issue is left for future work.

Experimental setup

Here we describe the experiments we conducted with DANCER and the different summarization models on two different datasets in order to demonstrate the effectiveness of the method. We first introduce the two datasets and present the details of the models we are using as well as the training and evaluation setup.

We employed two large-scale publicly available summarization datasets that focus on scientific papers, namely arXiv and PubMed . The arXiv dataset was created directly from LaTeX files that were taken from the arXiv repository of electronic preprints. The files were processed and converted to plain text using Pandochttps://pandoc.org to preserve section information. All citation markers and math formulas were replaced by special tokens. The resulting dataset includes approximately 215k documents with abstracts. The average full text length is 6,913 words and the average abstract length is 292 words.

The PubMed dataset was created from the XML files that are part of the Open Access collection of the PubMed Central (PMC) repository. In contrast to the arXiv dataset, the citation markers were completely removed, while the math equations were converted to plain text. This dataset consists of approximately 133k documents with abstracts. The average full text length is 3,224 words and the average abstract length is 214 words.

Although there is an obvious inconsistency between the pre-processing steps applied to the two datasets we decided to not perform any additional pre-processing in order to be comparable with previously published work. For the same reason we use the predefined training, validation and test set splits. Both datasets are already processed in such a way that only the first level section headings are used as section information and all subsections headings were included as plain text. Also, all figures and tables have already been removed along with text styling options for both datasets. As discussed in Section 3, our method splits each document into multiple training examples based on the discourse structure of the document. As a result, we end up with a lot more training examples than documents. Detailed statistics for both datasets are presented in Table II.

2 Model details

Our LSTM Pointer-Generator model is implemented in Tensorflow and is based on the original implementationhttps://github.com/abisee/pointer-generator of . The hyperparameter selection is similar to the setup suggested in . Our model has a bidirectional LSTM layer of 256 units for the encoder and a unidirectional LSTM layer of 256 units for the decoder.

We restrict the vocabulary to 50,000 word tokens for both the input and output and use word embeddings of size 128. We do not use pre-trained word embeddings, but rather learn them from scratch during training, as suggested in .

Our models were trained on a single Nvidia 1080 GPU with a batch size of 16. We train all of our models using Adagrad with 0.15 learning rate and initialize the accumulator to 0.1. We clip the gradients to have a maximum norm of 2, but avoid using any regularization. During training we are regularly (every 3,000 steps) measuring the loss and the ROUGE-1 F-score on the validation set of the dataset in order to monitor the learning of our model. We end the training when the validation loss stops improving.

For the training, input sequences are truncated to 500 word tokens while padding the shorter ones with zeros to the same length. In our experiments we found that the target sequences created with DANCER rarely exceed 100 words and the average target length is 69 words as shown in Table II. Considering that fact we restrict the length of each target summary to the first 100 words for computational efficiency. We have found that it is preferable to train with the full length sequences from the beginning of the training rather than starting off with highly truncated sequences and then increasing the sequence length after convergence. This is in contrast to the common practice suggested in . We believe one possible reason might be that training with very short and generic sequences first could lead the model to converge into a local optimum and have a hard time getting out of there once the sequence length is increased.

For the prediction phase, we use beam search decoding with 4 beams and generate a maximum of 120 tokens per section. We are also using the mechanism described previously to avoid repeating the same trigrams. Once we have generated a summary for each section, we concatenate the generated summaries in order to get the final summary.

For the RUM variant of the Pointer-Generator model we keep the encoder part the same but we replace the LSTM units of the decoder with RUM units. The RUM unit implementation is taken from the original codehttps://github.com/rdangovs/rotational-unit-of-memory of . All other parameters are similar to the ones used for the LSTM based model.

2.2 PEGASUS

We are using the pre-trained PEGASUS model and the Tensorflow codehttps://github.com/google-research/pegasus that was open-sourced by the authors of the paper. The model itself is the PEGASUSLARGE model described in the paper. It has 16 Transformer blocks for the encoder and decoder with hidden size of 1,024 units, 16 self-attention heads and feed-forward layer size of 4,096 units. It is pre-trained on a combination of the C4 and HugeNews datasets with the GSG objective. We are using the checkpoints open sourced by the authors of the PEGASUS paper to initialize our model and further fine-tune them using DANCER on the arXiv and PubMed datasets.

Our models are fine-tuned on a cloud compute instance with a single Nvidia Tesla T4 GPU. We fine-tune using Adafactor with a learning rate of 0.0001 and a batch size of 6 due to GPU memory limitations. During our fine-tuning we are using input sequences of 512 subwords and target sequences of 128 subwords. This is different than the original PEGASUS setup that uses 1,024 and 256 subwords respectively again due to limited resources. The rest of the hyper-parameters are identical to the ones used in the original paper. The subword vocabulary used is the Unigram vocabulary that was built and open sourced by the PEGASUS paper and has 96,000 subwords. The arXiv model is fine-tuned with DANCER for 60k steps, while the PubMed model is fine-tuned for 40k steps. Our models were not extensively fine-tuned since this was outside the scope of our paper. Therefore, additional hyper-parameter tuning and more fine-tuning steps could potentially lead to even better performance.

For the prediction phase, we use beam search decoding with 5 beams and generate a maximum of 128 tokens per section and then combine them to get the final summary.

3 Baselines and state-of-the-art methods

We compare DANCER with several well known extractive and abstractive baselines as well as state-of-the-art methods. The baseline methods we are comparing against are a simple Lead-10 extractor, which extracts the first 10 sentences of the input, LexRank , SumBasic , LSA , Attention Seq2Seq , Pointer-Generator Seq2Seq , Discourse-Aware Summarizer Sent-CLF, Sent-PTR and TLM-I+E . Attention Seq2Seq is an abstractive sequence-to-sequence model with attention. Pointer-Generator is similar to our LSTM Pointer-Generator model without DANCER. Discourse-Aware Summarizer is a hierarchical extension of the Pointer-Generator model. Sent-CLF and Sent-PTR are extractive models also based on hierarchical LSTMs whith Sent-CLF treating the sentence selection as a sequence classification problem while Sent-PTR uses a sentence pointer to select which sentences to extract. TLM-I+E is a hybrid model that first uses either Sent-PTR or Sent-CLF to extract sentences and then a Transformer language model similar to conditioned on the extracted sentences to generate the summary text. State-of-the-art models include the original PEGASUS model fine-tuned without DANCER on the two datasets and the BigBird-PEGASUS variant that is based on the pre-trained PEGASUS model extended with sparse attention.

Results and discussion

The results of our experiments on the arXiv and PubMed datasets are shown in Tables III and IV respectively. We are reporting the full-length F-score of the ROUGE-1, ROUGE-2 and ROUGE-L metrics computed using the official pyrouge packagehttps://pypi.org/project/pyrouge/0.1.3. All our reported ROUGE scores have a $95\%$ confidence interval of at most $\pm 0.25$ as reported by the official ROUGE script. The results of SumBasic, LexRank, LSA, Attention Seq2Seq, Pointer-Generator Seq2Seq and Discourse-Aware Summarizer are taken directly from , while the results of Sent-CLF, Sent-PTR and TLM-I+E come from . Finally, results of PEGASUS and BigBird-PEGASUS are taken from .

Looking at the comparisons between the different DANCER variants we can see that the PEGASUS model is the clear winner. This is expected since it is a much more powerful and advanced model which makes use of extensive unsupervised pre-training. The price for the better performance is the increased requirements in terms of memory and processing power of PEGASUS compared to the simpler RNN models.

On the other hand both RNN models exhibit similar performance with the RUM model outperforming the LSTM model on the arXiv dataset, while performing slightly worse on PubMed. Given the observation that LSTM based models tend to copy more phrases from the source than RUM based models , we hypothesize that the target abstracts in PubMed include a higher amount of text that is copied directly from the full text, compared to arXiv.

In order to validate this hypothesis we computed the percentage of n-grams in the target summaries that are copied from the source. In Figure 3 we show these percentages for both datasets. It is clear that the target abstracts in the PubMed dataset have a greater percentage of copied 2-grams, 3-grams and 4-grams compared to the arXiv dataset.

In addition, we found that when using a decoder with RUM units, the training is more stable than when using a decoder with LSTM units and converges steadily at a lower loss value. This is in line with the observation that RUM based models exhibit larger gradients and as a result have more robust training compared to LSTM based models . On the other hand, we also found that models with a RUM based decoder need more steps to converge to the final loss, compared to models with an LSTM based decoder.

Overall, based on our experiments and analysis of the three different DANCER models, we conclude that the DANCER PEGASUS model is clearly superior to the RNN models. Nevertheless, when combined with DANCER both RNN models achieve a surprisingly good performance, although not as good as the PEGASUS model. This leads us to believe that there might still be some merit in using RNN sequence-to-sequence models especially in some low resource scenarios.

2 DANCER vs baselines and the state of the art

Based on the numbers of tables III and IV we can see that DANCER works well with both the Pointer-Generator and the PEGASUS model. DANCER improves the performance of the Pointer-Generator model by almost 10 ROUGE-1 points which is a very significant improvement considering that the underlying model is the same. This model is also on par with models such as TLM-I+E as well as Sent-PTR despite using a significantly simpler architecture.

When combined with stronger models such as PEGASUS we can see that DANCER can still lead to improved results with minimal additional effort and resources. Moreover, we empirically show that DANCER can take advantage of strong pre-trained models, such as PEGASUS, and increase the effectiveness of task specific fine-tuning.

Our experiments show that DANCER PEGASUS is on par with the BigBird-PEGASUS model, which is the current state-of-the-art, without modifying the underlying model architecture of PEGASUS. With that in mind, it is possible that a combination of BigBird-PEGASUS with DANCER could further improve results although it was not in the scope of this work to explore that. Furthermore, more extensive optimization of the DANCER PEGASUS model, as well as additional training could lead to even better results but this was not the focus of this work. In general, the experimental results suggest that DANCER is a very easy to implement way to boost the performance of different summarization models with minimal additional effort and resources.

Going back to Figure 3, we notice that both datasets have a high percentage of text copied directly from the source, which explains the high performance of all extractive approaches, even simple ones, like LexRank and Lead-10. Usually it is way easier for extractive models to achieve higher ROUGE scores due to the way that ROUGE metrics are calculated. Since the metric is purely based on the overlap of the the generated text with the target text and in many cases the target summary includes a parts that are copied from the source input, ROUGE scores clearly favor extractive summarization approaches. Nevertheless, we can see that advanced abstractive models such as PEGASUS, BigBird-PEGASUS and DANCER PEGASUS manage to outperform most extractive models by a significant margin. This is important since abstractive summarization is a more challenging task, but also more closely resembles the way humans do summarization.

In the Appendix of this paper we present sample summaries for a couple of papers generated by our models trained on the arXiv dataset. These samples demonstrate the quality of the summaries we can produce using our proposed methods as well as directly compare the outputs produced by the different summarization models.

Conclusion

We presented DANCER, a novel summarization method for long documents. We focused on the summarization of academic articles, but the same method can easily be applied to different types of long documents, such as financial reports. We have demonstrated quantitatively through experiments on the arXiv and PubMed datasets that this method combined with a basic sequence-to-sequence RNN model can still achieve good performance. We also show that using a stronger model such as PEGASUS we can achieve results that are on par with the state-of-the-art on both datasets.

We have also evaluated the advantages of using a combination of LSTM and RUM units inside the sequence-to-sequence model in terms of ROUGE F1 as well as training stability and convergence. We have found that including RUM units in the decoder of the model can lead to a more stable training and better convergence as well as improved ROUGE scores, when the target sequence includes less text directly copied from the source sequence.

Overall, we have focused on the effectiveness of our proposed method regardless of the complexity of the core model. We emphasize that DANCER is a simple yet effective extension that can boost the performance of different summarization models with minimal additional effort and resources. In future work we would like to combine DANCER with more complex summarization models that could potentially further improve summarization quality as well as apply DANCER summarization on domains other than academic articles.

References