HiStruct+: Improving Extractive Text Summarization with Hierarchical Structure Information

Introduction

Texts, especially long documents, contain internal hierarchical structure like sections, paragraphs, sentences, and tokens. When we manually summarize a text, the hierarchical text structure usually plays a key role. Taking a scientific paper as an example, we might focus more on the sections with the titles of “methodology”, “discussion”, and “conclusion” while paying less attention to the sections like “background”. Furthermore, the sentences within one section could have closer relationship with each other, than the ones outside this section. Understanding not only the sequential relations between the sentences but also the internal hierarchical text structure helps us better determine the important sentences within a document. Similarly, a neural summarization model could benefit from these hierarchical structure information.

In this paper, we focus on extractive text summarization of single documents, which is the task of binary sentence classification with labels indicating whether a sentence should be included in a summary. Recently, pre-trained language models based on Transformer Vaswani et al. (2017), such as BERT Devlin et al. (2019), have been widely used to extract contextual representations from texts. The pre-trained Transformer language models (TLMs) can be easily reused for fine-tuning on the downstream tasks, so that the representations already learned from the large pre-training corpora are preserved. Liu and Lapata (2019) have achieved the state-of-the-art (SOTA) performance by fine-tuning BERT for extractive summarization on short document datasets including CNN/DailyMail. However, the TLMs consider merely the sequential-context-dependency by adding a linear positional encoding to each input token embeddings. The hierarchical text structure information is not taken into account explicitly.

We propose a novel approach to formulate, extract, encode and inject the hierarchical structure (HiStruct) information explicitly into an extractive summarization model (HiStruct+ model), which consists of a TLM for sentence encoding and two stacked inter-sentence Transformer layers for hierarchical learning and extractive summarization (see Figure 1). We experiment with BERT Devlin et al. (2019), RoBERTa Liu et al. (2019), and Longformer Beltagy et al. (2020) as underlying TLMs. The HiStruct information utilized in our work includes the section titles and the hierarchical positions of sentences, which are encoded using our proposed novel methods. The resulting embeddings can be injected into the TLM sentence representations to provide the HiStruct information for the summarization task.

The HiStruct+ models are evaluated on short documents (i. e., CNN/DailyMail See et al. (2017)) and long documents (i. e., PubMed and arXiv Cohan et al. (2018)) with various hierarchical characteristics. Our models produce competitive results on CNN/DailyMail and set the SOTA ROUGEs for extractive summarization on PubMed and arXiv to a new level. We also compare the HiStruct+ models with the corresponding strong baselines, which differ from our models only in that the HiStruct information is not injected. Using various experimental settings, our models collectively outperform the baselines on the three datasets, indicating the effectiveness of the proposed HiStruct encoding methods. The improvements are especially substantial on PubMed and arXiv, which contain longer scientific papers with conspicuous hierarchical structures. Ablation studies suggest that the performance gains are mainly contributed by the hierarchical position information of sentences.

Our contributions in this work are four-folds: (1) We conceptualize novel measures to compare the internal hierarchical structure of the datasets. (2) We propose novel methods to formulate the HiStruct information and implement data preprocessing to extract them from the raw datasets. (3) We propose novel methods to encode and inject the HiStruct information into an extractive summarization model explicitly. The effects of different encoding settings and injection settings are systematically investigated. (4) The data containing the extracted HiStruct information, the best HiStruct+ models, as well as the scripts for preprocessing, training and evaluation are available on GitHubhttps://github.com/QianRuan/histruct.

Related Work

Extractive Text Summarization (ETS) is to classify sentences within a document with labels indicating whether a sentence should be included in the summary. Liu and Lapata (2019) fine-tune BERT with stacked Transformer layers and a sigmoid classifier (BERTSUMEXT). Instead of directly utilizing the existing Transformer encoder for document encoding, Zhang et al. (2019) pre-train a hierarchical Transformer encoder consisting of a sentence encoder and a document encoder (HIBERT) and fine-tune it for ETS. For long documents, Xiao and Carenini (2019) propose a RNN-based ETS model incorporating both the global and the local context (ExtSum-LG). To address the problem of redundancy in extractive summaries, the authors further improve their work by introducing redundancy reduction Xiao and Carenini (2020). They systematically explore and compare different methods including Trigram Blocking Paulus et al. (2018), RdLoss, MMR-Select and MMR-Select+ Xiao and Carenini (2020). Trigram Blocking is a traditional redundancy reduction method that avoids adding a candidate sentence to the summary if it has trigram overlap with the previously selected sentences. Their previous model combined with the redundancy reduction methods produce SOTA performance for ETS on PubMed and arXiv Xiao and Carenini (2020).

Previous works on extractive summarization model hierarchical structure of documents by introducing a hierarchical attention, where they first learn contextual token representations based on the linear dependencies between tokens and then add additional CNN Cheng and Lapata (2016) or RNN Nallapati et al. (2017) or Transformer Zhang et al. (2019); Liu and Lapata (2019) layer(s) to learn document-level representations for each sentence based on the linear dependencies between sentences. However, they learn hierarchical representations of sentences in an implicit way. The models are like black boxes, lacking interpretability. In contrast, our proposed approach enriches sentence representations in an explicit way by using section titles and hierarchical positions of sentences as additional HiStruct information, which is more intuitive and interpretable.

Abstractive text summarization (ATS) is to generate summaries with new sentences which are not present in the source text. BERTSUMABS Liu and Lapata (2019) uses the pre-trained BERT as the encoder in its encoder-decoder architecture. Instead of simply using the pre-trained BERT, recent works, including T5 Raffel et al. (2020), BART Lewis et al. (2020) and PEGAUSUS Zhang et al. (2020) pre-train encoder-decoder models specifically for seq2seq tasks. The first attempt at addressing neural abstractive summarization of long documents is undertaken by Cohan et al. (2018). Aksenov et al. (2020) overcome the length limitations of BERT by a new method of BERT-windowing, allowing it to deal with longer documents. Gidiotis and Tsoumakas (2020) propose a divide-and-conquer approach to train a model to summarize each part of the document separately. To address the essential issue of the quadratic full attention operation of TLMs, Zaheer et al. (2020) propose BigBird with a sparse attention mechanism.

Hybrid text summarization combines extractive summarization, abstractive summarization, or other techniques as a two-stage hybrid system. MatchSum Zhong et al. (2020) is a recent work that first selects sentences from a document using an extractive model and builds a set of candidate summaries based on them. The summarization task is then formulated as a semantic text matching problem between the source document and the candidate summaries. Pilault et al. (2020) presents a hybrid system that consists of an extractive model and a Transformer language model. The Transformer language model employs an encoder-decoder architecture for abstractive summarization, conditioned on the sentences extracted by the extractive model.

2 Injection of Additional Information

The idea of injecting additional information to TLM is inspired by two former works, LAMBERT Garncarek et al. (2020) and LayoutLM Xu et al. (2020), where the visual layout information is injected into BERT by adjusting its input embeddings. These models were not proposed for text summarization and they cannot be applied to plain texts since the layout positions have to be obtained from scanned document images. In contrast, our approach makes use of the internal HiStruct information, which can be found in most types of textual data. Moreover, we enrich the output representations from the TLM instead of adjusting the input embeddings. This saves compute resources since TLM pre-training is not required.

Methodology

Hierarchical position of a sentence is represented in the proposed method as a vector of its positions at each hierarchy-level.

Given the $s$-th sentence within a document, its hierarchical position is formulated as a 2-dimensional vector $(a_{s},b_{s})$, denoted as the sentence structure vector $SSV_{s}$, where $a_{s}$ represents the linear position of the section containing the sentence and $b_{s}$ is the linear position of the sentence within the section. All sentences within the same section have the same value in the first dimension of the SSV, indicating the close relationships between them. The second dimension indicates more precisely their linear relations within the section. By this very simple numerical formulation, hierarchical relations between sentences are clearly identified.

Section titles exist in particular in long documents like scientific papers. They usually imply the section content and describe the common topic for its sub-sentences Ostendorff et al. (2020). In our work, we propose to utilize the corresponding section title as an additional HiStruct information when encoding its sub-sentences. There exist typical section titles in scientific papers. Similar section titles like “Conclusion”, “Conclusions” and “Concluding remarks” have the same semantic meaning and can be grouped into one typical section title class of “Conclusions”. This is also taken into consideration when encoding the section titles.

2 Hierarchical Structure Encoding

Hierarchical position embedding is based on the existing linear position encoding methods (PE), including the sinusoidal method (sin) used by Transformer Vaswani et al. (2017) and the learnable method (la) used by BERT Devlin et al. (2019). We use one of the PEs to encode the two dimensions of a SSV respectively, resulting in two embeddings. Using the la PE, the embeddings are initialized randomly and trained with the entire summarization model. Using the sin PE, the two embeddings are calculated simply by Equations 2 and 3 as described by Vaswani et al. (2017).

where $pos$ is the value in one dimension of the SSV and $i$ is the $i$-th dimension of the resulting embedding.

Given the $s$-th sentence with the hierarchical position of $(a_{s},b_{s})$, and the desired size of the output embeddings $d$, the Sentence Hierarchical Position Embedding (sHE) can be generated by Equations 4, 5, 6, using different combination modes.

where the symbol | denotes vector concatenation.

Using one of the PEs (i. e., sin or la) associated with one of the combination modes (i. e., sum, mean or concat), it totals six different settings of the hierarchical position encoding method: sin-sum, sin-mean, sin-concat, la-sum, la-mean and la-concat.

(Classified) section title embedding is generated by the same pre-trained TLM, which is involved in the summarization model. We have two options to encode section titles: section title embedding (STE) and classified section title embedding (classified STE). A section title embedding is generated by feeding the tokenized section title into the TLM and summing up the last hidden states at each token position as a single embedding. Similar section titles consisting of similar tokens lead to embeddings that are already similar to each other in some way. We also manually pre-define typical section title classes and the corresponding intra-class section titles depending on the datasets and the domains. Using the classified STE, all intra-class STEs are replaced with the embedding of its corresponding class. In the case that a section title does not belong to any class or it falls into more than one class, the original STE is used.

3 Model Architecture

Figure 1 illustrates the overview architecture of the proposed HiStruct+ model. The model consists of a base TLM for sentence encoding and two stacked inter-sentence Transformer layers for hierarchical learning and extractive summarization. The sequence on top is the input document, tokenized by the corresponding tokenizer of the involved TLM. The input embeddings to the TLM are the same as in the original TLM. In order to represent individual sentences, we insert a BOS token at the start of every sentence. Only the BOS token embeddings are preserved as the initial sentence representations ($S_{s}$). Each sentence representation is first enriched with a Sentence Linear Position Embedding, which encodes its linear position within the whole document. An additional Sentence Hierarchical Position Embedding ($sHE_{s}$) can be added, which is generated by encoding the hierarchical position of the sentence using the proposed hierarchical position encoding method. If section titles are available, we can further enrich the sentence representation by adding a STE or classified STE ($STE_{s}$). The sentence representations with the injected HiStruct information are fed to the two stacked Transformer encoder layers to learn inter-sentence document-level hierarchical contextual features. The result is a set of Hierarchical Contextual Sentence Embeddings ($HS_{s}$). The final output layer is a sigmoid classifier, which calculates the confidence score $\hat{y}_{s}$ of including the $s$-th sentence in the extractive summary based on the $HS_{s}$. The loss of the summarization model is the binary classification entropy of the prediction $\hat{y}_{s}$ against the gold label $y_{s}$.

The two HiStruct injection components shaded in light-green are optional. Removing these from the HiStruct+ model based on BERT, the architecture is identical to BERTSUMEXT Liu and Lapata (2019), which is a strong baseline against our models on CNN/DailyMail. When using RoBERTa and Longformer as the base TLM, we also construct a baseline model without the two components. The comparison baselines are named as TransformerETS in this paper. The effectiveness of injecting HiStruct information using the proposed methods can be systematically investigated by comparing our HiStruct+ model to the corresponding TransformerETS baseline which uses the same base TLM and the same input length, but is unaware of the HiStruct information.

Experimental Setup

Our models are evaluated on three benchmark datasets for single document summarization, including CNN/DailyMail See et al. (2017), PubMed and arXiv Cohan et al. (2018). Table 4 presents detailed statistics of the datasets.

The three datasets represent different document types ranging from short news articles to long scientific papers. To emphasize the difference in the hierarchical structure among different datasets, we define the concepts of hierarchical depth (hi-depth) and hierarchical width (hi-width). The hi-depth refers to the number of the hierarchy-levels within the document. Scientific papers have a deeper hierarchy consisting of sections, paragraphs, sentences and tokens (i. e., hi-depth = 4). In news articles, paragraphs are not further grouped into sections (i. e., hi-depth = 3). In this case, we use paragraphs instead of sections as the highest hierarchy level when representing the hierarchical position of sentences (i. e., the first dimension of the SSVs). The hierarchical width, $\textnormal{hi-width}={\dfrac{N_{s}}{N_{hsh}}}$, is the ratio of total number of sentences $N_{s}$ and the number of the text-units regarding the highest structure hierarchy $N_{hsh}$. It indicates how many sentences are there on average in every paragraph/section. The more sentences are there, the second dimension of the SSVs has a more wide range of values, and the values in the first dimension of the SSVs differ a lot from the linear sentence positions. Larger hi-depth and larger hi-width indicate that the hierarchical structure of the dataset is more conspicuous.

We hypothesize that the proposed method works better on datasets with more conspicuous hierarchical structures, where hi-depth and hi-width are larger. This will be proved by comparing the performance improvements on the three datasets with different hierarchical characteristics.

CNN/DailyMail is included as an exemplary dataset with less conspicuous hierarchical structure compared to PubMed and arXiv. The average hi-width over all documents is 1.33, which is much smaller than those in PubMed and arXiv. The dataset contains more than 310k news articles. We use the standard splits given by See et al. (2017) for training, validation, and testing.

During data preprocessing, we first split documents into sentences and paragraphs respectively with the Stanford CoreNLP toolkit Manning et al. (2014). The sentences and paragraphs are tokenized, resulting in the lists of sentence tokens and the lists of paragraph tokens. SSVs corresponding to each sentence can be obtained by comparing those lists side by side. For all three datasets, we use a greedy selection algorithm similar to Nallapati et al. (2017) and Liu and Lapata (2019) to select sentences from documents as the gold extractive summaries (ORACLE). Sentences in the ORACLE summaries are assigned with the gold label 1.

PubMed and arXiv contain longer scientific papers. PubMed contains papers in the bio-medical domain, while arXiv contains papers in various domains. The average hi-width over all PubMed documents is 15.79, in arXiv it is 37.33. We use the original splits given by Cohan et al. (2018) for training, validation, and testing. SSVs are obtained by tokenizing the sentences and sections of every document respectively. The details on the generation of section title embeddings and classified section title embeddings can be found in Appendix A.2.

2 Implementation Details

We implement our model based on BERTSUMEXT Liu and Lapata (2019) and use HuggingFace Transformers Wolf et al. (2020) to make use of the pre-trained instances of BERT, RoBERTa and Longformer. On CNN/DailyMail, we select 3 sentences with Trigram Blocking. On PubMed and arXiv, 7 sentences are extracted while Trigram Blocking is not applied (see more details with regard to implementation in Appendix A.3 and A.4).

Results and Discussion

We evaluate the performance of our summarization models automatically using ROUGE metrics Lin (2004) including F1 ROUGE-1 (R1), ROUGE-2 (R2) and ROUGE-L (RL). Tables 1, 2 and 3 summarize the performance of our models in comparison to the baselines and the previously reported SOTA results on CNN/DailyMail, PubMed and arXiv respectively. On all three datasets, ablation studies are systematically conducted to investigate the contributions of different experimental settings. To analyze the output summaries from an overall perspective, we plot the distribution of the extracted sentences on each dataset and compare it to the ORACLE summaries and those outputted by the comparison baseline (see Figure 2). Appendix A.6 demonstrates human evaluation of extracted summaries for a more intuitive understanding about the superiority of the proposed system.

ROUGE results on CNN/DailyMail are summarized in Table 1. The first three blocks highlight the results reported by the corresponding papers of abstractive, extractive, and hybrid summarization systems. The best results regarding the respective type of the summarization system are underlined. In the baselines block, the first two lines highlight the ORACLE results that build the upper bounds for extractive systems taking the same number of input tokens. The LEAD-n baselines simply select the first n sentences in a document as its extractive summary. Despite its simplicity, the LEAD-3 baseline already achieves relatively competitive performance. The three TransformerETS models are the corresponding comparison baselines that use the same model architecture and experimental settings as our models but without injected HiStruct information. The following block presents the results of our HiStruct+ models based on different TLMs with various input lengths. To make the evaluation results comparable to the SOTA extractive model BERTSUMEXT, we follow their approach and report the averaged results of three best checkpoints.

Regardless the base TLM and input length, our HiStruct+ models collectively outperform the corresponding TransformerETS baselines by merely injecting the hierarchical position information of sentences. However, the performance improvements gained by our models on CNN/DailyMail are small. One of the reasons might be that we merely inject the hierarchical position information of sentences, section titles are not available. Furthermore, as discussed in Section 4, the hierarchical structure of the CNN/DailyMail documents is not so obvious as those in PubMed and arXiv.

Compared to the SOTA extractive model, our best HiStruct+ model produces competitive results. The R2 and RL scores are improved slightly. The model can be reused in many hybrid approaches. When we apply MatchSum based on our best model, the ROUGE results are further increased.

Ablation studies on CNN/DailyMail (see the results and detailed discussions in Appendix A.5) suggest that the setting la-sum works best for hierarchical position encoding. Two stacked Transformer layers in the summarization model perform better than one or three Transformer layers. When taking longer inputs than the length limit of the TLM, substantial improvements are achieved by using the copied token position embeddings for initialization instead of random initialization. The extracted summaries are analyzed in more detail by plotting the proportions of the extracted sentences at each linear position within the whole document as shown in Figure 2(a). The model in green is our best-performed HiStruct+ model on CNN/DailyMail. The model in orange is the corresponding comparison baseline without injected HiStruct information. The model in blue is the ORACLE system, which produces the gold extractive summaries. We can observe that the ORACLE summary sentences are distributed across documents more smoothly, while our HiStruct+ model and the baseline model tend to select the first sentences and fail to select sentences that appear at later positions within the documents. Compared to the baseline, the HiStruct+ model leads to more similar proportions as the ORACLE summaries at the most sentence indices.

2 Results on PubMed

ROUGE results on PubMed are summarized in Table 2. As shown in the baselines block, the ORACLE upper bounds for extractive summarization are increased significantly by increasing the input length, which makes it possible to exploit potential gains from modeling longer input. The LEAD-n baselines do not produce competitive results on PubMed. It indicates that the first sentences in PubMed are not so informative as those in CNN/DailyMail. The last two TransformerETS models in the block are the comparison baselines that are unaware of HiStruct.

The last block in Table 2 presents the results of two groups of HiStruct+ models, grouped by the base TLM used in the summarization model. In PubMed, we can choose to inject the sentence hierarchical position embeddings (sHEs) with or without the section title embeddings (STEs). STEs can be replaced by classified STEs. This can result in three different injection settings for a model group, namely sHE, sHE+STE, and sHE+STE(classified). For each model setting, we report the results of the best-performed checkpoint.

Our best HiStruct+ model on PubMed is a model based on Longformer-base taking 15,000 input tokens, which injects the sHEs and the classified STEs into the extractive model. It achieves ROUGE results of 46.59/20.39/42.11, which beat the SOTA extractive model ExtSum-LG+MMR-Select+ collectively on all three ROUGE metrics with improvements of 1.2/0.02/1.12. Taking the SOTA abstractive and hybrid approaches into account, our results are still very competitive.

All HiStruct+ models produce the competitive results that are better than or very close to the former SOTA results for extractive summarization. They also collectively outperform the TransformerETS baselines by a large margin on all evaluation metrics. The overperformance is much more substantial than that on CNN/DailyMail, even if only the hierarchical position information is injected. This supports our hypothesis that the proposed model works better on datasets with more conspicuous hierarchical structures.

Ablation studies on PubMed suggest that the largest improvement of our models against the baseline is contributed by the hierarchical position information of sentences. This is observed when we compare the three models in the first group of HiStruct+ models with the first TransformerETS baseline. Injecting merely sHE, the results are already increased by 4.07/3.88/3.86. When the section title embedding (STE) is included additionally, the results are further increased by 0.73/0.65/0.68. When using classified STE instead, the ROUGEs are increased by a small margin of 0.1/0.1/0.09. Comparing the second group of HiStruct+ models to the second TransformerETS baseline, it is also observed that injecting the sHE leads to the largest performance gain.

The extracted summaries analysis on PubMed test set is demonstrated in Figure 2(b). The model in green is our best-performed HiStruct+ model on PubMed, the model in orange is the corresponding TransformerETS baseline, the model in blue is the ORACLE system. The ORACLE summaries are distributed across documents evenly. The TransformerETS baseline favors the first 5 sentences and ignores the sentences appearing at later positions. In contrast, our HiStruct+ model overcomes the problem of focusing merely on the first sentences. The outputs of the HiStruct+ model are close to the ORACLE summaries. It indicates that by injecting HiStruct information explicitly using our method, the model successfully learns the deeper internal hierarchical structure of the PubMed documents and relies less on the linear sentence positions.

3 Results on arXiv

ROUGE results on arXiv are summarized in Table 3. The results of the HiStruct+ models are presented in two groups. The first group takes 15k input tokens, while the second group increases the input length to 28k. In the groups, different injection settings are compared.

Our best-performed HiStruct+ model on arXiv is an extractive model based on Longformer-base with 28k input tokens, injecting the sHEs with the original STEs. This model beats the results achieved by ExtSum-LG+RLoss and sets the new SOTA ROUGEs for extractive summarization on arXiv to 45.22/17.67/40.16.

All HiStruct+ models collectively outperform the corresponding TransformerETS baselines (i. e., the last two models in the baselines block) by a large margin on all ROUGE scores. On this dataset, the HiStruct+ improvement is much more significant than those on both CNN/DailyMail and PubMed. The arXiv dataset has the largest hi-width among the three datasets and the hierarchical structure is most conspicuous, which might be the reason why the HiStruct+ models yield the largest performance improvements on arXiv.

Ablation studies in the first HiStruct+ group also suggest that the largest improvement of our HiStruct+ model against the TransformerETS baseline is contributed by injecting the sentence hierarchical position information, which is encoded as sHEs. The effect of using the classified STE on arXiv is opposite to that on PubMed. The summarization performance declines slightly when we replace the STE with the classified STE. This outcome occurs in the second group of HiStruct+ models as well. We notice the fact that there are 500k unique section titles in arXiv, while PubMed contains 164k unique section titles. Accordingly, it becomes much more difficult to group a large number of section titles correctly into several section classes. Furthermore, the PubMed dataset contains papers mostly in the bio-medical domain. The structure of those papers tends to follow specific writing conventions in the bio-medical sciences. The arXiv dataset, in contrast, contains scientific papers that are not limited to a specific domain. As consequence, the document structure and the writing styles are more diverse.

The extracted summaries analysis on arXiv is demonstrated in Figure 2(c). The baseline (in orange) tends to select the first sentence and the sentences indexed between 10 and 20, while it excludes sentences at later positions. It is clearly observed that the summary sentences extracted by the HiStruct+ model are evenly distributed, the informative sentences appearing at later positions are not ignored.

Conclusions

This work addresses hierarchical modeling for extractive text summarization by explicitly leveraging hierarchical structure information, including section titles, as well as hierarchical position information of the sentences. We propose an intuitive and interpretable approach to formulate, extract, encode and inject the hierarchical structure information into an extractive summarization model.

The proposed HiStruct+ models are systematically evaluated on CNN/DailyMail, PubMed, and arXiv. On PubMed, our model increases the former SOTA ROUGEs for extractive summarization by 1.2/0.02/1.12. On arXiv, the new SOTA results for extractive summarization are set to 45.22/17.67/40.16. Our ablation studies suggest that the SOTA performance are mostly gained by providing the hierarchical position information of sentences to the summarization model. When comparing the HiStruct+ models with the baselines that are unaware of the HiStruct information, improvements are consistently observed on all three datasets under various experimental settings, indicating the effectiveness of the proposed method. Moreover, our experiments show that the more conspicuous hierarchical structure the dataset has, the larger the improvements of our method are. The proposed metrics of hi-depth and hi-width determine whether it is worth using our method by comparing the metrics of any dataset to those of the three involved datasets.

Utilizing the HiStruct information also for abstractive summarization is subject of future work. Similarly, we see great potential in an encoder-decoder architecture with the proposed HiStruct injection components.

Acknowledgements

The work presented in this article has received funding from the German Federal Ministry of Education and Research (BMBF) through the project QURATOR (no. 03WKDA1A).

References

Appendix A Appendix

The CNN/DailyMailhttps://cs.nyu.edu/~kcho/DMQA/, PubMed and arXivhttps://github.com/armancohan/long-summarization datasets are used in experiments. We use the original splits provided by See et al. (2017) and Cohan et al. (2018) for training, validation and testing.

A.2 Pre-defined Section Title Classes

The pre-defined dictionaries of the typical section title classes and the corresponding in-class section titles are released in our GitHub project (see Section 1). There are 164,195 unique section titles in PubMed, and 500,015 in arXiv, which are encoded as section title embeddings (STE) respectively using the proposed encoding method.

For PubMed, we define 8 section title classes: introduction, background (i. e., background, review and related work), case (i. e., case reports), method, result, discussion, conclusion and additional information (i. e., additional information such as conflicts of interest, financial support and acknowledgments). For arXiv, we define 10 classes: introduction, background, case, theory (i. e., problem formulation and proof of theorem), method, result, discussion, conclusion, reference and additional information. Classified STEs are prepared accordingly by replacing the original STEs of the intra-class section titles with the encoding of the section title class.

A.3 Implementation Details

The learning rate schedule follows Liu and Lapata (2019) with warm-up. On CNN/DailyMail, we train the HiStruct+ models and the TransformerETS baselines 50,000 steps with 10,000 warm-up steps. On PubMed and arXiv, the models are trained 70,000 steps with 10,000 warm-up steps when taking 15,000 tokens as input. When training models on arXiv with 28,000 input tokens, we train 100,000 steps with 10,000 warm-up steps.

The number of the extracted sentences depends on the dataset. On CNN/DailyMail, we follow Liu and Lapata (2019) to select 3 sentences for each document as its extractive summary and apply Trigram Blocking Paulus et al. (2018) to reduce the redundancy of the selected sentences. On PubMed and arXiv, 7 sentences are extracted without Trigram Blocking.

The length limit of the original TLM is overcome by adding extra token linear position embeddings (tPE) to cover the desired length. The additional tPE are then trained with the whole summarization model. Instead of initializing them randomly, we copy the original tPE of the TLM multiple times until the desired length is covered.

The HiStruct+ models and the TransformerETS baselines are trained on 3 GPUs (NVIDIA® Quadro RTX™ 6000 GPUs with 24GB memory) with gradient accumulation every two steps. Checkpoints are saved and evaluated on the validation set every 1,000 steps. The top-3 checkpoints based on the validation loss are kept. The batch size varies with the base TLM and the input length. On CNN/DailyMail, the base TLM is fine-tuned with the whole summarization model. Due to resource limitation, the TLM (i. e., Longformer) is not fine-tuned when training the summarization model on PubMed and arXiv with longer inputs.

A.4 Model Architectures and Experimental Settings

The detailed model architectures and experimental settings for the models trained on CNN/DailyMail, PubMed and arXiv are summarized in Table 5, Table 6 and Table 7 respectively. The detailed model architectures and experimental settings include:

Base TLM: the Transformer language model used for sentence encoding in the summarization system.

Input length: how many tokens are taken as input.

Extra tPE: how to initialize the extra input token linear position embeddings when taking longer input. We can choose to randomly initialize them or copy the original ones.

FT: whether the base TLM is fine-tuned with the entire summarization model.

TL: the number of the Transformer layers stacked upon the base TLM for extractive summarization.

WS: warmup steps, how many steps are used for warming-up of the learning rate.

BS: batch size, how many documents are used as one batch during training.

AC: accumulation count, gradient accumulation every $k$ steps.

GPU: the number of GPUs used for training, we use NVIDIA® Quadro RTX™ 6000 GPUs with 24GB memory.

HiStruct: the injection setting. Hierarchical structure information that can be injected into the summarization model are: sHE (i. e., sentence hierarchical position embeddings), STE (i. e., section title embeddings), or STE(classified) (i. e., classified section title embeddings)

HPE: the hierarchical position encoding method used in the model. The method is based on the sinusoidal (sin) or the learnable (la) linear position encoding method associated with a combination mode (i. e., sum/mean/concat)

#PE: the numbers of the learned position embeddings for each hierarchy-level of the hierarchical positions and the linear sentence positions, when using the learnable position encoding method. We set them to a same value during training.

SS: saving steps, save checkpoints every $k$ steps.

n: select $n$ sentences as the extractive summary for each document.

TB: trigram blocking, whether to apply Trigram Blocking during sentence selection

A.5 Ablation Studies on CNN/DailyMail

The effect of token-level hierarchical position embeddings is investigated in experiments. The hierarchical position embeddings of tokens are generated as followings:

Given the $t$-th token within the document, its hierarchical position is represented by Equation 7:

where $a_{t}$ represents the linear position of the section which contains the token, $b_{t}$ is the sentence’s position within the section and $c_{t}$ is the linear position of the token within the sentence.

Given the $t$-th token and the desired size of the output embeddings $d$, its token hierarchical position embeddings (tHE) is encoded by Equations 8, 9, 10, using different combination modes.

Initial experiments are conducted to assess the summarization performance of the HiStruct+ models with or without the tHE. For this purpose, we compare a HiStruct+ model merely injecting sentence hierarchical position embeddings (i. e., sHE) with a HiStruct+ model with both sentence and token hierarchical position embeddings (i. e., sHE& tHE). That is, it adds the corresponding tHEs to the input embeddings at each input position, which are fed into the TLM. It also injects sHEs into the output sentence representations.

Table 8 summarizes the evaluation results of three groups of HiStruct+ models based on different TLM with various input lengths. In each group, all experimental settings and parameters are the same, except for the injection setting of tHE. The experimental results suggest that the HiStruct+ models with merely sHE consistently outperform the HiStruct+ models with both sHE & tHE under various circumstances. The reason might be that we directly fine-tune the TLM on the extractive summarization task. When adding extra tHE to the input embeddings to the TLM, we do not pre-train the TLM with the adjusted inputs. It is reasonable that the TLM has difficulties in understanding of the new inputs based on the knowledge learned from the original format of encoding. Previous works, such as LayoutLM Xu et al. (2020), LamBERT Garncarek et al. (2020) and HIBERT Zhang et al. (2019), which adjust the input embeddings or the encoder architecture of the pre-trained TLM, continue to pre-train the TLM on their own data. Continuing pre-training of the language models is a core part of these works and leads to significant improvements on downstream tasks. Due to lack of computing resources, we are not able to pre-train the language models. Furthermore, the key goal of our work is to experiment with various methods to make use of the internal hierarchical text structure information for extractive summarization. In this work, we conduct further experiments without token-level hierarchical position information and leave for future work the pre-training of language models with the adjusted input embeddings.

The effect of different settings for hierarchical position encoding is investigated. As explained previously, based on different position encoding (PE) methods (i. e., sin or la) associated with various combination modes (i. e., sum, mean or concat), we have totally six different settings for hierarchical position encoding: sin-sum, sin-mean, sin-concat, la-sum, la-mean and la-concat. We investigate the effect of those 6 encoding settings systematically in experiments while keeping the rest settings and parameters the same, so that the evaluation results are comparable.

Table 9 summarizes the evaluation results of six HiStruct+ models using the six encoding settings respectively, which are all trained on CNN/DailyMail based on BERT-base with 1,024 input tokens, injecting merely sHE. We observe that when using the la method, the combination mode sum leads to better results compared to the other modes (see the first three columns in Table 9). When using the sin method, the various combination modes do not make a conspicuous difference in summarization performance. The sum and concat modes perform slightly better. When using the sum mode, the la and the sin methods produce similar results (see the first row in Table 9).

The effect of the encoding settings la-sum vs. sin-sum is further investigated in experiments. As discussed above, the encoding settings la-sum and sin-sum lead to similar results. We conduct experiments to further investigate the effect of using these methods. We also compare our HiStruct+ models with the corresponding TransformerETS baseline which differs from our models only in that it does not take into account extra HiStruct information.

Table 10 includes the ROUGEs of three set of comparison models, which use different TLM with various input lengths. In each group, the first model is the baseline without HiStruct injection. The second model and the third model differ from each other only with regard to the encoding setting. The experimental results suggest that both of the settings improve the summarization performance of the baseline model. It is also observed that the la-sum method outperforms the sin-sum method slightly on CNN/DailyMail. The differences are not substantial.

The effect of the number of the stacked Transformer layers is investigated in our experiments. We fine-tune an extended BERT-base model with 1,024 input tokens for extractive summarization. We construct the HiStruct+ models with 1, 2, 3 stacked Transformer layers respectively, while keeping all other settings the same. The results of those three HiStruct+ models are reported in the first block in Table 11. It is suggested that two stacked Transformer layers perform best in our HiStruct+ models for extractive summarization.

The effect of different initialization strategies for the additional input Token Linear Position Embeddings is also investigated in experiments. When taking input texts longer than the original input length of the base TLM, we need to add extra Token Linear Position Embeddings (tPE) for each extended position. We can choose to randomly initialize the extra tPE or copy the original ones to cover the extended input length. To investigate the effect of different initialization strategies, we use the basic settings of the HiStruct+ model with two summarization layers, namely the second model in the first block in Table 11. To build the comparison model, only the initialization strategy is changed to random. As shown in the second block in Table 11, substantial improvements are achieved by using the copied tPEs for initialization instead of random initialization. ROUGE-1, ROUGE-2 and ROUGE-L are increased by 2.84, 2.51 and 2.95 respectively.

The effect of the Sentence Linear Position Embeddings is also investigated in experiments. As shown in Figure 1, besides the hierarchical positions of each sentence, we also take the linear position of each sentence within the whole document into account by adding a Sentence Linear Position Embedding (sPE) to each sentence representation. We assess the effect of the sPE by comparing two HiStruct+ models with or without the sPE. The second model in the first block in Table 11 is compared to a model that differs from it only in the injection of sPE. The results are shown in the third block in Table 11. The HiStruct+ model with sPE outperforms the HiStruct+ model without sPE by a small margin regarding all ROUGE metrics.

A.6 Human Evaluation of Extracted Summaries

To have a more intuitive understanding about the superiority of the proposed system, we showcase two samples in Figure 3 for human evaluation and case analysis. The extractive summaries predicted by the HiStruct+ model and the baseline model are demonstrated respectively, in comparison with the gold summary (i.e., the abstract of the paper). To construct a final summary, top-7 sentences with the highest scores predicted by the model are extracted, and then combined in their original order.

The first arXiv sample shows that the baseline simply selects the first sentences. The predicted summary focuses on detailed background knowledge and lacks an overview of the proposed work. In contrast, our HiStruct+ model selects sentences at later positions. The first five sentences introduce the main content from an overall perspective. The last two sentences draw conclusions and give an outlook to future work, which is indicated by the phrases highlighted in green.

The PubMed sample also indicates that the baseline favors the first sentences, which is consistent with our observations in Figure 2. Although the last two sentences highlight the same conclusion as in the gold summary that locally informed diagnosis and treatment strategies are needed, too much background information is unnecessarily included in the first five sentences. Our HiStruct+ model selects more informative sentences at later positions. The predicted summary covers all key parts of the gold summary: 1). the statistics are reported (i.e., 26% of primary tuberculosis (tb) was multidrug resistant (mdr)); 2). the novel strain s256 is mentioned; 3). the conclusion is highlighted. The overall topic of the work is especially highlighted by the sentence with the green-colored phrase.