Generating Representative Headlines for News Stories

Introduction

Today’s news consumers are inundated with news content—over two million news articles and blog posts are published everydayhttp://www.marketingprofs.com/articles/2015/27698/2-million-blog-posts-are-written-every-day-heres-how-you-can-stand-out. As a result, services that organize news articles have become popular among online users. One approach is to categorize articles into pre-defined news topics, each with a short category label, such as “Technology,” “Entertainment”, and “Sports”. While more organized, redundant content can still appear within each topic. Another more effective approach is to further group articles within each topic based on news stories. Here, each story consists of a cluster of articles reporting the same event. News stories make it more efficient to complete a reader’s news consumption journey — the reader can move from story to story and dive into each one as desired. However, when we only present a list of article titles, readers can hardly get the gist of a story until they have read through several articles, as article titles are tailored to specific articles and do not provide an overview of the entire story. Also, titles can be too long to scan through, especially on mobile devices.

To tackle this problem, we propose to summarize news stories with succinct and informative headlines. For example, “Raptors vs. Bucks” in Figure 1 headlines the cluster of news articles regarding the game between the two teams. Intuitively, headline is a useful complement to news clusters—users can quickly identify what stories they are planning to read in depth, without skimming a flotilla of unorganized news feeds. In practice, the value of headlines is also affirmed: Google News attaches a headline at the top of its story full coverage pagehttps://www.blog.google/products/news/new-google-news-ai-meets-human-intelligence/. However, it remains a challenging research problem to automatically generate story headlines. First of all, selecting an existing article title may not be suitable, as article titles can be too long (especially for readers on mobile devices) or unilateral (incomplete perspective) to cover the general information of the story. Next, relying on human editors to curate high-quality story headlines is inefficient, as it is not only expensive but also hard to scale due to the vast amount of emerging news stories and strict latency requirements.

To this end, we propose to study the problem of automatically generating representative headlines for news stories. The main idea behind it, known as “document summarization”Article and document are used interchangeably throughout this paper. We use the former in the context of news and the latter for document summarization., has been studied for over six decades (Luhn, 1958). The summarization task is to compress a single article into a concise summary while retaining its major information and reducing information redundancy in the generated summary (Filatova and Hatzivassiloglou, 2004; Mihalcea, 2005; Woodsend and Lapata, 2010). As a special task of summarization, single-document headline generation has also been thoroughly studied, whose generated summaries are not longer than one sentence (Banko et al., 2000; Dorr et al., 2003; Zajic et al., 2002). Recently, end-to-end neural generation models have brought encouraging performance for both abstractive summarization (Cheng and Lapata, 2016; Nallapati et al., 2016; Narayan et al., 2018b; Zhang et al., 2018b; See et al., 2017) and single-document headline generation (Gavrilov et al., 2019; Hayashi and Yanagimoto, 2018; Rush et al., 2015; Nallapati et al., 2016). Summarization of multiple documents has also gained much attention (Carbonell and Goldstein, 1998; Barzilay et al., 1999; Haghighi and Vanderwende, 2009; Liu et al., 2018; Fabbri et al., 2019), while headline generation for a set of documents remains a challenging research problem.

The main challenge comes from the lack of high-quality training data. Current state-of-the-art summarization models are dominated by neural encoder-decoder models (You et al., 2019; Zhang et al., 2018b; Hayashi and Yanagimoto, 2018; Gavrilov et al., 2019). The high performance of these data-hungry models is supported by massive annotated training samples. For single-document headline generation, such training data can be easily fetched from unlimited news articles with little cost: existing article-title pairs form perfect training samples. Unfortunately, such a mapping is unavailable for multi-document settings. Manually writing the summary or headline of a set of documents is also much more time consuming than that in the context of single-document summarization. Hence, recent multi-document summarization (MDS) models either seek to adapt from single-document models (Lebanoff et al., 2018; Zhang et al., 2018c) or leverage external resources such as Wikipedia pages (Liu et al., 2018; Liu and Lapata, 2019). (Fabbri et al., 2019) recently provides a crowd-sourced dataset for multi-document summarization, but such resources remain absent for multi-document headline generation.

To facilitate standard study and evaluation, we publish the first dataset for multi-document headline generation. The published dataset consists of 367K news stories with human-curated headlines, 6.5 times larger than the biggest public dataset for multi-document summarization (Fabbri et al., 2019). Large as it may seem, 367K news stories is still a drop in the ocean compared with the entire news corpus on the Web. More importantly, manual curation is slow and expensive, and can hardly scale to web-scale applications with millions of emerging articles every day. To this end, we propose to further leverage the unlabeled news corpus in two ways. Existing articles are first treated as a knowledge base and we automatically annotate unseen news stories by distant supervision (i.e., with one of the article titles in the news story). We then propose a multi-level pre-training framework, which initializes the model with a language model learned from the raw news corpus, and transfers knowledge from single-document article-title pairs. The distant supervision framework enables us to generate another dataset for training without human effort, which is $6$ times larger than the aforementioned human-curated dataset. We show that our model solely based on distant supervision can already outperform the same model trained on the human-curated dataset. In addition, fine-tuning the distantly-trained model with a small number of human-labeled examples further boosts its performance (Section 7.3). In real-world applications, the process of grouping news stories, which is viewed as a prerequisite, is not always perfect. To tackle this problem, we design a self-voting-based document-level attention model, which proves to be robust to the noisy articles in the news stories (Section 7.4). Improving the quality of clustering is out of the scope of this work, but remains an interesting future direction.

Our contributions are summarized as follows:

We propose the task of headline generation for news stories, and publish the first large-scale human-curated dataset to serve the research communityData: https://github.com/google-research-datasets/NewSHeadCode: https://github.com/tensorflow/models/tree/master/official/nlp/nhnet;

We propose a distant supervision approach with a multi-level pre-training framework to train large-scale generation models without any human annotation. The framework can be further enhanced by incorporating human labels, but significantly reduces the demand of labels;

We develop a novel self-voting-based article attention module, which can effectively extract salient information jointly shared by different articles, and is robust to noises in the input news stories.

Problem Description

Given a news story $\mathcal{A}$ as a cluster of news articles regarding the same event, where each article ${\bf{a}}\in\mathcal{A}$ is composed of a token sequence $[a^{1},a^{2},\ldots]$, we aim to generate a succinct and informative headline of the story, represented as another token sequence ${\bf{y}}=[y^{1},y^{2},\ldots]$, such as “Raptors”: $y^{1}$, “vs.”: $y^{2}$ and “Bucks”: $y^{3}$ in Figure 1 for a list of articles discussing about the series between the two teams.

Although the original title of each article can be a strong signal for headline generation, it is not included in our model input as (1) it increases the risk of generating clickbaity and biased headlines, and (2) high-quality titles can be missing in some scenarios (e.g., user-generated content). For these reasons, we only consider the main passage of each article as model input at this stage.

The NewSHead Dataset

To help future research and evaluation, we publish, to the best of our knowledge, the first expert-annotated dataset, NewSHead, for the task of News Story Headline Generation. The NewSHead dataset is collected from news stories published between May 2018 and May 2019. NewSHead includes the following topics: Politics, Sports, Science, Business, Health, Entertainment, and Technology as depicted in Figure 2(a). A proprietary clustering algorithm iteratively loads articles published in a recent time window and groups them based on content similarity. For each news story, curators from a crowd-sourcing platform are requested to provide a headline of up to 35 characters to describe the major information covered by the articles in each story. The curated headlines are then validated by other curators before they are included in the final dataset. Note that a story may contain hundreds of articles, and it is not realistic to ask curators to read through all the articles before curating a headline. Thus, only three to five representative articles that are close to the cluster centroid are picked to save human efforts.

Table 1 shows the statistics of our dataset and the existing datasets for multi-document summarization (MDS). In NewSHead, each news story consists of 3-5 news articles. This gives us 367K data instances, which is $6.5$ times larger than the biggest dataset for multi-document summarization (Fabbri et al., 2019). We split the dataset by timestamps: the timestamps of all articles in the validation set are strictly greater than those in the training set. The same goes for the test set vs. validation set. By avoiding overlapped time window, we can penalize overfitted models that memorize observed labels. Overall, we generate 357K stories for training, 5K stories for validation, and 5K stories for testing, respectively. As for the human-curated reference labels, as Table 1 shows, the lengths of curated story headlines are much shorter than traditional summaries, and even shorter than article titles in our dataset depicted in Figure 2(c). Figure 1 shows an example of a curated news story. The story headline is much more concise than article titles in the cluster, and covers only general information shared by the articles.

Large as it may seem, the training data in NewSHead remains a drop in the ocean compared with the entire news corpus, still leaving substantial room for performance improvement, under the assumption that modern models can achieve better performance with more data (Radford et al., [n.d.]; Liu et al., 2019). Nevertheless, manual annotation is slow and expensive. The amount of work and resource needed to create the NewSHead dataset is already so much that scaling it up to just $500K$ instances seems to be cost-prohibitive. Facing this practical challenge, in the next section, we will present a novel framework based on distant supervision to fetch additional training data without human annotation.

Learning Framework

Learning an end-to-end generation model requires a large amount of annotated training data. Human annotation, however, is slow and expensive. It is thus hard for human annotation alone to provide sufficient data or scale to future scenarios. In the following sections, we present a novel framework that leverages multiple levels of supervision signals to alleviate or even remove the dependency on human annotation. As shown in Figure 3, we seek natural supervision signals in the existing news corpus to pre-train the representation learning and language generation module of our framework. This process includes Language Model Pre-training from massive text corpus, and transferring knowledge from article-title pairs (Single-Doc Pre-training). We then propose to generate heuristic training labels via Multi-Doc Distant Supervision. These supervision signals can be automatically fetched from the existing data with almost no cost. Later in Section 7, we show that models trained from these free signals can outperform those purely trained on the manually curated training set.

In this section, we show how to generate abundant training data from the existing corpus without human story curators. A related technique, namely distant supervision, has proven to be effective in various information extraction tasks (e.g., Relation Extraction (Mintz et al., 2009) and Entity Typing (Ren et al., 2015)). The basic idea is to use existing task-agnostic labels in a knowledge base (KB) to heuristically generate imperfect labels for specific tasks. For example, in Entity Typing, one can match each entity mention (e.g., Donald Trump) in the sentence to an entry in the KB, and then label it with existing types of the entry (e.g., Politician, Human, etc). In this way, by leveraging existing labels and some heuristics, there is no need to spend extra time to curate labels for the specific task of interest.

Here, we view the news corpus as a KB, and treat existing article titles as candidate labels for news stories. Note that not all article titles are suitable as story headlines, since, as we mentioned, some titles are too specific to cover the major information of the story. Hence we need to automatically select high-quality story labels from many candidates without annotations from human experts. Specifically, given a news corpus, we first group news articles into news stories. This is an unsupervised process—the same as what we did for creating the NewSHead dataset. For each story $\mathcal{A}$, we aim to get its heuristic headline $\hat{{\bf{y}}}$ by selecting the most representative article title:

where $\mathcal{T}_{\bf{a}}$ stands for the title of article ${\bf{a}}$ and $f(\mathcal{T},{\bf{a}})$ stands for the semantic matching score between any title $\mathcal{T}$ and article ${\bf{a}}$. Note that ${\bf{a}}$ only contains tokens in the main passage. In other words, the score of an article title is the average matching score between the title and other articles in the story.

The only problem now is to compute the matching score $f(\mathcal{T},{\bf{a}})$. Instead of defining a heuristic score (e.g., lexical overlaps), we train a scorer with a binary classification task, where $f(\mathcal{T},{\bf{a}})$ is the probability that $\mathcal{T}$ can be used to describe ${\bf{a}}$. Training data can be fetched by using existing article-title pairs as positive instances and sampling random pairs as negative instances. We use a BERT-pre-trained Transformer model for this classification task with a cross-entropy loss.

We then follow Equation 1 to generate heuristic labels for unlabeled news stories. It is likely that none of the article titles in the story is representative enough to be the story headline. Hence, we only include positive labelsA label is positive if its average prediction score among all articles is above 0.5. in the generated training data (leaving around 20% of the stories). The length distribution of the generated labels is shown in Figure 2(d). The average length is longer than human labels, but is in a reasonable area to generate enough training instances. This way, we generate around $2.2$M labeled news stories without relying on human curators. The new dataset, namely NewSHeaddist, is $6$x larger than the annotated NewSHead dataset. Table 2 shows an example of heuristically generated labels in NewSHeaddist. Among all five candidate titles, the second title is ranked as the top choice since it well describes the general information of the story. In comparison, the last title is not suitable as it is too specific and does not match some of the articles.

This way we can easily generate abundant high-quality training labels. This generation process does not depend on human curators. NewSHeaddist is hence easy to scale as time goes by and the size of the news corpus gets larger.

2. Pre-training With Natural Supervision

With the distantly supervised dataset comes a natural question: how far can we progress on the story headline generation task without human annotation? To make the most of the massive unlabeled corpus, we apply pre-training techniques to enhance our model. As an overview, our model includes an encoder-decoder module as its building block, and an article-level attention layer to integrate information from different articles in the story. The detailed model architecture will be presented in Section 5.1. At the pre-training stage, aside from Multi-Doc Distant Supervision, we aim to initialize different modules with two kinds of Natural Supervision signals from the existing news corpus as follows.

Language Model Pre-training transfers knowledge from the massive raw corpus in the news domain to enhance the representation learning module of our model. We followed the BERT pre-training paradigm (Devlin et al., 2019) to construct a dataset based on the main passages of over 50M news articles collected from the WebOne can use CommonCrawl to fetch web articles.. This dataset consists of 1.3 billion sentences in total. Both tasks of masked language model and next sentence prediction are included. The learned parameters are used to initialize the encoder module and the word embedding layer. The decoder module can also be initialized with encoder parameters as suggested in literature (Rothe et al., 2019). However, cross attention between the encoder and decoder modules remains uninitialized in this phase.

Single-Doc Pre-training leverages abundant existing article-title pairs to train the encoder-decoder module to enhance both representation learning and language generation. In this step, we further adjust parameters for the encoder and decoder modules, together with cross attentions between them. We clean the 50M raw news articles and leave only 10M high-quality article-title pairs for training. For data cleaning, we first filter out article titles that are either too short (¡15 characters) or too long (¿65 characters). We then use additional classifiers to remove article titles that are clickbaity, offensive or opinion-like. These additional classifiers are trained by binary labels collected from crowd-sourcing platforms. Note that this filtering step is not mandatory in the framework.

The model is further trained on distant supervision data with weights initialized from previous stages. The previous two pre-training stages are used to initialize the encoder and decoder modules in the single document setting. When it comes to multiple documents, where the model involves additional document attentions, we train such parameters together with previously mentioned model components.

Experiments show that models trained with above cost-free signals can even outperform models trained on manually curated training data. By fine-tuning the model on human-curated labels, we can combine the two sources of supervision and further improve performance.

The NHNet Model

In this section, we elaborate on the mathematical mechanism of our multi-doc news headline generation model, namely NHNet. We extend a standard Transformer-based encoder-decoder model to multi-doc setting and propose to use an article-level attention layer in order to capture information common to most (if not all) input articles, and provide robustness against potential outliers in the input due to clustering quality. We analyze the model complexity compared to the standard Transformer model.

Figure 4 illustrates the basic architecture of our generation model. To take full advantage of massive unlabeled data, we start from a Transformer-based single-document encoder-decoder model as the building block. The single-doc model generates decoding output for each article in the cluster individually. To effectively extract common information from different articles, the model fuses decoding outputs from all articles via a self-voting-based article-level attention layer. The framework proves to be not only easy to pre-train with distant supervision, but also robust to potential noises in the clustering process.

We start from a standard Transformer-based encoder-decoder model (Vaswani et al., 2017) as the building block. In the single-doc setting, the sequence of tokens from an input article ${\bf{a}}$ passes through a standard $L$-layer Transformer unit, with $h$ attention heads. At decoding step $i$, the model takes in the full input sequence along with the output sequence produced up to step $i-1$ (i.e., $[y^{1},y^{2},\dots,y^{i-1}]$), and yields a $d_{H}$-dimensional hidden vector (let it be denoted by $H_{{\bf{a}}}^{i}$). The end-to-end single-doc architecture would end up predicting the next token $y^{i}$ in the output sequence from $H_{{\bf{a}}}^{i}$ as follows.

Extending to the multi-doc setting, we let each input article pass through the same Transformer unit and independently yield $H_{{\bf{a}}}^{i}$, i.e., apply the single-doc setting on all input articles up to the point of hidden vector representation. Then we compute the vector representation of the input article group $\mathcal{A}$ as the weighted sum of $\{H_{{\bf{a}}}^{i}\}$, i.e., $H_{\mathcal{A}}^{i}=\sum_{{\bf{a}}\in\mathcal{A}}w_{\bf{a}}H_{{\bf{a}}}^{i}$. The weights $\{w_{\bf{a}}\}$ are determined by a similarity matrix learned via article-level attention (detailed next in Section 5.2). Finally, for predicting the next token $y^{i}$, we use $H_{\mathcal{A}}^{i}$ in place of $H_{{\bf{a}}}^{i}$ in Equation 2.

As we will show in Section 5.2, the article-level attention introduces significantly fewer parameters to learn in addition to those in a standard Transformer model.

2. Voting-based Article-level Attention

where $K_{\bf{a}}$ is the key vector of article ${\bf{a}}\in\mathcal{A}$, which is usually linearly transformed from its encoded representations. Such a design is intuitive, but ignores the interaction between articles. For headline generation, our goal is to capture the common information shared by most articles in the story, where inter-article connections are important to consider. More importantly, the clustered news story itself may not be perfectly clean: some articles may be loosely related, or even unrelated to other articles in the story. In these scenarios, attention scores over articles should be determined by all articles together, instead of relying on an external referee.

To this end, we design a simple yet effective self-voting-based article attention layer. The basic idea is to let each article vote for other articles in the story. Articles with higher total votes from other articles shall have higher attention scores. The advantages are two folds: common information shared by most articles will be echoed and amplified as it gets more votes, while irrelevant articles will be downplayed in this interactive process to reduce harmful interference on the final output. Specifically, given the representation vector $H_{\bf{a}}$ of article ${\bf{a}}$, we calculate its query vector and key vector as $Q_{\bf{a}}=W_{Q}\cdot H_{\bf{a}}$ and $K_{\bf{a}}=W_{K}\cdot H_{\bf{a}}$, respectively, where $W_{Q}$ and $W_{K}$ are learnable matrices shared by all articles. The attention score of ${\bf{a}}$ is then computed as

where $\exp(Q_{{\bf{a}}^{\prime}}^{T}\cdot K_{\bf{a}})$ represents the vote of ${\bf{a}}^{\prime}$ on ${\bf{a}}$. Figure 5 illustrates the similarities and differences between the referee attention and self-voting attention. Among the three example articles, the first two are introducing the game between the two teams, while the third one is focusing on injury information, which is too specific to be included in the headline. Through the self-voting process, the article group finds that the third article is more distant from the central topic, and hence downplays its weight. As a result, the generated headline focuses more on the game between the Raptors and Bucks instead of injury information. The referee attention can hardly achieve the same goal as it ignores other articles when assigning the weight to each individual article. As one can expect, the self-voting attention module is also more robust to potential noises in the cluster. In the sanity checking experiments, the attention module usually gives weights close to zero for intentionally added noisy articles, demonstrating its capability in detecting off-topic articles, whereas the referee attention can hardly sense the differences.

The standard Transformer model consists of $O(L\cdot d_{H}\cdot h\cdot d_{v})$ parameters, where $d_{v}$ is the dimension of projected value vector space, and $L$, $d_{H}$, $h$ are as defined in Section 5.1. Adding article-level attention introduces an additional $O(d_{H}\cdot h\cdot d_{v})$ parameters—$1/L$ of what is already required by Transformer.

Experimental Settings

For standard evaluation, we compare all methods on the test set of NewSHead, and tune parameters on the validation set.

As headline generation for news stories is a new task, instead of a “state-of-the-art” race between various models, we are more curious about the following questions regarding realistic applications:

How far can we go without any human annotation but using existing natural supervision signals only?

Can we further boost the performance of distantly supervised models by incorporating human labels?

How does the performance change with more human labels?

How robust are these methods to potential noises in news stories, as the story clustering process can be imperfect?

Although no previous methods follow exactly the same setting of our task, variants of methods for multi-document summarization (MDS) can serve as our baseline methods with minor adjustments. Specifically, we consider two families of models.

Extractive Methods. Extractive MDS models cannot be directly applied as they generate the summary by selecting sentences from the document set, while our expected output is a concise headline. Extracting suitable words from article bodies is also challenging. Here we consider two competitive baseline methods that (cheatingly) extract information from article titles:

LCS extracts the longest common (word) sequence of article titles in the story. In case that no common word sequence exists, we relax the constraint of finding a common sequence shared by at least two articles.

RepTitles uses the title scorer that we introduced in Section 4.1 for distant supervision, to select the most representative article title in the story as the predicted headline.

Note that the article titles are unavailable for abstractive models including our model.

Abstractive Methods. Abstractive models take article bodies as input and generate the story headline in an end-to-end manner. Since our full model makes use of different kinds of additional natural supervision (e.g., unlabeled corpus and article-title pairs), it would be unfair to compare with models that are not designed to use such signals. For illustration, we compare with both such traditional models (e.g., WikiSum (Liu et al., 2018)) and models designed to leverage additional supervision for pre-training (SinABS (Zhang et al., 2018c)). Additional enhancements are applied to baseline models to make them even stronger and more comparable to our model. All methods are pre-trained with the same resources.

WikiSum (Liu et al., 2018), as a representation of supervised abstractive models, generates summaries from the concatenation of the ordered paragraph list in the original articles. It proposes a decoder-only module with memory-compressed attention for the abstractive stage.

Concat first concatenates body texts in all articles of a story, and then uses the single-document encoder-decoder building block of our model to generate the headline. This way, every attention layer in Transformer is able to access tokens in the entire story. To avoid losing word position information after concatenation, the positional encoding is reset at the first token for each article, so the model can still identify important leading sentences of each article.

SinABS (Zhang et al., 2018c) is a recently proposed model which adapts and outperforms the state-of-the-art MDS models by pre-training on single-document summarization tasks. It uses a referee attention module to integrate encoding outputs from different articles as the representation for the article set. For fair comparison, we replace the original LSTM encoder with the transformer architecture, with the same parameter size as our model.

SinABS (enhanced): The original model only makes use of knowledge from single articles. We further enhance it with distant supervision (H+LSD in Table 3).

NHNet is the model we proposed in Section 5.

We test these baseline methods with different training settings, and report detailed performance comparison and analysis in Section 4.

2. Datasets

As introduced in the multi-level training framework in Sections 3 and 4, the datasets used in this work include

Language model Pre-training (L) with 50M articles;

Single-Doc Pre-training (S) with 10M articles;

Distant Supervision NewSHeaddist (D) with 2.2M stories;

Human annotations from NewSHead (H) with 357k stories.

The entire NewSHead contains 367k instances, among which we use 5k for validation and 5k for final testing.

The labels for training and testing in this work are uncased, since both human-labeled and distantly supervised headline labels may contain various case formats, which can influence the learning and evaluation process. In application, the task of recovering case information from generated uncased headlines, which is also called truecasing in natural language processing, is treated as a separate task. Meanwhile, we discover that simple majority voting from cased frequent n-grams in the story content is an accurate solution.

3. Reproduction Details

We use the WordPiece tool (Wu et al., 2016) for tokenization. The vocabulary size is set to 50k and is derived from uncased news corpus. Each sentence in the news article is tokenized into subwords, which significantly alleviates out-of-vocabulary (OOV) issues. We also tried to incorporate copy mechanism (Gu et al., 2016), another popular choice to reduce OOV, but did not see significant improvement. For better efficiency and less memory consumption, we use up to 200 WordPiece tokens per article as input.

As for the Transformer model, we adopt a standard ($L$=)12-layer architecture, with ($h$=)16 heads, hidden states of ($d_{H}$=)768 dimensions, and projected value space of ($d_{v}$=)48 dimensions. For training, we use the Adam (Kingma and Ba, 2014) optimizer with a 0.05 learning rate and a batch size of 1024. For every model we use the same early stopping strategy to alleviate overfitting: we first let the model train for 10k steps, after that we stop the training process once the model cannot steadily achieve higher performance on the validation dataset.

We implement the model in Tensorflow and train the model on cloud TPUs. Our code will be released together with the NewSHead dataset.

4. Evaluation Metrics

For evaluation we use the open source scoring tool https://github.com/google-research/google-research/blob/master/rouge/scoring.py that aggregates scores from a bootstrapping process. We report average results on the 5k NewSHead test set. We use the following metrics to evaluate the generated headlines:

ROUGE measures n-gram overlap between predicted headlines and gold labels. Here we report the R-1, R-2, R-L scores in terms of (p)recision ($\frac{\#overlap}{\#predicted}$), (r)ecall ($\frac{\#overlap}{\#gold}$) and F1 ($\frac{2\cdot p\cdot r}{p+r}$).

Relative Length measures the ratio between the length of predicted headlines and the gold labels ($\frac{LenPredict}{LenGold}$). Here we report the ratio in terms of both words (Len-W) and characters (Len-C). The closer the relative length is to $1.0$, the more likely it is that the generated headline has a similar length to the gold headline, whose distribution is shown in Figure 2(b).

Experiment Results

In the following, we answer the questions raised in the beginning of the former section.

Table 3 shows the performance of compared methods trained with different combinations of datasets. Generally, abstractive methods outperform extractive methods, even though the latter have access to already summarized information from existing article titles. Among abstractive methods, the carefully designed concatenation model can achieve comparable performance with the existing state-of-the-art. When enhanced with the distantly supervised training data (H+LSD), the SinABS model can have stronger performance. Our model, when trained with the same resources, consistently outperforms baseline methods.

2. How effective is distant supervision?

To investigate the necessity of manual curation for this task, we compare the fully supervised model with human annotations only and the distantly supervised model with all natural supervision signals (i.e., Language Model Pre-training and Single-Document Pre-training). To our surprise, the distantly supervised model outperforms the fully supervised model by a significant margin, even though the distantly supervised training labels have different styles and lengths compared to those in the final test set. The result is encouraging by revealing an effort-light approach for this task. Instead of relying on human experts to curate story headlines for over a year, we can automatically mine high-quality headlines and natural supervision signals from existing news data for training in one day. This observation can serve as a solid foundation for the future development of large-scale production models. Fine-tuning the learned model with human labels can further boost performance. As an ablation study, we investigate the model fine-tuned with full human annotation under different pre-training settings. Starting from the fully supervised model with only human annotations (H), every pre-training process (+L+S+D) brings considerable improvements as we expect.

3. How many manual labels do we need?

Our model achieves the best performance when combining human annotation with all kinds of distant and natural supervision (H($100\%$)+LSD). Since the distantly supervised model may still need some manual labels to adjust the style and length of its generated headlines, it is hence meaningful to investigate the trade-off between the number of manual labels and the final performance to reasonably save human efforts.

Figure 6 shows how test performance changes as we use different ratios of manual labels to fine-tune the distantly supervised model. Generally, more manual labels lead to better test performance, as one can expect. However, different models have various demand for the human labels to achieve the same performance. As shown, when only $2\%$ of human labels are available, the supervised model, even with Language Model Pre-training (H($2\%$)+L), achieves significantly worse performance than the model trained with $100\%$ human labels. Our model with distant supervision (H($2\%$)+LSD), on the contrary, outperforms the fully supervised model by a large margin. More labels beyond this amount only bring slight improvements. This further validates our idea: through distant supervision, we are able to learn high-quality models with very few manual labels.

4. Are attention modules robust to noises?

So far we have mostly considered models in ideal settings, where the news stories for training and testing are relatively clean and validated. In real-world scenarios, however, automatically clustered news stories can be noisy, i.e., they may include some irrelevant or even off-topic articles. This is when the article-level attention module plays its role by assigning different weights to articles.

In this experiment, we intentionally add noises to the training and testing stories (both for NewSHead and NewSHeaddist) by randomly replacing an article in each story with a randomly sampled article from the entire corpus. Under the same architecture, we compare three different article-level attention designs: (1) Uniform Attention, which assigns equal weight to each article; (2) Referee Attention, which determines article weights by an external “referee” vector, as introduced in Section 5.2 and Figure 5; (3) Our self-voting-based Attention. We compare their performances under all pre-training settings.

Table 4 shows the performance of different attention designs. Among all attention modules, Referee Attention achieves the worst performance, which is as expected since Referee Attention can assign inappropriately large weights to noisy articles, severely corrupting the final output. This would be further verified with real examples in Section 8. For comparison, the simple Uniform Attention module will at least avoid focusing on the wrong articles, and hence achieves better performance than Referee. From the model perspective, the Referee model is more complex and is harder to train with limited resources. This experiment also shows that, when on-topic articles dominate the story, even simple uniform attention can achieve satisfactory performance by integrating decoding outputs from different articles, but the traditional Referee Attention can produce risky results. Our Self-voting-based Attention achieves best performances under all settings, thanks to its capability of leveraging the dynamic voting process between articles to emphasize shared common information and identify noises. When tested without any pre-training (H), where the initial article representations are far from perfect, the performance gap between different attention designs becomes more vivid.

Case Study

We conduct case studies to better understand the advantages of the proposed model. Table 5 compares the full model with the variant without fine-tuning on human supervision and without distant supervision on the clean dataset. As one may find, the model using pure distant supervision can already generate headlines of high quality in terms of representativeness and informativeness. Fine-tuning on human labels further adjusts the length and style of the outputs by dropping undesirable verbosity such as full person names and prepositions.

The model without distant supervision fails to pay attention to important words, which results in less meaningful headlines that are often inarticulate. For instance, in the news stories discussing Walmart’s move against Amazon Prime delivery, the model accidentally generates a somewhat meaningless headline regarding Amazon, as the word “Amazon” appears frequently in the news stories. In contrast, the full model generates a high-quality headline with very close semantics to the human label, replacing “announces” with “to offer”. A similar case can also be observed in the story concerning “new Pokemon mobile game”.

Table 6 shows a representative comparison between Self-voting Attention and Referee Attention on the noisy dataset as described in Section 7.4. When an outlier article is added to the story (article 2), the Referee Attention still assigned a relatively high attention weight to it, and hence introduced false information to the headline (“game seven”). On the contrary, our model successfully identified the outlier through the dynamic voting process, and avoided adding noise to the generated headline.

During our study, we also find cases that can be improved. Specifically, in some cases, headlines generated by existing models may focus on different information than human labels. For example, a story labeled as “biden on china tariffs” by human experts is labeled by our algorithm as “biden criticizes trump on tariffs”. Both are good summaries for the story, but focus on different aspects. In the future, we may consider two directions to further satisfy personalized information needs.

Related Work

Three lines of research are closely related: document summarization, news headline generation, and language model pre-training.

Single-Document Summarization (SDS) has been studied for over six decades (Luhn, 1958). Early extractive methods incorporate handcrafted features and graph-based structural information to select informative sentences to form the summary (Filatova and Hatzivassiloglou, 2004; Mihalcea, 2005; Woodsend and Lapata, 2010). Neural extractive models achieve significant improvement by taking advantage of effective representation learning (Cheng and Lapata, 2016; Nallapati et al., 2017; Narayan et al., 2018b; Zhang et al., 2018b). Recent success of seq2seq models has inspired various abstractive summarization methods with an end-to-end encoder-decoder architecture to achieve new state-of-the-art performance (Rush et al., 2015). The encoder module represents input sentences with word embeddings, linguistic features (Nallapati et al., 2016), and abstract meaning representation (Takase et al., 2016; Liao et al., 2018). The input sequence is encoded to an intermediate representation and decoded to target sequence with an RNN or Transformer and their variants (Nallapati et al., 2016; Chen et al., 2016; Li et al., 2017; Liu et al., 2018). To alleviate the Out-Of-Vocabulary (OOV) issue, prior works also incorporate various copy mechanisms (Vinyals et al., 2015; Gu et al., 2016; See et al., 2017) in the framework. Recent works improve the quality of generated summaries in length (Kikuchi et al., 2016; Makino et al., 2019; Narayan et al., 2018a) and informativeness (Pasunuru and Bansal, 2018; Li et al., 2018; You et al., 2019; Peyrard, 2019). In addition, many alternative evaluation metrics (Mao et al., 2019; Narayan et al., 2019; Xenouleas et al., 2019) have been proposed recently due to the limits of ROUGE.

Multi-Document Summarization (MDS), on the other hand, aims to generate summaries for a set of documents. Early works on MDS explore both extractive (Carbonell and Goldstein, 1998; Radev et al., 2004; Erkan and Radev, 2004; Mihalcea and Tarau, 2004; Haghighi and Vanderwende, 2009) and abstractive methods (McKeown and Radev, 1995; Radev and McKeown, 1998; Barzilay et al., 1999; Ganesan et al., 2010). End-to-end abstractive models for MDS are limited by a lack of large-scale annotated datasets. Recent works either try to leverage external resources (Liu et al., 2018; Liu and Lapata, 2019) or adapt single-document summarization models to MDS tasks (Baumel et al., 2018; Zhang et al., 2018c; Lebanoff et al., 2018). The recently developed multi-news dataset (Fabbri et al., 2019) provides the first large-scale training data for supervised MDS, while such a dataset is still absent for the multi-document headline generation task.

Headline Generation is a special task of Document Summarization to generate headline-style abstracts of articles (Shen et al., 2017), which are usually shorter than a sentence (Banko et al., 2000). Over the past decade, both rule-based (Dorr et al., 2003), compression-based (Filippova and Altun, 2013; Filippova et al., 2015) and statistical-based methods (Banko et al., 2000; Zajic et al., 2002) have been explored with handcrafted features and linguistic rules. Recent state-of-the-art headline generation models are dominated by end-to-end encoder-decoder architectures (Rush et al., 2015; Hayashi and Yanagimoto, 2018; Zhang et al., 2018a; Gavrilov et al., 2019; Murao et al., 2019). Similar to summarization models, the encoder module considers different formats of input representation including word position embedding (Chopra et al., 2016), abstractive meaning representations (Takase et al., 2016) and other linguistic features (Nallapati et al., 2016). Pointer networks (Nallapati et al., 2016) and length-controlling mechanisms (Kikuchi et al., 2016) are also developed for this task. However, to the best of our knowledge, the task of headline generation for multiple documents has barely been explored before.

Language Model Pre-training has been proven to be effective in boosting the performance of various NLP tasks with little cost (Radford et al., [n.d.]; Peters et al., 2018; Devlin et al., 2019; Zhang et al., 2019b; Sun et al., 2019; Radford et al., 2019). Recent works applying pre-trained language models also achieve significant success in summarization and headline generation tasks (Song et al., 2019; Rothe et al., 2019; Zhang et al., 2019a). In this work, we study how pre-training at different levels can benefit multi-document headline generation.

Conclusions

In this work, we propose to study the problem of generating headline-style abstracts of articles in the context of news stories. For standard research and evaluation, we publish the first benchmark dataset, NewSHead, curated by human experts for this task. The slow and expensive curation process, however, calls for effort-light solutions to achieve abundant training data from web-scale unlabeled corpus. To this end, we propose to automatically annotate unseen news stories by distant supervision where a representative article title is selected as the story headline. Together with a multi-level pre-training framework, this new data augmentation approach provides us with a $6$ times larger dataset without human curator and enables us to fully leverage the power of Transformer-based model. A novel self-voting-based article attention is applied afterward to better extract salient information shared by multiple articles. Extensive experiments have been conducted to verify NHNet’s superior performance and its robustness to potential noises in news stories.

References