Text-to-Text Pre-Training for Data-to-Text Tasks

Introduction

Natural language generation from structured data, or data-to-text Kukich (1983); McKeown (1985), is the task of generating natural language text conditioned on source content provided in the form of structured data such as a table, graph etc. Some example applications include task oriented dialog Wen et al. (2015), summarizing weather forecasts Sripada et al. ; Goldberg et al. (1994), etc.

In this work we study the applicability of large scale text-to-text transfer learning learning for this task. In particular, we focus on pre-training in the form of the “Text-to-Text Transfer Transformer” (T5) models released by Raffel et al. (2019). Fine-tuning T5 achieves state-of-the-art results on diverse benchmarks spanning task oriented dialogue (MultiWoz), tables-to-text (ToTTo) and graph-to-text (WebNLG). Empirical results further demonstrate the following:

Pre-training greatly improves robustness of models to out-of-domain inputs.

By leveraging pre-training, a simple end-to-end transformer model can outperform sophisticated, multi-stage pipelined approaches and other exotic architectures like graph neural networks.

T5 outperforms alternatives like BERT Devlin et al. (2018) and GPT-2 Radford et al. (2019).

Our approach is simple, only scratching the surface of what is possible. There is much to be explored in the space of leveraging unlabelled data, developing unsupervised objectives etc. that are more tailored for generating text from structured data. We hope our work serves as a useful baseline for future research, as pre-training becomes ever more prevalent for this task.

Related Work

Data-to-Text Early research on data-to-text focused on rule-based methods Reiter and Dale (2000), while recent works have favored neural approaches Wen et al. (2015). Liu et al. (2018) generate text by conditioning language models on tables, Puduppully et al. (2019) explicitly model entities and Marcheggiani and Perez-Beltrachini (2018) encode structured data using graph convolutional networks. Ferreira et al. (2019) and Moryossef et al. (2019) find that neural pipelined approaches perform better than end-to-end models. Transfer Learning Devlin et al. (2018) showed that unsupervised pre-training can greatly benefit tasks like, question answering, summarization etc. In particular, Raffel et al. (2019) perform a large scale study of different training objectives, model capacity and size of data. Peng et al. (2020) and Chen et al. (2019b) show that pre-training in the form of GPT-2 can indeed improve performance on the data-to-text task as well.

Pre-training

We rely on the T5 pre-trained models released by Raffel et al. (2019). They consist of a transformer based encoder-decoder architecture. These models were pre-trained in a multitask fashion with an unsupervised “span masking” objective on Common Crawl data as well as supervised translation, summarization, classification, and question answering tasks. Note that none of the supervised tasks include language generation from structured data. Initial experiments with T5 variants trained on a purely unsupervised objective did not show any difference in performance.

To study the impact of model capacity, we experiment with different T5 variants - Small (60 million parameters), Base (220 million), Large (770 million) and 3B (3 billion).

Fine-tuning

Our modeling approach is simple. The data-to-text task is cast in the text-to-text framework by representing the structured data as a flat string (linearization). Figure 1 shows examples of the input representation for each dataset. We then fine-tune T5 on the data-to-text corpus for a small number of steps.

Following Raffel et al. (2019), models are fine-tuned with a constant learning rate of 0.001. We use a batch size of 131,072 tokens, and a maximum input length of 512 tokens. The maximum training steps is set to 5K for WebNLG, while the larger ToTTo dataset is trained for 10K steps. The T5 vocabulary consists of 32,000 sentencepieces. All the model parameters are updated in the fine-tuning process.

The best checkpoint is chosen based on the BLEU Papineni et al. (2002) score on the development set. Decoding is done via greedy search. In the final evaluation, for each dataset we rely on metrics used by prior work.

Datasets

We conduct experiments on 3 English datasets spanning a variety of domains.

ToTTo Parikh et al. (2020) consists of Wikipedia tables paired with natural language descriptions. The input is a set of cells from a table, along with metadata such as the title of the table.

MultiWoz Budzianowski et al. (2018) is a corpus of 10K human-human dialogs for developing task oriented dialogue systems. For the NLG task, a meaning representation encapsulating system actions must be verbalized into natural language response.

WebNLG Gardent et al. (2017), where the task is to convert a graph of subject-object-predicate triples into a textual description.

Each dataset uses a different kind of structured data (tables, meaning representations and graph/triples). Table 1 lists the sizes of the three datasets and Figure 1 shows examples for each.

Results and Discussion

The evaluation is done using BLEU and METEOR Lavie and Agarwal (2007), similar to Ferreira et al. (2019). The test set is split into two parts - seen and unseen. The examples in the unseen set are drawn from domains not present in the training set, along with roughly 100 new predicates. Some of the baselines we compare with are:

Melbourne, a neural encoder-decoder approach, which scored the highest in the automatic evaluation of the WebNLG challenge Gardent et al. (2017). The model relies on delexicalization, where entities are replaced with placeholders.

GTR-LSTM Distiawan et al. (2018), which employs a graph based triple encoder.

Step-by-Step Moryossef et al. (2019) which splits the generation procedure into a planning stage followed by a neural generation stage.

Pipeline-Transformer Ferreira et al. (2019), a pipelined neural system consisting of discourse ordering, text structuring, lexicalization and referring expression generation.

DualEnc Zhao et al. (2020), the current state-of-the-art system. It consists of a graph convolution network based planning model which first predicts the order of the triples, followed by a separate LSTM with attention and copy mechanism model to generate the text. To train the planning model, the approach relies on extra annotations for the triple ordering. Such annotations are can be expensive and time consuming to obtain, especially for large, complex inputs.

Results are reported in Table 2, for the overall test set as well as the Seen and Unseen splits. T5-Large performs the best across BLEU as well as METEOR. It improves over DualEnc by 4.3 BLEU on the overall test set. It also displays excellent generalization to new domains and relations, with a 14 BLEU improvement on the unseen test set. The results indicate that with pre-training, end-to-end neural models can surpass sophisticated pipelined approaches while being much more robust to domain shift.

2 ToTTo

Following Parikh et al. (2020), BLEU and PARENT are employed as evaluation metrics for this table-to-text generation task. PARENT is a reference less, word-overlap based metric that reflects the factual accuracy of generated text relative to the structured data. Dhingra et al. (2019) find that PARENT correlates better with human factual accuracy judgements in comparison to other generation metrics like ROGUE Lin (2004) and METEOR. The following baseline models are compared:

Pointer Generator See et al. (2017b) - An LSTM based seq2seq model with attention and pointer network based copy mechanism.

BERT-to-BERT Rothe et al. (2019) - A transformer based encoder-decoder model, where both the encoder and decoder are initialized with BERT.

Since it deals with open domain tables, ToTTo is arguably the most challenging dataset. Notably, it features a hidden test set, which is split into two halves - Overlap and Non-Overlap. The Non-Overlap test set features examples that are out-of-domain from the training set.

Results are reported in Table 3. T5-3BWe used beam search with a width of 10 for the test set submission. achieves state-of-the-art results The leaderboard can be found at https://github.com/google-research-datasets/totto., improving upon the BERT baseline by 5.5 BLEU and 5.8 PARENT. Moreover, the model is more robust to out-of-domain tables, with larger improvements of 6.6 BLEU and 7.5 PARENT on the Non-Overlap test set. Table 4 reports results on the development set for the different T5 model sizes. T5-Small outperforms BERT-to-BERT, even though it has 3x fewer parameters (220M vs 60M).

3 MultiWoz

Evaluation on MultiWoz is done using BLEU and SER (Slot Error Rate). SER is the fraction of examples where at least one slot value from the structured data is not expressed in the predicted response. The metric is noisy since the comparison is done via exact match, does not accoutn for paraphrases and does not cover all slots.

HDSA Chen et al. (2019a) is a transformer based architecture that encodes the dialog acts into a multi-layer hierarchical graph, with individual attention heads modeling specific nodes in graph.

SC-GPT2 Peng et al. (2020) is a GPT-2 (345M parameters) model that is further pre-trained on a large data-to-text dialog corpus consisting of 400,000 examples and finally fine-tuned on MultiWoz. This 2 stage pre-training approach is currently state-of-the-art for Multiwoz.

Results are reported in Table 5. All T5 based models (including T5-small which has 5x fewer parameters) outperform SC-GPT2 by 4-5 BLEU without any in-domain pre-training. We note that the SER score on MultiWOZ is slightly worse in comparison with SC-GPT. SC-GPT generates 5 predictions for each input and then ranks them based on the SER score itself, which naturally leads to better slot error rates. On the other hand, we generate a single output.

4 Human Evaluation

We conduct a human evaluation study on WebNLG. Human raters are presented with predicted text, along with up to 3 ground truth references. They are asked to judge the prediction along two axes - (1) Accuracy - A binary rating to gauge whether the prediction conveys the same information as the gold references and (2) Naturalness - A five point scale between 1-3, with 3 indicating a perfectly fluent and grammatical response. Each prediction is rated by 3 raters. For accuracy, we take the majority vote and for naturalness we take the average. We evaluate 500 examples, equally split between the Seen and Unseen test sets.

The evaluation is performed for T5-Large and the current state-of-the-art DualEnc model. Results are reported in Table 6. On the Seen set, both models perform well, with T5 being rated better across both metrics. On the Unseen set, DualEnc shows a large drop of 24% in accuracy while the fluency degrades to just 1.99. Remarkably, T5 sees only a marginal drop, scoring 90% on accuracy and 2.33 on fluency. Table 7 shows some qualitative examples.

5 Impact of model capacity

Our experiments with different T5 variants of varying sizes shed some light on how model capacity impacts performance. The results suggest that it largely depends on the size and complexity of the dataset. For instance, MultiWoz exhibits the least variation in the structured data and is fairly large at 56k examples. Here, even the smallest model T5-Small, is on par with the larger models. WebNLG has only 18K examples and features roughly 200 distinct relations. On the seen test set, all models perform comparably. However, on the unseen test set we notice that performance increases with model size. In particular, there is a stark jump of 10 BLEU when going from T5-Small to T5-Base, implying that model capacity is critical for out-of-domain generalization. A similar trend is observed for ToTTo (Table 4), with a noticeable improvement from Small to Base, followed by smaller improvements upto T5-3B.

Conclusion

In this study we evaluated pre-training in the form of T5 for the data-to-text task. We found that it leads to state-of-the-art results, while greatly improving robustness to out-of-domain inputs. In the future, we hope to design unsupervised pre-training objectives that are specifically tailored for the data-to-text task. We also hope to extend this work to multiple languages, especially low resource ones.

References