ToTTo: A Controlled Table-To-Text Generation Dataset

Introduction

Data-to-text generation (Kukich, 1983; McKeown, 1992) is the task of generating a target textual description ${\bm{y}}$ conditioned on source content ${\bm{x}}$ in the form of structured data such as a table. Examples include generating sentences given biographical data (Lebret et al., 2016), textual descriptions of restaurants given meaning representations (Novikova et al., 2017), basketball game summaries given boxscore statistics (Wiseman et al., 2017), and generating fun facts from superlative tables in Wikipedia (Korn et al., 2019).

Existing data-to-text tasks have provided an important test-bed for neural generation models (Sutskever et al., 2014; Bahdanau et al., 2014). Neural models are known to be prone to hallucination, i.e., generating text that is fluent but not faithful to the source (Vinyals and Le, 2015; Koehn and Knowles, 2017; Lee et al., 2018; Tian et al., 2019) and it is often easier to assess faithfulness of the generated text when the source content is structured (Wiseman et al., 2017; Dhingra et al., 2019). Moreover, structured data can also test a model’s ability for reasoning and numerical inference (Wiseman et al., 2017) and for building representations of structured objects (Liu et al., 2018), providing an interesting complement to tasks that test these aspects in the NLU setting (Pasupat and Liang, 2015; Chen et al., 2019; Dua et al., 2019).

However, constructing a data-to-text dataset can be challenging on two axes: task design and annotation process. First, tasks with open-ended output like summarization Mani (1999); Lebret et al. (2016); Wiseman et al. (2017) lack explicit signals for models on what to generate, which can lead to subjective content and evaluation challenges (Kryściński et al., 2019). On the other hand, data-to-text tasks that are limited to verbalizing a fully specified meaning representation (Gardent et al., 2017b) do not test a model’s ability to perform inference and thus remove a considerable amount of challenge from the task.

Secondly, designing an annotation process to obtain natural but also clean targets is a significant challenge. One strategy employed by many datasets is to have annotators write targets from scratch (Banik et al., 2013; Wen et al., 2015; Gardent et al., 2017a) which can often lack variety in terms of structure and style (Gururangan et al., 2018; Poliak et al., 2018). An alternative is to pair naturally occurring text with tables (Lebret et al., 2016; Wiseman et al., 2017). While more diverse, naturally occurring targets are often noisy and contain information that cannot be inferred from the source. This can make it problematic to disentangle modeling weaknesses from data noise.

In this work, we propose ToTTo, an open-domain table-to-text generation dataset that introduces a novel task design and annotation process to address the above challenges. First, ToTTo proposes a controlled generation task: given a Wikipedia table and a set of highlighted cells as the source ${\bm{x}}$, the goal is to produce a single sentence description ${\bm{y}}$. The highlighted cells identify portions of potentially large tables that the target sentence should describe, without specifying an explicit meaning representation to verbalize.

For dataset construction, to ensure that targets are natural but also faithful to the source table, we request annotators to revise existing Wikipedia candidate sentences into target sentences, instead of asking them to write new target sentences (Wen et al., 2015; Gardent et al., 2017a). Table 1 presents a simple example from ToTTo to illustrate our annotation process. The table and Original Text were obtained from Wikipedia using heuristics that collect pairs of tables ${\bm{x}}$ and sentences ${\bm{y}}$ that likely have significant semantic overlap. This method ensures that the target sentences are natural, although they may only be partially related to the table. Next, we create a clean and controlled generation task by requesting annotators to highlight a subset of the table that supports the original sentence and revise the latter iteratively to produce a final sentence (see §5). For instance, in Table 1, the annotator has chosen to highlight a set of table cells (in yellow) that support the original text. They then deleted phrases from the original text that are not supported by the table, e.g., After winning the German under-23 100 m title, and replaced the pronoun she with an entity Gabriele Becker. The resulting final sentence (Final Text) serves as a more suitable generation target than the original sentence. This annotation process makes our dataset well suited for high-precision conditional text generation.

Due to the varied nature of Wikipedia tables, ToTTo covers a significant variety of domains while containing targets that are completely faithful to the source (see Table 4 and the Appendix for more complex examples). Our experiments demonstrate that state-of-the-art neural models struggle to generate faithful results, despite the high quality of the training data. These results suggest that our dataset could serve as a useful benchmark for controllable data-to-text generation.

Related Work

ToTTo differs from existing datasets in both task design and annotation process as we describe below. A summary is given in Table 2.

Most existing table-to-text datasets are restricted in topic and schema such as WeatherGov (Liang et al., 2009), RoboCup (Chen and Mooney, 2008), Rotowire (Wiseman et al., 2017, basketball), E2E (Novikova et al., 2016, 2017, restaurants), KBGen (Banik et al., 2013, biology), and Wikibio (Lebret et al., 2016, biographies). In contrast, ToTTo contains tables with various schema spanning various topical categories all over Wikipedia. Moreover, ToTTo takes a different view of content selection compared to existing datasets. Prior to the advent of neural approaches, generation systems typically separated content selection (what to say) from surface realization (how to say it) (Reiter and Dale, 1997). Thus many generation datasets only focused on the latter stage (Wen et al., 2015; Gardent et al., 2017b). However, this decreases the task complexity, since neural systems have already been quite powerful at producing fluent text. Some recent datasets (Wiseman et al., 2017; Lebret et al., 2016) have proposed incorporating content selection into the task by framing it as a summarization problem. However, summarization is much more subjective, which can make the task underconstrained and difficult to evaluate (Kryściński et al., 2019). We place ToTTo as a middle-ground where the highlighted cells provide some guidance on the topic of the target but still leave a considerable amount of content planning to be done by the model.

Annotation Process

There are various existing strategies to create the reference target ${\bm{y}}$. One strategy employed by many datasets is to have annotators write targets from scratch given a representation of the source (Banik et al., 2013; Wen et al., 2015; Gardent et al., 2017a). While this will result in a target that is faithful to the source data, it often lacks variety in terms of structure and style (Gururangan et al., 2018; Poliak et al., 2018). Domain-specific strategies such as presenting an annotator an image instead of the raw data (Novikova et al., 2016) are not practical for some of the complex tables that we consider. Other datasets have taken the opposite approach: finding real sentences on the web that are heuristically selected in a way that they discuss the source content (Lebret et al., 2016; Wiseman et al., 2017). This strategy typically leads to targets that are natural and diverse, but they may be noisy and contain information that cannot be inferred from the source (Dhingra et al., 2019).To construct ToTTo, we ask annotators to revise existing candidate sentences from Wikipedia so that they only contain information that is supported by the table. This enables ToTTo to maintain the varied language and structure found in natural sentences while producing cleaner targets. The technique of editing exemplar sentences has been used in semiparametric generation models (Guu et al., 2018; Pandey et al., 2018; Peng et al., 2019) and crowd-sourcing small, iterative changes to text has been shown to lead to higher-quality data and a more robust annotation process (Little et al., 2010). Perez-Beltrachini and Lapata (2018) also employed a revision strategy to construct a cleaner evaluation set for Wikibio Lebret et al. (2016).

Concurrent to this work, Chen et al. (2020) proposed LogicNLG which also uses Wikipedia tables, although omitting some of the more complex structured ones included in our dataset. Their target sentences are annotator-generated and their task is significantly more uncontrolled due to the lack of annotator highlighted cells.

Preliminaries

Our tables come from English Wikipedia articles and thus may not be regular grids.In Wikipedia, some cells may span multiple rows and columns. See Table 1 for an example. For simplicity, we define a table ${\bm{t}}$ as a set of cells ${\bm{t}}=\{{\bm{c}}_{j}\}_{j=1}^{\tau}$ where $\tau$ is the number of cells in the table. Each cell contains: (1) a string value, (2) whether or not it is a row or column header, (3) the row and column position of this cell in the table, (4) The number of rows and columns this cell spans.

Let ${\bm{m}}=(m_{\textrm{page-title}},m_{\textrm{section-title}},m_{\textrm{section-text}})$ indicate table metadata, i.e, the page title, section title, and up to the first 2 sentences of the section text (if present) respectively. These fields can help provide context to the table’s contents. Let ${\bm{s}}=(s_{1},...,s_{\eta})$ be a sentence of length $\eta$. We define an annotation exampleAn annotation example is different than a task example since the annotator could perform a different task than the model. ${\bm{d}}=({\bm{t}},{\bm{m}},{\bm{s}})$ a tuple of table, table metadata, and sentence. Here, ${\bm{D}}=\{{\bm{d}}_{n}\}_{n=1}^{N}$ refers to a dataset of annotation examples of size $N$.

Dataset Collection

We first describe how to obtain annotation examples ${\bm{d}}$ for subsequent annotation. To prevent any overlap with the Wikibio dataset (Lebret et al., 2016), we do not use infobox tables. We employed three heuristics to collect tables and sentences:

We search for tables and sentences on the same Wikipedia page that overlap with a non-date number of at least 3 non-zero digits. This approach captures most of the table-sentence pairs that describe statistics (e.g., sports, election, census, science, weather).

Cell matching

We extract a sentence if it has tokens matching at least 3 distinct cell contents from the same row in the table. The intuition is that most tables are structured, and a row is usually used to describe a complete event.

Hyperlinks

The above heuristics only consider sentences and tables on the same page. We also find examples where a sentence ${\bm{s}}$ contains a hyperlink to a page with a title that starts with List (these pages typically only consist of a large table). If the table ${\bm{t}}$ on that page also has a hyperlink to the page containing ${\bm{s}}$, then we consider this to be an annotation example. Such examples typically result in more diverse examples than the other two heuristics, but also add more noise, since the sentence may only be distantly related to the table.

Using the above heuristics we obtain a set of examples ${\bm{D}}$. We then sample a random subset of tables for annotation, excluding tables with formatting issues: 191,693 examples for training, 11,406 examples for development, and 11,406 examples for test. Among these examples, 35.8% were derived from number matching, 29.4% from cell matching, and 34.7% from hyperlinks.

Data Annotation Process

The collected annotation examples are noisy since a sentence ${\bm{s}}$ may only be partially supported by the table ${\bm{t}}$. We thus define an annotation process that guides annotators through incremental changes to the original sentence. This allows us to measure annotator agreement at every step of the process, which is atypical in existing generation datasets.

The primary annotation task consists of the following steps: (1) Table Readability, (2) Cell highlighting, (3) Phrase Deletion, (4) Decontextualization. After these steps we employ a final secondary annotation task for grammar correction. Each of these are described below and more examples are provided in the Table 3.

If a table is not readable, then the following steps will not need to be completed. This step is only intended to remove fringe cases where the table is poorly formatted or otherwise not understandable (e.g., in a different language). 99.5% of tables are determined to be readable.

Cell Highlighting

An annotator is instructed to highlight cells that support the sentence. A phrase is supported by the table if it is either directly stated in the cell contents or meta-data, or can be logically inferred by them. Row and column headers do not need to be highlighted. If the table does not support any part of the sentence, then no cell is marked and no other step needs to be completed. 69.7% of examples are supported by the table. For instance, in Table 1, the annotator highlighted cells that support the phrases 1995, World Championships, individually, and relay. The set of highlighted cells are denoted as a subset of the table: ${\bm{t}}_{\textrm{highlight}}\in{\bm{t}}$.

Phrase Deletion

This step removes phrases in the sentence unsupported by the selected table cells. Annotators are restricted such that they are only able to delete phrases, transforming the original sentence: ${\bm{s}}\rightarrow{\bm{s}}_{\textrm{deletion}}$. In Table 1, the annotator transforms ${\bm{s}}$ by removing several phrases such as After winning the German under-23 100 m title.

On average, ${\bm{s}}_{\textrm{deletion}}$ is different from ${\bm{s}}$ for 85.3% of examples and while ${\bm{s}}$ has an average length of 26.6 tokens, this is reduced to 15.9 for ${\bm{s}}_{\textrm{deletion}}$. We found that the phrases annotators often disagreed on corresponded to verbs purportedly supported by the table.

Decontextualization

A given sentence ${\bm{s}}$ may contain pronominal references or other phrases that depend on context. We thus instruct annotators to identify the main topic of the sentence; if it is a pronoun or other ambiguous phrase, we ask them to replace it with a named entity from the table or metadata. To discourage excessive modification, they are instructed to make at most one replacement.Based on manual examination of a subset of 100 examples, all of them could be decontextualized with only one replacement. Allowing annotators to make multiple replacements led to excessive clarification. This transforms the sentence yet again: ${\bm{s}}_{\textrm{deletion}}\rightarrow{\bm{s}}_{\textrm{decontext}}$. In Table 1, the annotator replaced she with Gabriele Becker.

Since the previous steps can lead to ungrammatical sentences, annotators are also instructed to fix the grammar to improve the fluency of the sentence. We find that ${\bm{s}}_{\textrm{decontext}}$ is different than ${\bm{s}}_{\textrm{deletion}}$ 68.3% of the time, and the average sentence length increases to 17.2 tokens for ${\bm{s}}_{\textrm{decontext}}$ compared to 15.9 for ${\bm{s}}_{\textrm{deletion}}$.

Secondary Annotation Task

Due to the complexity of the task, ${\bm{s}}_{\textrm{decontext}}$ may still have grammatical errors, even if annotators were instructed to fix grammar. Thus, a second set of annotators were asked to further correct the sentence and were shown the table with highlighted cells as additional context. This results in the final sentence ${\bm{s}}_{\textrm{final}}$. On average, annotators edited the sentence 27.0% of the time, and the sentence length slightly increased to 17.4 tokens from 17.2.

Dataset Analysis

Basic statistics of ToTTo are described in Table 5. The number of unique tables and vocabulary size attests to the open domain nature of our dataset. Furthermore, while the median table is actually quite large (87 cells), the median number of highlighted cells is significantly smaller (3). This indicates the importance of the cell highlighting feature of our dataset toward a well-defined text generation task.

Table 6 shows annotator agreement over the development set for each step of the annotation process. We compute annotator agreement and Fleiss’ kappa Fleiss (1971) for table readability and highlighted cells, and BLEU-4 score between annotated sentences in different stages.

As one can see, the table readability task has an agreement of 99.38%. The cell highlighting task is more challenging. 73.74% of the time all three annotators completely agree on the set of cells which means that they chose the exact same set of cells. The Fleiss’ kappa is $0.856$, which is regarded as “almost perfect agreement” ($0.81$ - $1.00$) according to Landis and Koch (1977).

With respect to the sentence revision tasks, we see that the agreement slightly degrades as more steps are performed. We compute single reference BLEU among all pairs of annotators for examples in our development set (which only contains examples where both annotators chose ${\bm{t}}_{\textrm{highlight}}\neq\emptyset$). As the sequence of revisions are performed, the annotator agreement gradually decreases in terms of BLEU-4: $82.19\rightarrow 72.56\rightarrow 68.98$. This is considerably higher than the BLEU-4 between the original sentence ${\bm{s}}$ and ${\bm{s}}_{\textrm{final}}$ (43.17).

2 Topics and Linguistic Phenomena

We use the Wikimedia Foundation’s topic categorization model (Asthana and Halfaker, 2018) to sort the categories of Wikipedia articles where the tables come from into a 44-category ontology.https://en.wikipedia.org/wiki/Wikipedia:WikiProject_Council/Directory Figure 1 presents an aggregated topic analysis of our dataset. We found that the Sports and Countries topics together comprise 53.4% of our dataset, but the other 46.6% is composed of broader topics such as Performing Arts, Politics, and North America. Our dataset is limited to topics that are present in Wikipedia.

Table 7 summarizes the fraction of examples that require reference to the metadata, as well as some of the challenging linguistic phenomena in the dataset that potentially pose new challenges to current systems. Table 4 gives one example that requires reasoning (refer to the Appendix for more examples).

3 Training, Development, and Test Splits

After the annotation process, we only consider examples where the sentence is related to the table, i.e., ${\bm{t}}_{\textrm{highlight}}\neq\emptyset$. This initially results in a training set ${\bm{D}}_{\textrm{orig-train}}$ of size 131,849 that we further filter as described below. Each example in the development and test sets was annotated by three annotators. Since the machine learning task uses ${\bm{t}}_{\textrm{highlight}}$ as an input, it is challenging to use three different sets of highlighted cells in evaluation. Thus, we only use a single randomly chosen ${\bm{t}}_{\textrm{highlight}}$ while using the three ${\bm{s}}_{\textrm{final}}$ as references for evaluation. We only use examples where at least 2 of the 3 annotators chose ${\bm{t}}_{\textrm{highlight}}\neq\emptyset$, resulting in development and test sets of size 7,700 each.

Without any modification ${\bm{D}}_{\textrm{orig-train}}$, ${\bm{D}}_{\textrm{dev}}$, and ${\bm{D}}_{\textrm{test}}$ may contain many similar tables. Thus, to increase the generalization challenge, we filter ${\bm{D}}_{\textrm{orig-train}}$ to remove some examples based on overlap with ${\bm{D}}_{\textrm{dev}},{\bm{D}}_{\textrm{test}}$.

For a given example ${\bm{d}}$, let ${\bm{h}}({\bm{d}})$ denote its set of header values and similarly let ${\bm{h}}({\bm{D}})$ be the set of header values for a given dataset ${\bm{D}}$. We remove examples ${\bm{d}}$ from the training set where ${\bm{h}}({\bm{d}})$ is both rare in the data as well as occurs in either the development or test sets. Specifically, ${\bm{D}}_{\textrm{train}}$ is defined as:

The $\textrm{count}({\bm{h}}({\bm{d}}),{\bm{D}}_{\textrm{orig-train}})$ function returns the number of examples in ${\bm{D}}_{\textrm{orig-train}}$ with header ${\bm{h}}({\bm{d}})$. To choose the hyperparameter $\alpha$ we first split the test set as follows:

The development set is analogously divided into ${\bm{D}}_{\textrm{dev-overlap}}$ and ${\bm{D}}_{\textrm{dev-nonoverlap}}$. We then choose $\alpha=5$ so that ${\bm{D}}_{\textrm{test-overlap}}$ and ${\bm{D}}_{\textrm{test-nonoverlap}}$ have similar size. After filtering, the size of ${\bm{D}}_{\textrm{train}}$ is 120,761, and ${\bm{D}}_{\textrm{dev-overlap}}$, ${\bm{D}}_{\textrm{dev-nonoverlap}}$, ${\bm{D}}_{\textrm{test-overlap}}$, and ${\bm{D}}_{\textrm{test-nonoverlap}}$ have sizes $3784$, $3916$, $3853$, and $3847$ respectively.

Machine Learning Task Construction

In this work, we focus on the following task: Given a table ${\bm{t}}$, related metadata ${\bm{m}}$ (page title, section title, table section text) and a set of highlighted cells ${\bm{t}}_{\textrm{highlight}}$, produce the final sentence ${\bm{s}}_{\textrm{final}}$. Mathematically this can be described as learning a function ${\bm{f}}:{\bm{x}}\rightarrow{\bm{y}}$ where ${\bm{x}}=({\bm{t}},{\bm{m}},{\bm{t}}_{\textrm{highlight}})$ and ${\bm{y}}={\bm{s}}_{\textrm{final}}$. This task is different from what the annotators perform, since they are provided a starting sentence requiring revision. Therefore, the task is more challenging, as the model must generate a new sentence instead of revising an existing one.

Experiments

We present baseline results on ToTTo by examining three existing state-of-the-art approaches (Note that since our tables do not have a fixed schema it is difficult to design a template baseline).

BERT-to-BERT (Rothe et al., 2020): A Transformer encoder-decoder model (Vaswani et al., 2017) where the encoder and decoder are both initialized with BERT (Devlin et al., 2018). The original BERT model is pre-trained with both Wikipedia and the Books corpus (Zhu et al., 2015), the former of which contains our (unrevised) test targets. Thus, we also pre-train a version of BERT on the Books corpus only, which we consider a more correct baseline. However, empirically we find that both models perform similarly in practice (Table 8).

Pointer-Generator (See et al., 2017): A Seq2Seq model with attention and copy mechanism. While originally designed for summarization it is commonly used in data-to-text as well (Gehrmann et al., 2018).

Puduppully et al. (2019): A Seq2Seq model with an explicit content selection and planning mechanism designed for data-to-text.

Details about hyperparameter settings are provided in the Appendix. Moreover, we explore different strategies of representing the source content that resemble standard linearization approaches in the literature (Lebret et al., 2016; Wiseman et al., 2017)

Full Table The simplest approach is simply to use the entire table as the source, adding special tokens to mark which cells have been highlighted. However, many tables can be very large and this strategy performs poorly.

Subtable Another option is to only use the highlighted cells ${\bm{t}}_{highlight}\in{\bm{t}}$ with the heuristically extracted row and column header for each highlighted cell. This makes it easier for the model to only focus on relevant content but limits the ability to perform reasoning in the context of the table structure (see Table 11). Overall though, we find this representation leads to higher performance.

In all cases, the cells are linearized with row and column separator tokens. We also experiment with prepending the table metadata to the source table.The table section text is ignored, since it is usually missing or irrelevant.

The model output is evaluated using two automatic metrics: BLEU Papineni et al. (2002) and PARENT (Dhingra et al., 2019). PARENT is a metric recently proposed specifically for data-to-text evaluation that takes the table into account. We modify it to make it suitable for our dataset, described in the Appendix. Human evaluation is described in § 8.2.

1 Results

Table 8 shows our results against multiple references with the subtable input format. Both the BERT-to-BERT models perform the best, followed by the pointer generator model.Note the BLEU scores are relatively high due to the fact that our task is more controlled than other text generation tasks and that we have multiple references. We see that for all models the performance on the non-overlap set is significantly lower than that of the overlap set, indicating that slice of our data poses significant challenges for machine learning models. We also observe that the baseline that separates content selection and planning performs quite poorly. We attest this to the fact that it is engineered to the Rotowire data format and schema.

Table 10 explores the effects of the various input representations (subtable vs. full table) on the BERT-to-BERT model. We see that the full table format performs poorly even if it is the most knowledge-preserving representation.

2 Human evaluation

For each of the 2 top performing models in Table 8, we take 500 random outputs and perform human evaluation using the following axes:

Fluency - A candidate sentence is fluent if it is grammatical and natural. The three choices are Fluent, Mostly Fluent, Not Fluent.

Faithfulness (Precision) - A candidate sentence is considered faithful if all pieces of information are supported by either the table or one of the references. Any piece of unsupported information makes the candidate unfaithful.

Covered Cells (Recall) - Percentage of highlighted cells the candidate sentence covers.

Coverage with Respect to Reference (Recall) - We ask whether the candidate is strictly more or less informative than each reference (or neither, which is referred to as neutral).

We further compute an oracle upper-bound by treating one of the references as a candidate and evaluating it compared to the table and other references. The results, shown in Table 9, attest to the high quality of our human annotations since the oracle consistently achieves high performance. All the axes demonstrate that there is a considerable gap between the model and oracle performance.

This difference is most easily revealed in the last column when annotators are asked to directly compare the candidate and reference. As expected, the oracle has similar coverage to the reference (61.7% neutral) but both baselines demonstrate considerably less coverage. According to an independent-sample t-test, this difference is significant at a $p<0.001$ level for both baselines. Furthermore, the baselines are considerably less faithful than the reference. The faithfulness of both models is significantly lower than the reference ($\chi^{2}$ test with $p<0.001$). The models do not differ significantly from each other, except for faithfulness in the non-overlap case, where we see a moderate effect favoring the book model.

Model Errors and Challenges

Table 11 shows predictions from the BERT-to-BERT Books model to illustrate challenges existing models face.

The model sometimes outputs phrases such as first, winning that seem reasonable but are not faithful to the table. This hallucination phenomenon has been widely observed in other existing data-to-text datasets (Lebret et al., 2016; Wiseman et al., 2017). However, the noisy references in these datasets make it difficult to disentangle model incapability from data noise. Our dataset serves as strong evidence that even when the reference targets are faithful to the source, neural models still struggle with faithfulness.

Rare topics

Another challenge revealed by the open domain nature of our task is rare or complex topics at the tail of the topic distribution (Figure 1). For instance, example 2 of Table 11 concerns microdrive capacities which is challenging.

Diverse table structure and numerical reasoning

In example 3, inferring six and five correctly requires counting table rows and columns. Similarly, in example 4, the phrases last and career highs can be deduced from the table structure and with comparisons over the columns. However, the model is unable to make these inferences from the simplistic source representation that we used.

Evaluation metrics

Many of the above issues are difficult to capture with metrics like BLEU since the reference and prediction may only differ by a word but largely differ in terms of semantic meaning. This urges for better metrics possibly built on learned models (Wiseman et al., 2017; Ma et al., 2019; Sellam et al., 2020). Thus, while we have a task leaderboard, it should not be interpreted as the definitive measure of model performance.

Conclusion

We presented ToTTo, a table-to-text dataset that presents a controlled generation task and a data annotation process based on iterative sentence revision. We also provided several state-of-the-art baselines, and demonstrated ToTTo could serve as a useful research benchmark for model and metric development. ToTTo is available at https://github.com/google-research-datasets/totto.

Acknowledgements

The authors wish to thank Ming-Wei Chang, Jonathan H. Clark, Kenton Lee, and Jennimaria Palomaki for their insightful discussions and support. Many thanks also to Ashwin Kakarla and his team for help with the annotations.

References

Appendix A Appendix

The Appendix contains the following contents:

Information about the variant of the PARENT metric (Dhingra et al., 2019) used for evaluation.

Examples of more complex tables in our dataset (Figure 2-Figure 5).

PARENT (Dhingra et al., 2019) is a metric recently proposed specifically for data-to-text evaluation that takes the table into account. We modify it to make it suitable for our dataset. Let $({\bm{x}}_{n},{\bm{y}}_{n},\hat{{\bm{y}}}_{n})$ denote one example that consists of a (source, target, prediction) tuple. PARENT is defined at an instance level as:

$E_{p}({\bm{x}}_{n},{\bm{y}}_{n},\hat{{\bm{y}}}_{n})$ is the PARENT precision computed using the prediction, reference, and table (the last of which is not used in BLEU). $E_{r}({\bm{x}}_{n},{\bm{y}}_{n},\hat{{\bm{y}}}_{n})$ is the PARENT recall and is computed as:

where $R({\bm{x}}_{n},{\bm{y}}_{n},\hat{{\bm{y}}}_{n})$ is a recall term that compares the prediction with both the reference and table. $R({\bm{x}}_{n},\hat{{\bm{y}}}_{n})$ is an extra recall term that gives an additional reward if the prediction $\hat{{\bm{y}}}_{n}$ contains phrases in the table ${\bm{x}}_{n}$ that are not necessarily in the reference ($\lambda$ is a hyperparameter).

In the original PARENT work, the same table ${\bm{t}}$ is used for computing the precision and both recall terms. While this makes sense for most existing datasets, it does not take into account the highlighted cells ${\bm{t}}_{highlight}$ in our task. To incorporate ${\bm{t}}_{highlight}$, we modify the PARENT metric so that the additional recall term $R({\bm{x}}_{n},\hat{{\bm{y}}}_{n})$ uses ${\bm{t}}_{highlight}$ instead of ${\bm{t}}$ to only give an additional reward for relevant table information. The other recall and the precision term still use ${\bm{t}}$.

A.2 Baseline details

BERT-to-BERT (Rothe et al., 2020) - Uncased model coupling both encoder and decoder as in original paper, with Adam optimizer (Kingma and Ba, 2015). learning rate = 0.05, hidden size = 1024, dropout = 0.1, beam size = 4.

Pointer Generator (See et al., 2017) - LSTM with hidden size 300, beam size=8, learning rate = 0.0003, dropout = 0.2, length penalty = 0.0, Adam optimizer (Kingma and Ba, 2015).

Content planner (Puduppully et al., 2019) - All of the original hyperparameters: content planner: LSTM with hidden size 1x600, realizer LSTM with 2x600, embedding size 600 for both, dropout=0.3, Adagrad optimizer (Duchi et al., 2011), beam size=5.