AlignScore: Evaluating Factual Consistency with a Unified Alignment Function

Introduction

Recent systems for natural language generation, such as summarization and dialogue systems, can produce fluent and coherent text. However, studies show the generated text can often contain factual consistency errors, such as contradictions with input information or hallucinations irrelevant to the context (Cao et al., 2018; Kryscinski et al., 2019; Nie et al., 2019a; Tan et al., 2020; Maynez et al., 2020; Deng et al., 2021).

It is thus crucial to develop automatic metrics that evaluate factual consistency of a claim (e.g., generated text) with regard to a context (e.g., model input). The evaluation, however, has long been a challenge. Recent work has devised various metrics based on specific pretrained functions, such as natural language inference (NLI) (Honovich et al., 2022a; Mishra et al., 2021; Kryscinski et al., 2020; Utama et al., 2022; Laban et al., 2022) and question answering (QA) (Durmus et al., 2020; Fabbri et al., 2022; Honovich et al., 2021; Fabbri et al., 2022). Specifically, an NLI-based metric measures if the claim is entailed by the context; while a QA-based metric first creates (question, answer) pairs from the claim and then checks if answering the questions with a QA model conditioning on the context will lead to the same answers.

However, by relying on specific functions trained with only narrow data (i.e., NLI or QA datasets), previous metrics have limited generalizability and fail to apply to diverse evaluation scenarios, including different types of factual consistency errors and varying lengths and characteristics of contexts/claims from different tasks and domains. For instance, a metric trained exclusively with NLI data of sentences in a certain domain tends to have difficulty in evaluating summaries of long documents in a different domain (Mishra et al., 2021; Laban et al., 2022). The limitations motivate a more holistic metric that develops a general understanding of factual consistency and generalizes to diverse evaluation scenarios.

In this paper, we propose AlignScore, a new general factual consistency metric based on a unified text-to-text information alignment function. In particular, we unify a wide range of data sources, and use the massive diverse data to train a general information alignment model that estimates an alignment score given two arbitrary text pieces. More specifically, we reformat and aggregate 15 datasets from 7 popular language tasks, including NLI, QA, paraphrasing, fact verification, information retrieval, semantic similarity, and summarization. This results in a total of 4.7M training examples with diverse characteristics, and yields an alignment function with great generalizability. We then build AlignScore using the alignment function as a building block. In particular, to handle long text and accommodate the different roles of context and claim, we develop a splitting strategy that breaks a context into coarse-grained chunks and a claim into fine-grained sentences. Aggregating the alignment scores between context-chunks and claim-sentences leads to the final factual consistency score.

In our experiments, we build AlignScore by finetuning the lightweight RoBERTa models (125M and 355M) for alignment. We evaluate AlignScore on the latest large-scale evaluation benchmarks, including SummaC (Laban et al., 2022), TRUE (Honovich et al., 2022b), and other testbeds, which contain a total of 22 challenging evaluation datasets. Our approach substantially outperforms previous state-of-the-art metrics in terms of different quality measures. Notably, our metric (355M) is on par with, and sometimes even much better than latest metrics based on orders-of-magnitude larger language models (e.g., ChatGPT and GPT-4). In particular, AlignScore shows strong generalizability on the 19 zero-shot datasets that were never seen during the alignment function training. We also conduct extensive ablation studies to demonstrate the effectiveness of the context splitting strategy and other modeling choices.

Related Work

Traditionally, generative systems are evaluated using n-gram based metrics (Papineni et al., 2002; Lin, 2004; Banerjee and Lavie, 2005; Popović, 2015). Recently, factual consistency metrics are often use task-specific language understanding capabilities, such as NLI and QA. To improve performance when evaluating generative tasks with long texts, NLI-based metrics adopt training sets with long premises (Honovich et al., 2022a; Mishra et al., 2021), use large synthetic datasets (Kryscinski et al., 2020; Utama et al., 2022), or use sentence level evaluation (Laban et al., 2022). A separate line of research formulates factual consistency evaluation as QA (Durmus et al., 2020; Fabbri et al., 2022; Honovich et al., 2021; Fabbri et al., 2022). Other consistency evaluation methods that use pretrained language models (LMs) include embedding matching (Zhang et al., 2020; Deng et al., 2021), finetuning LMs to directly regress human evaluation scores (Sellam et al., 2020), and using LMs to score candidates based on weighted log probability (Yuan et al., 2021; Liu et al., 2022). CTC (Deng et al., 2021) develops a suite of text generation evaluation metrics based on the similar concept of alignment. Yet we define alignment in a more general way to enable integration of diverse training data, and deliver AlignScore as a more effective metric focusing on factual consistency. Concurrent work proposes to combine large language models (LLMs) with prompting to evaluate different aspects of generated text, including factual consistency (Fu et al., 2023; Liu et al., 2023; Gao et al., 2023). Our proposed AlignScore shows stronger performance with a much smaller model size.

Recent work converts related but different tasks into the same input-output format to train unified models. Raffel et al. (2020) propose to unify text generation tasks into a text-to-text conditional generation problem. Sanh et al. (2022) further show that the text-to-text generation framework, combined with natural language prompting, improves zero-shot task generalization to unseen tasks. Zhong et al. (2022) develop a unified automatic evaluation metric by framing different aspects of NLG evaluation as a Boolean Question Answering problem. Recent studies also present task unification as an effective approach to improve model performance and generalizability in multi-modal tasks (Xie et al., 2022; Zhang et al., 2021; Wang et al., 2022).

Methods

We introduce the AlignScore metric built on top of a unified alignment function. We first train the alignment function by unifying a large diversity of data sources (Section 3.1). We then define AlignScore by combining the alignment function with a new context/claim splitting and aggregation strategy (Section 3.2).

Given two pieces of text $\boldsymbol{a}$ and $\boldsymbol{b}$, we consider $\boldsymbol{b}$ to be aligned with $\boldsymbol{a}$ if all information in $\boldsymbol{b}$ is present in $\boldsymbol{a}$ and does not contradict $\boldsymbol{a}$. Conceptually, we model information alignment as a function that maps the text pair $(\boldsymbol{a},\boldsymbol{b})$ to a label $y$ that characterizes the level of alignment:

A holistic and generalizable alignment function must account for all types of consistency errors, domains, and data distributions. Therefore, in order to learn the alignment function, we want to adapt and aggregate diverse language tasks to form a unified alignment training corpus (Figure 1). In this work, we collect 15 datasets spanning 7 well-established tasks, including NLI, fact verification, paraphrase, semantic textual similarity, QA, information retrieval, and summarization. We present an overview of these datasets in Table 1 and include more details in Section A.1 and A.2 in the appendix.

The vast diversity of input/output formats across the above tasks poses significant challenge for unifying them into a uniform alignment training corpus. To unify input formats, we convert each sample into a text pair $(\boldsymbol{a},\boldsymbol{b})$. For tasks that do not cleanly fit into the text pair format, such as QA (where each sample contains a question, an answer, and a context) and information retrieval (where each sample contains a query, an answer, and a supporting document), we use a sequence-to-sequence model (Song, 2022) to convert the question answer pair into a single declarative sentence (underlined items in Figure 1; See Section C.1 for examples).

To unify output formats, while it is possible to transform all tasks into binary classification, instead we convert them into a set of related alignment problems to preserve as much information as possible from the original datasets (Figure 1). Specifically, we devise 3 options for the alignment label $y$:

More concretely, for tasks that come with discrete labels, depending on their setup, the alignment function predicts either the binary classification label $y_{\text{bin}}$ (paraphrase, QA, information retrieval, and summarization) or the 3-way classification label $y_{\text{3way}}$ (NLI, and fact verification); for tasks with continuous labels (semantic textual similarity), the alignment function predicts the regression label $y_{\text{reg}}$. Here a higher $y_{\text{reg}}$ indicates that more information in $\boldsymbol{b}$ is supported by $\boldsymbol{a}$.

We build the alignment model consisting of a language model (e.g., RoBERTa; Liu et al., 2019) and 3 individual linear layers as the 3-way classification ($y_{\text{3way}}$), binary classification ($y_{\text{bin}}$), and regression ($y_{\text{reg}}$) heads. First, we feed into the language model the concatenation of the text pair $(\boldsymbol{a},\boldsymbol{b})$ and use the contextual embedding of the special begin-of-sentence token as the encoded representation, $\boldsymbol{h}$. Then, the classification and regression heads map $\boldsymbol{h}$ into an estimation of $y_{\text{3way}}$, $y_{\text{bin}}$, and $y_{\text{reg}}$ through logistic regression and linear regression, respectively. We use cross entropy loss for both 3-way and binary classification, and mean squared error loss for regression. The joint loss function is:

where $\lambda_{1},\lambda_{2},\lambda_{3}$ are scalar weights. In our experiments, we set $\lambda_{1}=\lambda_{2}=\lambda_{3}=1$.

2 The AlignScore Metric

As the definition of factual consistency is closely related to the information alignment problem, one naive way of building a factual consistency metric is simply using the alignment model to estimate the alignment score of the text pair (context, claim). However, this approach (also referred to as "document level evaluation"; Laban et al., 2022) has several drawbacks.

First, generative tasks often contain long inputs, especially long contexts, that go beyond the input length limit of a language model (e.g., source documents in summarization tasks can easily exceed the 512-token limit of a RoBERTa model). Consequently, if long inputs are not explicitly handled (Kryscinski et al., 2020; Mishra et al., 2021), language-model-based metrics could silently drop important information because of truncation.

Second, information contained in a claim often spreads across multiple sentences in the context. To verify the factual consistency of a claim, a metric needs access to long context spans. Therefore, evaluating the claim against individual context sentences (as in previous sentence level evaluation; Laban et al., 2022; Amplayo et al., 2022) can degrade metric performance as paragraph- and document-level semantic information is lost.

Third, humans typically assign consistency scores in a continuous spectrum that reflect the amount of consistency errors in the samples. Similarly, good metrics should produce fine-grained scores. Unfortunately, as classification tasks make up most of the training data (only semantic textual similarity datasets provide continuous labels), our alignment model tends to assign scores close to the two extremes, limiting its effectiveness if used directly as a factual consistency metric.

Conceptually, to resolve the first challenge, we need to split the context into chunks such that when concatenated with a claim, the resulting sequence does not exceed the input length limit. By picking a large enough chunk size, we allow the model to reason over longer context spans, mitigating the second issue. Since sentences in a claim tend to be self-contained statements, an effective way to make the metric produce more fine-grained scores is to evaluate claim sentences independently of each other Laban et al. (2022). Specifically, for each sentence in the claim (green rectangles in Figure 2), we evaluate it against all context chunks (yellow rectangles in Figure 2) using the alignment function. Then, we select the highest alignment score (lines labeled with numbers in Figure 2) for each claim sentence. Intuitively, this step identifies the context chunk that most strongly supports each claim sentence, and the highest score reflects how well the claim sentence is supported. Finally, we use the average value of all highest scores as the factual consistency score. This addresses the third challenge, as taking the average prevents a single inconsistent claim sentence from dominating the final score. Alternatively, the average value of highest scores can be roughly interpreted as "the proportion of the claim that are factually consistent with respect to the context", which naturally leads to a more fine-grained metric. As we show in experiments, our novel chunk level evaluation method consistently outperforms document level (which risks truncation) and sentence level evaluation.

where $\boldsymbol{o}$ is the context, $\boldsymbol{l}$ is the claim, $\{\boldsymbol{o^{\prime}_{i}}\}$ is the set of context chunks, $\{\boldsymbol{l^{\prime}_{j}}\}$ is the set of claim sentences, and $\operatorname*{alignment}(\cdot)$ is the probability of the model predicting the aligned label in the 3-way classification setting. In practice, for RoBERTa models (that have an input length limit of 512 tokens) we split the context into chunks at sentence boundaries such that each chunk contains roughly 350 tokens. We use the output of the 3-way classification head, our ablation studies reveal that it performs better than the binary classification head and the regression head (Section 4.5).

Experiments

In this section, we evaluate AlignScore on a wide range of benchmarks and show it consistently outperforms existing metrics (Section 4.1-4.4). We also conduct extensive ablation study in Section 4.5.

We use RoBERTa (Liu et al., 2019) to implement the alignment model. We denote AlignScore based on RoBERTa-base/large as AlignScore-base/large.

We follow common practice (Liu et al., 2019; Devlin et al., 2019) and train the model for 3 epochs with a batch size of 32 in all the experiments. Training samples are randomly sampled across the converted upstream NLP tasks. Due to resource constraints we only use the first 500k samples in each dataset for training, resulting in a total of 4.7 million training samples. Training details are listed in Appendix A.3.

2 Benchmarks

Following Deng et al. (2021), Fabbri et al. (2022), Zhong et al. (2022) and Gabriel et al. (2021), we evaluate factual consistency metrics using TRUE benchmark (Honovich et al., 2022a) (consists of 11 datasets in diverse domains), SummaC benchmark (Laban et al., 2022) (includes 6 large summarization datasets), and a set of other latest datasets including XSumFaith (Maynez et al., 2020), SummEval (Fabbri et al., 2021), QAGS-XSum (Wang et al., 2020), QAGS-CNNDM (Wang et al., 2020), FRANK (Pagnoni et al., 2021) and SamSum (Gliwa et al., 2019).

SummaC benchmark standardizes the task of summary inconsistency detection by casting it as a binary classification problem. Following Laban et al. (2022), we 1) tune the threshold of metrics on the validation sets, and then compute the balanced accuracy (Brodersen et al., 2010) on the test sets, 2) report the AUC-ROC (Bradley, 1997) of each metric. TRUE benchmark covers summarization, dialogue, paraphrase and fact verification tasks. It also assigns binary labels to samples based on whether the entire claim is factually consistent with the context. We report AUC-ROC of each metric following Honovich et al. (2022a). We also collect 6 popular factual consistency evaluation datasets, namely XSumFaith, SummEval, QAGS-XSum, QAGS-CNNDM, FRANK and SamSum. We compute instance-level Pearson, Spearman, and Kendall’s tau correlation coefficients between metric scores and human annotated consistency scores.

3 Baselines

We compare AlignScore with state-of-the-art metrics, which we categorize into question answering (QA), similarity matching, regression, NLI, and miscellaneous. We use open-source code and models released by authors. Additionally, we also compare with latest LLM-based metrics.

QA Based Metrics adapt question generation (QG) and question answering (QA) models to automatically evaluate factual consistency. We include the latest QAFactEval (Fabbri et al., 2022), QuestEval (Scialom et al., 2021), and FEQA (Durmus et al., 2020) as our baselines.

Similarity Matching Based Metrics vary in their granularity and matching functions. We report BLEU (Papineni et al., 2002) and ROUGE-1/2/L (Lin, 2004), which compute token-level string matching scores. We also include the named-entity level metric NER-Overlap introduced in Laban et al. (2022). BERTScore (Zhang et al., 2020) uses token-level embedding to compute scores, for which we use the best variant (microsoft/deberta-xlarge-mnli) recommended by the authorshttps://github.com/Tiiiger/bert_score. We also use SimCSE (Gao et al., 2021) as sentence-level embedding matching function, with the best released model sup-simcse-roberta-largehttps://github.com/princeton-nlp/SimCSE.

Regression Based Metrics learn to estimate ground truth scores directly. We use BLEURT (Sellam et al., 2020) with its recommended checkpoint (BLEURT-20)https://github.com/google-research/bleurt as our baseline.

NLI Based Metrics methods also vary in their granularity. We use a RoBERTa-large (Liu et al., 2019) model finetunedhttps://huggingface.co/roberta-large-mnli on MultiNLI (Williams et al., 2018b) as a baseline for document-level evaluation, where the model evaluates a candidate against the entire context. Our baselines also include the DAE (Goyal and Durrett, 2020) metric, which decomposes text at the level of dependency arcs. For sentence-level baseline, we use SummaC-ZeroShot and SummaC-Conv introduced in the SummaC Benchmark (Laban et al., 2022) and FactCC (Kryscinski et al., 2020) which is trained on synthetic data.

Miscellaneous Besides the above metrics, we also use competitive metrics including UniEval (Zhong et al., 2022), CTC (Deng et al., 2021), BARTScore (Yuan et al., 2021) and BLANC (Vasilyev et al., 2020) as baselines.

UniEval is a unified multi-dimensional metric, capable of evaluating different aspects of text generation. We use the Consistency variant as the baseline. Deng et al. (2021) propose CTC, which is based on token-level information alignment. We use its discriminative variant trained on synthetic CNN/DailyMail (See et al., 2017) (D-CNNDM) as our baseline. For BARTScore, we use the pretrained BART-Large-CNNhttps://github.com/neulab/BARTScore checkpoint.

LLM-Based Metrics Concurrent work proposes to utilize LLMs for NLG evaluation. GPTScore uses the log probability of an LLM generating the target text conditioned on the prompt as the metric score (Fu et al., 2023). G-EVAL first augments its prompts with chain-of-thoughts and then evaluates texts by form-filling (Liu et al., 2023). Gao et al. (2023) uses ChatGPT in place of human annotators in four popular human evaluation setups (ChatGPT in Table 5). As we directly compare with correlation coefficients reported by Fu et al. (2023); Liu et al. (2023); Gao et al. (2023), results on some datasets are not available.

4 Results

We report AUC-ROC on the test set of the SummaC Benchmark in Table 2. A higher AUC-ROC score indicates the metric is better at detecting factual consistency errors. Our AlignScore-large achieves the best average performance on the SummaC benchmark, scoring the highest in 4 out of 6 datasets. We also present the balanced accuracy in Appendix (Table 9), where AlignScore-large also establishes new state-of-the-art results.

4.2 Results on TRUE Benchmark

The results on the TRUE benchmark are shown in Table 3, where AlignScore-large gets the highest average AUC-ROC score. It outperforms baselines on 7 out of 11 tasks while staying competitive on the rest. For a fair comparison, we also report the average AUC-ROC (denoted as AVG-ZS) excluding datasets that the alignment function is trained on (PAWS, VitaminC and FEVER). The performance of AlignScore remains to be on top, outperforming strong baselines like QAFactEval, UniEval, and SummaC-CONV. This demonstrates AlignScore generalizes well to unseen data (e.g., DialFact dataset in the dialogue domain).

4.3 Results on Other Datasets

We present Pearson correlation coefficients of various metrics on other factual consistency datasets in Table 4. We also report Spearman correlation and Kendall’s tau coefficients in Appendix (Table 10 and 11). The AlignScore-large metric outperforms previous metrics in terms of overall performance, including the competitive QAFactEval and UniEval metrics, dominating 6 out of 7 datasets. We note that DAE and QuestEval perform better on XSumFaith dataset. Similar to Fabbri et al. (2022), we speculate it is because the relatedness between the token-level annotation of XSumFaith and the fine-grained metrics.

We also compare our metric with LLM-based metrics in Table 5. Result shows AlignScore has comparable performance with LLM-based metrics on SummEval. And it outperforms LLM-based metrics on QAGS-XSum and QAGS-CNNDM, showing the capability and efficiency of our proposed metric.

5 Ablation Study

To understand 1) which classification head is more suitable for factual consistency evaluation, 2) which splitting method is more effective, and 3) which upstream NLP task contributes the most to the superior performance of AlignScore, we conduct 3 ablation studies. The experiments in this section are all based on AlignScore-base.

We keep the same splitting method as in Section 3.2 and change the heads that generate alignment scores. We first use the regression head (AlignScore-base-REG) and the binary classification head (AlignScore-base-BIN). Then, we compare these two heads with our proposed AlignScore-base, which adopts the 3-way classification head. We present the results in Figure 3, which shows the 3-way classification head consistently performs better than the regression head and the binary classification head.

Then, we keep the 3-way classification head and change the splitting method. Following Amplayo et al. (2022), we implement SMART-L and SMART-N, and use our alignment model as the sentence matching function. SMART-L uses sentence-level evaluation and aggregates the alignment scores through a soft version of Longest Common Subsequence (LCS), while SMART-N aggregates using greedy matching between N-sentences. In our experiments, we set N=1. We also implement AlignScore without any splitting (denoted as AlignScore-base-DOC) where the inputs are directly fed into the model. The result in Figure 4 shows that our chunk level splitting method performs best compared to the other 3 methods. It demonstrates that our splitting method helps AlignScore capture salient information from long contexts.

We study the contribution of each upstream NLP task by excluding one task at a time to train the alignment model. The results are shown in Figure 5. When the QA task is removed, the performance of the metric is the worst, indicating QA datasets make the biggest contribution to metric performance. Similarly, fact verification task has the second largest contribution. Surprisingly, with the removal of the NLI task, the model performs better on a majority of benchmarks, showing the NLI task plays a negative role in the training. We speculate that it is because 1) premises and hypothesises in NLI datasets are generally shorter, which differs from most factual consistency benchmarks and datasets, 2) other NLP tasks have larger-scale and higher quality datasets.

Conclusion

We propose AlignScore, a holistic factual consistency metric based on a unified alignment function. To learn the alignment function, we adapt 7 well established language understanding tasks into a unified alignment task , resulting in 4.7M diverse training samples. Experiments show AlignScore achieves state-of-the-art performance on SummaC and TRUE Benchmark, has higher correlation with human judgements than competing metrics, and generalizes well to unseen data.

Limitations

Interpretability. Although AlignScore shows high correlation with human judgments, it is hard to interpret the reasoning behind its predictions. Therefore, an interesting future research direction is to develop interpretable factual consistency metrics that can accurately identify words or spans in the input that contain factual consistency errors and (or) produce human readable explanations justifying its predictions.

Synthetic data. Our alignment training data contains datasets augmented with synthetic data. While ablation studies show that synthetic data helps improve metric performance, our rule-based method for generating synthetic data could generate noisy data that may not accurately model the error types and distributions produced by real world generative systems. Thus, analyzing the quality of synthetic data and developing more effective ways to generate synthetic data is an interesting research topic.

Language coverage. While we show AlignScore generalize well to unseen data, it only covers a single language, English. Undoubtedly, factual consistency evaluation is also important for more resource-constrained languages or in a multilingual setting. Consequently, future research could focus on extending the Align metric to multiple languages, including resource-constrained languages.

Ethics Statement

AlignScore is intended as an automatic metric to be used in NLP research. While it has state-of-the-art performance, it can produce false positives and false negatives, and may not be appropriate for applications other than its intended use. As it is trained on publicly available datasets, the metric might be affected by biases inherent to those datasets.

References

Appendix A Implementation Details

We adapt datasets from 7 NLP tasks into the information alignment format. An overview of our unified training sets is shown in Table 1.

Tasks that cleanly fit into the form of the alignment problem, including NLI, fact verification, and paraphrase datasets are adapted by mapping the original labels into either binary or 3-way classification alignment labels. Next, we discuss how we adapt semantic textual similarity (STS), QA, and information retrieval (IR) tasks.

STS datasets contain pairs of sentences labeled with semantic similarity scores. We use STS datasets in the regression task by normalizing the score to between 0 and 1.

A QA sample consists of a context paragraph, a question, and a ground truth answer. One can derive the ground truth answer given the context and the question. To convert QA samples into a format suitable for binary classification, we use a pretrained sequence-to-sequence model to convert question-answer pairs into declarative sentences (Song, 2022; Demszky et al., 2018). Sentences generated from ground truth answers form aligned pairs with corresponding contexts, while sentences generated from wrong options form not-aligned samples. For samples with unanswerable questions, we first use a QA modelhttps://huggingface.co/valhalla/t5-base-qa-qg-hl to generate wrong answers, and then turn them into not-aligned samples using the above method.

A sample in an information retrieval dataset consists of a query-answer pair and a list of passages, some of which can be used to answer the query. Similar to QA datasets, we adapt information retrieval datasets for binary classification by converting query-answer pairs into declarative sentences and then pairing them with passages. If a passage can be used to answer the corresponding query, we consider the sample to have aligned label. Otherwise it is assigned not-aligned.

A.2 Synthetic Data

We further augment our training set with synthetic data based on the WikiText-103 corpus (Merity et al., 2017) and the WikiHow summarization dataset (Koupaee and Wang, 2018).

To generate aligned samples, we create a paraphrase of each sentence in WikiText-103 through back translation using a neural machine translation model (Junczys-Dowmunt et al., 2018). For the WikiHow dataset, we use source documents as text $\boldsymbol{a}$, and the ground truth summaries together with extractive summaries generated by an extractive summarizer (Barrios et al., 2016) as text $\boldsymbol{b}$ to form aligned samples.

Inspired by recent work in creating factually inconsistent samples (Deng et al., 2021; Kryscinski et al., 2020), we randomly mask 25% of the tokens in text $\boldsymbol{b}$ from the aligned samples and infill with a masked language modeling model Sanh et al. (2019). The resulting sentences are semantically different from the originals and are used in not-aligned samples.

A.3 Training the Alignment Model

We use the Transformershttps://huggingface.co/docs/transformers/index library to implement the proposed model, and the PyTorch Lightning framework to train our model.

The alignment model is optimized with AdamW (Loshchilov and Hutter, 2019). The learning rate is first warmed up to a peak of 1e-5, and then linearly decayed. The hyperparameters used to train AlignScore-base and AlignScore-large are shown in Table 6.

We don’t split the context and claims into chunks in the training for simplicity.

A.4 Cleaning Evaluation Datasets

Certain datasets we use for evaluation contain artifacts that could hurt model performance. Notable issues include claims having escape sequences (-LRB- and -RRB- instead of parentheses) and being uncased (all lower case) while contexts do not have escape sequences and are cased.

We use rule-based methods to remove these artifacts. Specifically, we replace escape sequences in claims with the original characters, capitalize the first letter of the first word in a sentence, and for words that appear in contexts, we fix their capitalization in the corresponding claims according to their occurrences in the contexts.

A.5 Computing Correlations

We first split the inputs to sentences with NLTK sentenizer. Then AlignScore computes the instance-level factual consistency score as stated in Section 3.2. We use scipy to compute Pearson correlation, Spearman correlation and Kendall’s tau correlation.

Appendix B Additional Experiment Details/Results

SummaC benchmark consists of 6 summarization datasets: CogenSum (Falke et al., 2019), XSumFaith (Maynez et al., 2020), Polytope (Huang et al., 2020), FactCC (Kryscinski et al., 2020), SummEval (Fabbri et al., 2021) and FRANK (Pagnoni et al., 2021). The datasets are standardized by binarizing each labels. Metrics are evaluated as classifiers on SummaC benchmark.

The SummaC Benchmark considers samples in PolyTope with AdditionDefined as: Unnecessary and irrelevant snippets from the source are included in the summary, OmissionDefined as: Key point is missing from the output, Inaccuracy IntrinsicDefined as: Terms or concepts from the source are misrepresented and thus unfaithful., Inaccuracy ExtrinsicDefined as: The summary has content not presented in the source and factually incorrect and Positive-Negative AspectDefined as: The output summary represents positive statements whereas the source segment is negative, and vice versa. errors to be negative samples. However, Addition and Omission do not imply factual consistency errors. Thus, we only consider samples with Inaccuracy Intrinsic, Inaccuracy Extrinsic and Positive-Negative Aspect errors to be factually incorrect. The reported PolyTope result uses this definition of errors.

We also report balanced accuracy, which deals with imbalanced datasets, in Table 9.

B.2 TRUE Benchmark

TRUE benchmark is for evaluating factual consistency metrics in summarization, dialogue, fact-verification and paraphrasing tasks. There are totally 11 datasets in this benchmark: FRANK (Pagnoni et al., 2021), SummEval (Fabbri et al., 2021), MNBM (Maynez et al., 2020), QAGS-CNNDM (Wang et al., 2020), QAGS-XSum (Wang et al., 2020), BEGIN (Dziri et al., 2022), Q${}^{2}_{\text{dataset}}$ (Honovich et al., 2021), DialFact (Gupta et al., 2022), PAWS (Zhang et al., 2019), FEVER (Nie et al., 2019b; Thorne et al., 2018) and VitaminC (Schuster et al., 2021). TRUE also treats factual consistency evaluation as a binary classification task and reports AUC-ROC.

The full names of the datasets in Table 3 are listed in Table 7.

B.3 Other Datasets

In addition to the Pearson correlation reported in Table 4, we also report the Spearman correlation and Kendall’s tau correlation on 9 datasets in Table 10 and 11, respectively. The full names of the abbreviations in Table 4, Table 10 and Table 11 are listed in Table 8.

We notice that the BLEU metric has comparable performance with some neural model based methods, which seems to contradict some previous findings. We attribute it to the case matching in the pre-processing, since BLEU is case sensitive.

Appendix C Sample Training Data

We show converted SQuAD v2 (Rajpurkar et al., 2018) samples below to illustrate the process of converting QA samples into the alignment format (discussed in Section A.1). Concretely, questions and answers are combined into declarative claims using a sequence-to-sequence model (Song, 2022; Demszky et al., 2018).

The Times Literary Supplement (TLS) first appeared in 1902 as a supplement to The Times, becoming a separately paid-for weekly literature and society magazine in 1914. The Times and the TLS have continued to be co-owned, and as of 2012 the TLS is also published by News International and cooperates closely with The Times, with its online version hosted on The Times website, and its editorial offices based in Times House, Pennington Street, London.

The editorial offices of The Times Literary Supplement is based in what location in London?

The editorial offices of The Times Literary Supplement is based in Times House, Pennington Street in London.

The 25,000 cotton growers in the United States of America are heavily subsidized at the rate of $2 billion per year although China now provides the highest overall level of cotton sector support. The future of these subsidies is uncertain and has led to anticipatory expansion of cotton brokers’ operations in Africa. Dunavant expanded in Africa by buying out local operations. This is only possible in former British colonies and Mozambique; former French colonies continue to maintain tight monopolies, inherited from their former colonialist masters, on cotton purchases at low fixed prices.

How many subsidized cotton growers are in the US?

25,000 subsidized cotton growers are in the US.

On October 28, 2015, IBM announced its acquisition of digital assets from The Weather Company—a holding company of Bain Capital, The Blackstone Group and NBCUniversal which owns The Weather Channel, including its weather data platforms (such as Weather Services International), websites (Weather.com and Weather Underground) and mobile apps. The acquisition seeks to use Watson for weather analytics and predictions. The acquisition does not include The Weather Channel itself, which will enter into a long-term licensing agreement with IBM for use of its data. The sale closed on January 29, 2016

When did the sale of Weather Company assets close?

The sale of Weather Company assets closed on January 29, 2016.

The dipole component of the magnetic field at the magnetic equator of Neptune is about 14 microteslas (0.14 G). The dipole magnetic moment of Neptune is about 2.2 × 1017 T·m3 (14 $\mu$T·RN3, where RN is the radius of Neptune). Neptune’s magnetic field has a complex geometry that includes relatively large contributions from non-dipolar components, including a strong quadrupole moment that may exceed the dipole moment in strength. By contrast, Earth, Jupiter and Saturn have only relatively small quadrupole moments, and their fields are less tilted from the polar axis. The large quadrupole moment of Neptune may be the result of offset from the planet’s centre and geometrical constraints of the field’s dynamo generator.

What is the dipole component of the magnetic field at the magnetic equator of neptune?

The dipole component of the magnetic field at the magnetic equator of neptune is 14 microteslas (0.14 G).

Qing dynasty rule in Tibet began with their 1720 expedition to the country when they expelled the invading Dzungars. Amdo came under Qing control in 1724, and eastern Kham was incorporated into neighbouring Chinese provinces in 1728. Meanwhile, the Qing government sent resident commissioners called Ambans to Lhasa. In 1750 the Ambans and the majority of the Han Chinese and Manchus living in Lhasa were killed in a riot, and Qing troops arrived quickly and suppressed the rebels in the next year. Like the preceding Yuan dynasty, the Manchus of the Qing dynasty exerted military and administrative control of the region, while granting it a degree of political autonomy. The Qing commander publicly executed a number of supporters of the rebels and, as in 1723 and 1728, made changes in the political structure and drew up a formal organization plan. The Qing now restored the Dalai Lama as ruler, leading the governing council called Kashag, but elevated the role of Ambans to include more direct involvement in Tibetan internal affairs. At the same time the Qing took steps to counterbalance the power of the aristocracy by adding officials recruited from the clergy to key posts.

What did the Qing commander do in 1732 and 1728?

The Qing commander publicly executed a number of supporters of the rebels in 1732 and 1728.