CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation

Introduction

Controlled text generation aims to generate texts under some control variables, including pre-specified content prefixes and attribute labels (such as sentiments and topics). Controlled text generation has been significantly advanced by large-scale pre-trained models with respect to generation quality and various control variables Keskar et al. (2019); Dathathri et al. (2020); Yang and Klein (2021); Liu et al. (2021a); Chan et al. (2021).

Despite the great success of these generation models, it becomes critical to evaluate the quality of generated texts accurately. Most of the existing studies adopt unsupervised and supervised metrics to measure the quality of generated texts under different combinations of control variables Dathathri et al. (2020); Chan et al. (2021). The evaluation is commonly conducted in a reference-free setting because it is challenging to collect sufficient high-quality references for each input of control variables in this open-ended text generation task Dathathri et al. (2020).

However, both unsupervised and supervised metrics have shown limitations in the evaluation of controlled text generation: 1) Unsupervised metrics such as perplexity Brown et al. (1992) can only provide task-agnostic evaluation regarding the overall quality of generated texts. However, controlled text generation tasks typically involve multiple evaluation aspects Deng et al. (2021), including the quality of generated texts themselves and the relationship between generated texts and control variables. It is thus not surprising that existing unsupervised metrics without multi-aspect interpretability have low correlations with human judgments Hashimoto et al. (2019). 2) Supervised metrics are commonly trained on the datasets of specific tasks to measure the corresponding aspects of generated texts (e.g., evaluating whether a generated text is accordant with the sentiment label) Dathathri et al. (2020); Chan et al. (2021). This may cause over-fitting to task-specific data and degrade the generalization ability of metrics Garbacea et al. (2019), thereby giving unstable evaluation of generated texts from different models or with different qualities Guan and Huang (2020).

To deal with the above issues, we propose an unsupervised reference-free metric called CTRLEval for evaluating controlled text generation models. This metric performs evaluation from different aspects without any training on task-specific data. Specifically, we formulate the evaluation of each aspect into “fill-in-the-blank” tasks whose input and output patterns can be designed based on the definition of the aspect. Then, we utilize a pre-trained model whose pre-training task is text infilling (such as PEGASUS Zhang et al. (2020a)) as our base model, and fuse the generation probabilities from these “fill-in-the-blank” tasks as the evaluation result. To alleviate the potential bias caused by the task design Zhao et al. (2021), we devise multiple text infilling tasks for each aspect and use the weighted sum of all the results as the final score. In this paper, we consider three aspects which are commonly used to measure the performance of controlled text generation models, including coherence Yuan et al. (2021), consistency Rashkin et al. (2020), and attribute relevance Dathathri et al. (2020). These evaluation aspects cover both the quality of generated texts and the relationship between generated texts and different control variables, which can provide a comprehensive evaluation result for controlled text generation. Experimental results show that our metric can maintain the generalization ability and achieve stable performance faced with model drift and quality drift.

We propose an unsupervised reference-free metric called CTRLEval for evaluating controlled text generation. This metric formulates three evaluation aspects (i.e., coherence, consistency, and attribute relevance) into multiple text infilling tasks, and utilizes the ensemble of generation probabilities from a pre-trained language model as the evaluation results.

We conduct experiments on two benchmark tasks including sentiment-controlled and topic-controlled text generation based on our collected evaluation set. Experimental results show that our proposed metric has higher correlations with human judgments, while obtaining better generalization of evaluating generated texts from different models and with different qualities.

Related Work

Early studies on controlled text generation adopt attribute label embeddings Ficler and Goldberg (2017); Zhou et al. (2018) or latent variables Hu et al. (2017); Ke et al. (2018); Zhou and Wang (2018) to learn the complex relationship between control variables and generated texts. With the development of large-scale generative pre-trained models, it is costly to re-train or fine-tune pre-trained models on the corpora with attribute annotations Keskar et al. (2019). Recent works resort to decoding-time methods and directly make pre-trained models generate texts towards desired attributes during inference, including PPLM Dathathri et al. (2020), GeDi Krause et al. (2020), FUDGE Yang and Klein (2021) and DEXPERTS Liu et al. (2021a). These works rely heavily on human evaluation because existing reference-free metrics including unsupervised and supervised ones are shown to have evident limitations for evaluating controlled text generation Dathathri et al. (2020).

2 Evaluation Metric for Text Generation

Automatic evaluation metrics are important for natural language generation tasks, which can be simply divided into referenced, reference-free (also known as unreferenced) and hybrid metrics: 1) Referenced metrics usually measure the relevance between generated texts and reference texts via lexicon overlap (such as BLEU Papineni et al. (2002), METEOR Banerjee and Lavie (2005) and ROUGE Lin (2004)) or embedding similarity (such as MoverScore Zhao et al. (2019), BERTScore Zhang et al. (2020b) and MARS Liu et al. (2021b)). 2) Reference-free metrics directly evaluate the quality of generated texts without references. Since unsupervised metrics like perplexity Brown et al. (1992) and distinct n-grams Li et al. (2016) can only provide a task-agnostic result which correlates weakly with human judgments Hashimoto et al. (2019); Tevet and Berant (2021), most of the reference-free metrics resort to supervised models. Specifically, they are trained to fit human-annotated ratings / labels (such as discriminator scores Shen et al. (2017)) or distinguish human-written texts from negative samples (such as UNION Guan and Huang (2020)). 3) Hybrid metrics contain both referenced and reference-free scores, such as RUBER Tao et al. (2018); Ghazarian et al. (2019), BLEURT Sellam et al. (2020) and BARTScore Yuan et al. (2021).

Compared with existing reference-free metrics which are unsupervised, our metric can support the evaluation of generated texts from different aspects via the full utilization of pre-trained models and the formulation of text infilling tasks, which fits the evaluation protocol of controlled text generation well. Also, in contrast with supervised reference-free metrics, our metric can avoid over-fitting task-specific data and maintain better generalization ability to evaluate generated texts from different models and with different qualities.

Method

Given the input $I=(X,a,Y)$ which consists of a content prefix $X$, an attribute label $a$, and a generated text $Y$, our goal is to acquire three evaluation results for coherence, consistency and attribute relevance, respectively.

As shown in Figure 1, our main idea is to formulate each evaluation aspect into multiple text infilling tasks and utilize the ensemble of the scores from each task as the final evaluation results. We denote each text infilling task as a pattern evaluator , which means evaluation with different input and output patterns. Inspired by the recent works on pattern-exploiting training Schick and Schütze (2021a, b) and prompt tuning Gu et al. (2021), we define each pattern evaluator as $E=(f,g)$, which consists of two pattern functions to build the input and output sequence of text infilling tasks, respectively. The score of each pattern evaluator is acquired from the generation probability of the encoder-decoder pre-trained language model whose pre-training task is to generate the masked part from the remaining texts of the input. For each aspect, we devise multiple pattern evaluators to alleviate the potential bias caused by the pattern design Zhao et al. (2021), and weight the scores of all the evaluators to obtain the final result:

where $N_{E}$ is the number of pattern evaluators, $S(I)$ denotes the overall score for each aspect, $\beta_{j}(I)$ is a factor to weight the pattern evaluators of the corresponding aspect and $s_{j}(I)$ indicates the score of each pattern evaluator based on the generation probability of the pre-trained model.

2 Evaluation Aspect

Coherence aims to measure whether the sentences in the generated text are semantically relevant to compose a coherent body Vakulenko et al. (2018); Yuan et al. (2021), which reflects the quality of the generated text itself. Assume that the generated text $Y$ consists of $M$ sentences, i.e., $Y=(Y_{1},Y_{2},\cdots,Y_{M})$, we devise $M$ pattern evaluators $E_{j}=(f_{j},g_{j})(1\leq j\leq M)$ to measure the relevance between each sentence and all the remaining sentences:

where $Y_{\backslash j}$ indicates the generated text $Y$ with the $j$-th sentence replaced by a mask token [M]. The score of each pattern evaluator $E_{j}$ can be computed via the log probability of the pre-trained model $P_{\theta}$:

Since specific and informative sentences are more likely to impact the quality of the whole text, we adopt normalized inverse sentence frequency (NISF) Zhang et al. (2018) of the output sentence which can reflect its specificity to weight each pattern evaluator:

where the inverse sentence frequency (ISF) of $Y_{j}$ is computed by the maximum inverse word frequency (IWF) of the words in $Y_{j}$. We estimate IWF on a general corpus BookCorpus Zhu et al. (2015), which is commonly adopted as the pre-training dataset in the existing works Devlin et al. (2019):

where $|C|$ indicates the total number of sentences in BookCorpus and $f_{w}$ denotes the number of sentences containing the word $w$. Thus, the evaluation result of coherence can be obtained by the ensemble of the scores from all the pattern evaluators:

2.2 Consistency

Consistency aims to evaluate whether the generated text is consistent to the content prefix Celikyilmaz et al. (2020); Rashkin et al. (2020). We devise two symmetric pattern evaluators $E_{X\rightarrow Y}$ and $E_{Y\rightarrow X}$ to evaluate the consistency between the content prefix and the generated text as follows:

where $Y_{\backslash X}$ denotes the remaining part of the generated text without the prefix. Similar to coherence, we still adopt the log probability of the pre-trained model as the pattern evaluator’s score and weight them with normalized inverse sentence frequency to obtain the final result of consistency:

2.3 Attribute Relevance

Attribute relevance aims to measure whether the generated text satisfies the attribute label Dathathri et al. (2020). To probe the relevance between generated texts and attribute labels, we first introduce a verbalizer $v(\cdot)$ which maps all the attribute labels $a$ in the attribute set $\mathcal{A}$ to the corresponding words Schick and Schütze (2021a). Then, we design the pattern evaluators $E_{j}=(f_{j},g_{j})(1\leq j\leq N_{AR})$ where $f_{j}(\cdot)$ adds prompts and a mask token to the generated text, and $g_{j}(\cdot)$ is set to be a verbalizer:

where $\textrm{Concat}(\cdot)$ indicates the concatenation of the prompt, the mask token, and the generated text in some order. We give an example for the pattern design of attribute relevance which is also shown in Figure 1. In this example, the attribute is set to be the sentiment $\mathcal{A}=\{\textrm{Positive,Negative}\}$, while the patterns are designed as $f(I)=\textrm{``}Y\;\textrm{It was }\texttt{[M]}.\textrm{''}$ and $g(I)=v(\textrm{Positive/Negative})=\textrm{good/bad}$.

Inspired by the existing works Schick and Schütze (2021a), we use the generation probability of the corresponding label word over all the label words as the score of the pattern evaluator:

Based on the assumption that the pattern evaluator is adequate to measure the data sample if the words of all the attribute labels are easily generated, we devise the unnormalized weighted score of each evaluator as the sum of generation probabilities over all the attribute labels:

Similarly, the evaluation result of attribute relevance can be acquired by the weighted sum of all the pattern evaluators’ scores:

Experiment

Since there is no standard benchmark dataset for evaluating controlled text generation, we construct an evaluation set to measure the correlation between automatic metrics and human judgments.

Task: We choose sentiment-controlled and topic-controlled text generation as the benchmark tasks, which are widely used in the existing works Dathathri et al. (2020); Chan et al. (2021). These two tasks require the models to generate texts conditioned on the given prefixes and sentiment / topic labels, respectively. In the task of sentiment-controlled text generation, we follow PPLM Dathathri et al. (2020) and CoCon Chan et al. (2021) to adopt 15 prefixes and 2 sentiment labels (i.e., positive and negative). As for topic-controlled text generation, we follow CoCon Chan et al. (2021) to adopt 20 prefixes and 4 topic labels (i.e., computers, politics, religion, and science).

Generation Models: We consider various generation models including CTRL Keskar et al. (2019), PPLM Dathathri et al. (2020), GeDi Krause et al. (2020), and CoCon Chan et al. (2021). These representative models support both the sentiment-controlled and topic-controlled text generation tasks, and cover different levels of generation abilities. We make these models generate 3 different samples for each unique pair of prefixes and attribute labels. We set the maximum length of generated texts to be 80 and remove the last sentence if it is not complete. We directly use the generation results if they have been released by the original papers. Otherwise, we run the original codes to obtain the generation results.

Human Annotation: We collect human ratings on the generated texts from Amazon Mechanical Turk (AMT). Each survey of AMT contains a prefix, an attribute label, and five generated texts including (a) four generated texts from the above four models respectively, and (b) one negative sample which is constructed by perturbing (e.g. sentence shuffling and dropping) another sample from the evaluation set Guan et al. (2021). We ask annotators to rate these texts with a 1-5 Likert scale for each aspect. To control the annotation quality, we discard the submissions if the annotator assigns a higher rating to the negative sample than other texts. We ensure that each generated text contains 5 valid ratings for each aspect, where the average value of valid ratings is used as the human judgments. We also calculate Krippendorff’s $\alpha$ Krippendorff (2018) to show the agreement of human ratings, which is 0.626 / 0.622 for sentiment-controlled / topic-controlled text generation tasks, respectively.

The statistics of the evaluation set are shown in Table 1.

2 Implementation Details

We choose PEGASUS Zhang et al. (2020a) as our base model in the overall result and also explore other pre-trained models in §4.8. The hyper-parameters of Transformer blocks are the same as PEGASUS-large with 568M parameters. As for the pattern evaluators in attribute relevance involving prompts and verbalizers which need to be additionally designed, we follow BARTScore Yuan et al. (2021) to first adopt manually devised seed prompts and verbalizers in the existing works Schick and Schütze (2021a, b), and then collect paraphrases to automatically expand our evaluator set. The statistics of pattern evaluators in attribute relevance are presented in Table 2. More details about the specific design of prompts and verbalizers are included in Appendix A.

3 Baselines

We choose several state-of-the-art reference-free metrics as our baselines:

Perplexity (PPL) Brown et al. (1992): This method calculates the perplexity of generated texts with a language model. We use GPT Radford et al. (2018) and PEGASUS Zhang et al. (2020a) as the base models since GPT is commonly used in the existing works Dathathri et al. (2020) and PEGASUS is our base model. They are denoted as PPL-GPT and PPL-PEGASUS, respectively.

Discriminator Score (DisScore) Kannan and Vinyals (2017); Chan et al. (2021): This method trains a discriminator with different objectives. We adopt the IMDB movie review dataset Maas et al. (2011) / HuffPost News category datasethttps://www.kaggle.com/rmisra/news-category-dataset Misra (2018) for sentiment-controlled / topic-controlled text generation tasks, respectively. For coherence and consistency, the discriminator is trained to distinguish human-written texts from manually constructed negative samples, where the ratio of positive and negative samples is 1:1. For attribute relevance, it is trained based on the sentiment / topic classification task, respectively Chan et al. (2021). Both the sentiment and topic discriminators are implemented based on BERT Devlin et al. (2019) and they achieve 94.15% / 91.54% on the corresponding test set, respectively.

UNION Guan and Huang (2020): This method is a self-supervised metric which is trained to distinguish human-written texts from the automatically perturbed negative samples with well-designed negative sampling strategies and multi-task learning. We use the same datasets as the discriminator score to train UNION.

BLEURT Sellam et al. (2020): This method is a supervised metric which is pre-trained on synthetic examples and then fine-tuned to fit human ratings. We used the same instruction in §4.1 to additionally annotate the generated texts to construct the training set for BLEURT, whose amount is the same as the evaluation set. There is no overlap between BLEURT’s training set and the evaluation set.

BARTScore Yuan et al. (2021): This method utilizes the generation probabilities of BART Lewis et al. (2020) to measure the relationship among sources, hypotheses, and references. Since this metric simultaneously contains referenced and reference-free parts, we only use the reference-free score in our experiments. We also use PEGASUS Zhang et al. (2020a) as the base model for a fair comparison, which is denoted as BARTScore-PEGASUS.

4 Overall Result

We follow the existing work Guan and Huang (2020); Yuan et al. (2021) to adopt Pearson ($r$), Spearman ($\rho$), and Kendall ($\tau$) correlation coefficients between automatic metrics and human judgments to measure the performance of different metrics.

The overall results on sentiment-controlled and topic-controlled text generation are shown in Table 3 and 4. We can observe that CTRLEval outperforms other baselines with a large margin, indicating the effectiveness of our metric on different evaluation aspects. In Table 4, unsupervised baselines can hardly measure the relevance between generated texts and attribute labels because they only provide a task-agnostic score which is weakly relevant to this specific aspect. For comparison, our metric, which supports the evaluation for different aspects of generated texts via the design of text infilling tasks, can obtain much better performance and even outperform the supervised baselines.

5 Ablation Study

To further investigate the effect of each module, we conduct ablation studies on the weight of pattern evaluators and the design of pattern functions. For the weight of evaluators, we use the mean, maximum and minimum values of all the evaluators as the final result rather than the weighted sum based on the factor $\beta$. As for the design of pattern functions, we fix the base model and replace our input and output patterns ($f\&g$) with those of PPL-GPT Radford et al. (2018) and BARTScore Yuan et al. (2021). The pattern functions of these ablation models are not designed for text infilling tasks. Both of them remove the mask token in the input pattern, and PPL-GPT additionally places the input pattern at the beginning of the output pattern.

The results in Table 5 show that each module in our metric contributes to the final performance. As for the weight of evaluators, we can observe that our weight factor performs better than common aggregation functions especially in consistency, indicating the necessity of the well-designed ensemble method when the number of pattern evaluators is small. Also, our pattern functions outperform those of other baselines, thereby showing the effectiveness of text infilling tasks which can fully utilize pre-trained models in an unsupervised setting.

6 Analysis on Generalization Ability

Generalization ability is essential for automatic metrics to evaluate open-ended text generation models. In this section, we will test whether our metric can be generalizable to measure the generated texts faced with model drift and quality drift.

To measure whether CTRLEval is reliable to assess the generated results of different models, we split the evaluation set into four subsets based on the generation model and calculate Pearson correlation between each metric and human judgments.

The results in Figure 2 show that our metric can outperform other baselines on the generated texts of all the generation models. Simultaneously, CTRLEval can achieve stable performance with smaller variances when evaluating different generation models, indicating that our metric can generalize to the model drift better.

6.2 Quality Drift

To evaluate the generalization ability of CTRLEval on the generated texts with different qualities, we follow the existing work Sellam et al. (2020); Guan and Huang (2020) to construct four biased subsets based on the coherence score of topic-controlled text generation. We first sort all the samples in the evaluation set and use the quartiles to split them into four subsets with the index from 0 to 3. Then, we create four biased subsets. For the $j^{th}$ subset, we sampled the generated texts which belong to the original $i^{th}$ subset with a probability of $\frac{1}{|j-i|+1}$ where $i,j=0,1,2,3$. Thus, the four biased subsets have different distributions of generated texts with different qualities, as shown in Figure 3.

We then calculate the Pearson correlation between each metric and human judgments. The results in Figure 3 show that CTRLEval has higher correlations than the baselines on the evaluation subsets with different qualities. Also, our metric can achieve more stable performance on different subsets, which shows our better generalization ability to deal with quality drift.

7 Analysis on the Number of Evaluators

To investigate how the number of pattern evaluators affects the performance, we randomly sample the evaluators 20 times when evaluating attribute relevance in topic-controlled text generation, and illustrate mean values and standard deviations of each number of evaluators in Figure 4.

Figure 4 shows that as the number of evaluators increases, the mean value of our performance can be persistently improved while the standard deviation is gradually reduced. This demonstrates the necessity of devising multiple pattern evaluators for each aspect, which can alleviate the bias brought by the pattern design. The comparison between the pattern functions of CTRLEval and other baselines indicates our superior performance on all the numbers of evaluators.

8 Analysis on Base Model

Since our method can adapt to different pre-trained models whose pre-training task is text infilling, we additionally choose BART Lewis et al. (2020) and T5 Raffel et al. (2020) as our base model, and present the results in Table 6.

Table 6 shows that PEGASUS and T5 obtain comparable performance on all the evaluation aspects, which indicates that our well-designed text infilling tasks can be transferable to T5 without considerable modification. As for BART which performs worse on consistency and attribute relevance, we conjecture that the fewer parameters and the form of pre-training tasks may limit the performance. Since the pre-training task of BART is to generate the complete text rather than only the masked part of the input text, it may not be good at the evaluation involving a short span of texts, such as the prefix in the evaluation of consistency and the label word in attribute relevance.

We also provide the analysis on the number of parameters in Appendix B and the case study in Appendix C.

Discussion

Extension to More Control Variables: In this paper, we evaluate the relationship between generated texts and two control variables (including content prefixes and attribute labels) via consistency and attribute relevance, respectively. We can also extend our metric to other control variables by designing additional pattern evaluators to measure the relationship between generated texts and each variable, respectively. We will further investigate the extensibility of our metric in the future work.

Design of Pattern Evaluators: With the rapid development of prompt tuning, recent works have proposed new methods on the design of prompts and verbalizers Gao et al. (2021); Lester et al. (2021), which provide alternatives to our metric in attribute relevance. Also, the weight factor of each evaluator can be set as diversity metrics Hashimoto et al. (2019) besides NISF in coherence and consistency. We will leave the exploration of more settings on pattern evaluators as the future work.

Conclusion

We present an unsupervised reference-free metric called CTRLEval for evaluating controlled text generation. This metric formulates the evaluation of different aspects into multiple text infilling tasks, and utilizes the ensemble of generation probabilities from a pre-trained model in different tasks as the evaluation result. Experimental results indicate that CTRLEval obtains higher correlations with human judgments and shows better generalization ability for addressing model drift and quality drift.

Acknowledgments

This work was supported by the National Science Foundation for Distinguished Young Scholars (with No. 62125604) and the NSFC projects (Key project with No. 61936010 and regular project with No. 61876096). This work was also supported by the Guoqiang Institute of Tsinghua University, with Grant No. 2019GQG1 and 2020GQG0005.

Ethics Statement

We construct an evaluation set for evaluating controlled text generation. The data samples in this set are all from the existing works with open-source codes, model checkpoints, and generated results. We directly use the generated results if the authors have released them. Otherwise, we adopt the same setting as the original papers to make these models generate texts. We do not apply extra selection strategies to the generated results.

We resort to Amazon Mechanical Turk (AMT) for the annotation of this evaluation set. We do not invade the privacy or collect personal information of annotators. We pay each annotator $0.06 for each survey including four generated texts and one negative sample. The payment is determined based on the length of data samples. We additionally ask annotators to check whether there is a potential ethical problem in the data, and remove these problematic data in the evaluation set. After annotation on AMT, we manually review all the annotated samples from an ethical perspective. However, we admit that there may still exist unpredictable bias in this evaluation set.

References

Appendix A Pattern Evaluator for Attribute Relevance

We first choose the prompts and verbalizers which have been shown to work well in the existing works on few-shot text classification Schick and Schütze (2021a); Gao et al. (2021) and generation Schick and Schütze (2021b) as the seed prompts and verbalizers. Then, we expand our prompt set with the following rules: 1) Switching the order of generated texts, prompts, and mask tokens; 2) Collecting the paraphrases of seed prompts just as BARTScore Yuan et al. (2021) does. All the prompts and verbalizers which are used in our experiments are shown in Table 8.

Appendix B Analysis on the Number of Parameters

We further conduct experiments on the base model with different numbers of parameters. Since the authors of PEGASUS Zhang et al. (2020a) do not release the model checkpoint of PEGASUS-base, we choose T5-small, T5-base and T5-large Raffel et al. (2020) as our base models respectively, and present the results in Table 7. The results show that larger numbers of parameters can benefit the model performance while degrading the computation efficiency.

Appendix C Case Study

To intuitively show how our metric works in the evaluation of controlled text generation, we provide some cases on the three evaluation aspects, including coherence (Figure 6), consistency (Figure 6), and attribute relevance (Figure 7). Since the range of various metrics is always different, it may be less meaningful to directly compare the absolute value of each metric. Thus, we follow the existing works Guan and Huang (2020); Liu et al. (2021b) to conduct a pairwise comparison on different samples.

The results in Figure 6 and 7 show that our metric can give accordant preferences with human judgments, indicating the effectiveness of our metric on all three evaluation aspects. To further show how each pattern evaluator works in the overall evaluation result, we take the second sample in Figure 7 as an example and visualize the weight $\beta(I)$ and score $s(I)$ in Figure 5. We can observe that most of the pattern evaluators assign high scores to this sample which agree with the human judgment. Simultaneously, the weight factor automatically reduces the effect of low-quality evaluators which also plays an important role in the final evaluation result.