Standardizing the Measurement of Text Diversity: A Tool and a Comparative Analysis of Scores

Introduction

Evaluation of LLM-generated texts is typically done with respect to accuracy or factuality, e.g., as measured via entailment (Tang et al., 2023), or text quality aspects such as coherence and fluency (e.g., estimated using LLMs as evaluators as in Liu et al. 2023). For summarization tasks where references are available, the similarity of generated outputs to these is also often measured. A complementary dimension of model performance is diversity, i.e., how much “boilerplate” content is repeated across LLM outputs.

The diversity of generated outputs is intuitively important: A model prone to repeating specific sentence constructions or boilerplate turns of phrase across its outputs will likely be deemed lower quality than an LLM capable of more diverse generations, all else being equal. Table 1 shows example repetitions over a news summarization dataset. We provide detailed examples including which model produced each set of examples in Appendix A.1. In this work we first analyze commonly reported diversity scores over English language outputs from several LLMs, and identify a few practical, (mostly) independent scores that characterize repetition. We also release diversity, an open-source Python package that can be used to explore and evaluate the diversity of a generated text datasets.https://pypi.org/project/diversity/

Background

We investigate automatic approaches for characterizing the diversity of outputs from LLMs. Our work is motivated by qualitative observations of such outputs, in which it is easy to notice undesirable repetition of formulaic responses.

Lack of diversity may result from repetition of long stretches of text or be due to subtle distributional patterns (Holtzman et al., 2019; Meister et al., 2022; 2023a). We focus on scores that are likely to capture overt repetition across outputs, and leave for future work similar analysis of semantic and structural diversity scores (Bär et al., 2012).

Image captioning and dialog are text generation tasks with a robust body of work quantifying the diversity of produced texts. In both domains, prior work has documented that models tend to produce the same text for different contexts. Li et al. (2016) find that four phrases account for about a third of all turns produced by a conversational agent. Devlin et al. (2015) report that more than half of the automatic captions are repeated verbatim for different images and most of these captions are seen as exact strings in training. Self-repetition (Salkar et al., 2022), i.e., an exact repetition of the same $n$-gram longer than four across different summaries is a way to adapt for a similar analysis in tasks where the generated text is longer, so exact matches rarely occur but repetition is common, especially relative to the training data (Wang et al., 2023a).

The above observations speak to the need for a standardized, easy to use method to quantify diversity. In this work we propose such standardization and show that compression ratio is a fast, convenient to compute score which is sufficient to capture the information in all token/type ratio related alternatives. However, we also find that compression ratios—and all scores considered—are moderately to strongly correlated with the length of texts, complicating interpretation. Many comparisons remain meaningful when accompanied with information about length, but in others no conclusions can be reliably drawn.

The association between length and measures of diversity is well-established in corpus linguistics (see detailed discussion in Brysbaert et al. 2016). The number of unique words in a corpus is a power function of the total words seen, where the power is less than 1. The number of total words grows linearly, while the number of unique words is sublinear, so longer texts have more repetitions of unique words and $n$-grams than shorter texts (Covington & McFall, 2010; McCarthy & Jarvis, 2010).

LLM generations can lead to reduced text diversity in a few ways. Guo et al. (2023) study the effect of consecutive rounds of distillation, in which a language model produces the data on which the next language model is trained. They report dramatic reduction in diversity over 10 iterations. However, this study did not report the length of text produced in consecutive distillation rounds. Given our findings, it would be prudent to check if output lengths remain comparable (or shorter). Padmakumar & He (2023a) show that when people write with the aid of an LLM (e.g., instructGPT) they produce less diverse writing than when they do not. Here we ascertain that these results are independent of length, indicating that there is a genuine reduction in diversity (rather than merely affecting lengths which in turn influence diversity measures).

Work across use-cases has shown that reducing the repetition—or, equivalently, increasing the diversity—in training data yields higher quality models. De-duplicating pre-training data leads to more efficient pre-training and better models that do not repeat pre-training data directly (Lee et al., 2022; Abbas et al., 2023). Removing fine-tuning summaries with repeated content improves summarization performance (Choubey et al., 2023). Given this, instruction diversity used for instruction tuning LLMs will likely also have implications for mode performance. We assess the diversity of instructions in common instruction tuning datasets.

A Smorgasbord of Text Diversity Scores

The variety of scores used to measure diversity across a corpus of texts derive from two core ideas: Computing average similarity between pairs of outputs produced by the same model for different inputs, and computing variants of token/type ratio. The former are adapted from common approaches to text generation evaluation by comparing with references, using standard measures of pairwise similarity; the latter track the diversity of vocabulary measured as the ratio of unique words to total words produced, with the outputs from a model concatenated into a single text.

We first describe each score, and then present insights regarding their mutual redundancy. We also consider their required run-times, which are lengthy for some metrics and may render them impractical for analysis of a large number of outputs. All scores are defined for a set of generated texts $D$, each conditioned on its respective input.

The quality of text in machine translation, summarization, and image captioning is often reported in terms of overlap with a reference text. This idea can be adapted to measure diversity across different outputs by using one generated text as a reference and measuring the similarity of other outputs against this. Self-BLEU measures similarity between all text pairs in $D$ using BLEU as the similarity score (Zhu et al., 2018). BLEU can be replaced with an arbitrary similarity score, e.g., ROUGE or BERTScore. These variants are called homogenization scores and have recently been used to compare the diversity of texts produced under several conditions (Padmakumar & He, 2023a).

Homogenization Score (ROUGE-L)

Here the similarity score of choice is ROUGE-L (Lin & Och 2004; Eq. 1). This quantifies overlap in terms of longest common sub-sequences between all pairs of text in a corpus instead of the fixed $n$-gram size used in other ROUGE variants:

Homogenization Score (BERTScore)

This homogenization score uses BERTScore to measure similarity between documents in Equation 1. Unlike the other scores, it does not count the repetition of specific tokens, but instead uses BERT embeddings to (ideally) capture “semantic” similarity beyond verbatim $n$-gram matches.

Self-repetition Score

Self-repetition was introduced to measure the tendency of LMs to repeat long $n$-grams across different outputs (Salkar et al., 2022).

Where $k$ is total number of 4-grams in a single document $d\in D$, and $N_{i}$ the number of other summaries in which 4-gram $i$ appears. The final score is the sum of $\text{SRS(}d\text{)}$ divided by the number of documents in the corpus $D$.

Moving Average Token-Type Ratio

The token-type ratio for a text is the unique token count divided by the total count of tokens. Moving Average Token Type Ratios (MATTRs) was introduced as a way to measure the lexical dynamics across a text which is insensitive to text length. The score captures the repetition of a given word in segments of text and does not explicitly account for longer repeated sequences (Covington & McFall, 2010).

N𝑁N-Gram Diversity Score

NGD extends the idea of token-type ratio to longer $n$-grams (Padmakumar & He, 2023a; Meister et al., 2023b; Li et al., 2023). It is defined as a ratio of the unique $n$-gram counts to all $n$-gram counts:

Where $D\oplus$ denotes the dataset $D$ concatenated into a single string. We use four as the maximum $n$-gram length. This method captures repeated sequences in addition to single token diversity.

Hypergeometric Distribution D

The probability of text under a Hypergeometric Distribution D (HD-D) is an another measure of lexical diversity (McCarthy & Jarvis, 2010).For both HD-D and MATTR, we use the implementation provided in the lexical-diversity package (https://pypi.org/project/lexical-diversity/). HD-D does not capture repetition of sub-sequences.

Compression Ratios

The diversity scores introduced so far are all a function of the number of repeated substrings across outputs. Some measure these over pairs of generated texts, others are computed for a concatenation of all outputs into a single text. Text compression algorithms are designed to identify redundancy of sequences of variable length in text.

We use gZip to compress the concatenated text of all outputs generated by a model. The compression ratio is then the ratio between the size of the compressed file to that of the original file. High compression ratios imply more redundancy:

Part-of-Speech Compression Ratio

To capture repeated syntactic patterns, we also compute compression ratios for part-of-speech (POS) tag sequences. We use the NLTK POS tagger https://www.nltk.org/api/nltk.tag.html and the Penn Treebank set of 36 tags.

Data and Models

We compute diversity scores for the outputs of six instruction tuned models on the CNN/DailyMail (Hermann et al., 2015) and XSUM (Narayan et al., 2018) English news summarization datasets. The models are: Llama-2 (Touvron et al., 2023a), GPT-4 (OpenAI, 2023), FlanT5-XXL (Longpre et al., 2023), StableLM (Taori et al., 2023; Chiang et al., 2023; Anand et al., 2023), Mistral (Jiang et al., 2023), and StableBeluga (Touvron et al., 2023b; Mukherjee et al., 2023).All models—except GPT-4—downloaded from HuggingFace (https://huggingface.co/models). We selected these models to cover a range of availability (open- and closed-source), and architectures (encoder-decoder, decoder-only).We use prompts for summarization provided by each model, where available. See Appendix A.2. We also measure the diversity of the input news articles, the first three sentences of the articles, and the reference summaries. The lengths of texts vary considerably by source, for reference and model-produced text alike, so we also note average lengths when reporting diversity.

Text Length as Confounder

To keep computational time and costs manageable, we randomly sample 500 inputs from CNN/DailyMail and XSUM for analysis. Table 3 reports diversity scores for the outputs generated by the six zero-shot LLMs for these inputs.

The top panel of the table shows scores for human-written texts: the original article given as input for summarization, the baseline summary consisting of the first three sentences of the news article and the human reference summary. These scores serve as a reference point with respect to which to interpret the scores for models. The expectation is that the human texts are more diverse than those produced by LLMs, with the caveat that the texts were scraped from the web, so may contain HTML, ads, and page layout artefacts which might be repetitive (Salkar et al., 2022).

The human texts differ by length and the sources of longer texts appear to be less diverse. The association between the length of the produced texts and their diversity is similarly pronounced in the XSUM dataset, as seen Table 4. Text length as a confounder for diversity has been reported in prior work (Salkar et al., 2022), along with potential methods to adjust for this, e.g., sampling blocks of fixed size (Covington & McFall, 2010).

Table 6 reports correlations between the number of words produced by each model and diversity scores. All scores of the token/type ratio family are highly correlated with length, while the pairwise similarity ones are only moderately correlated. Self-BLEU has low correlation with length.

Diversity of Model Summaries

The confound of length complicates reporting. On both CNN/DM and XSUM (cf. Tables 3 and 4), StableLM produces the longest summaries. All scores indicate that these are the least diverse, most likely due to the length confound. In both sets of results, we look for models that produce shorter summaries that are less diverse. These findings are notable and hold, despite length differences.

Three types of differences are marked in the tables. Model summaries that are shorter but less diverse than human summaries are marked in bold. Human texts here are written by journalists, so the expectation is that they would be more diverse. More bold entries in a column indicate that the score captures difference between human and machine diversity, which is a desirable trait. Underlined entries mark models that are less diverse than other models that produce longer summaries. The more underlined entries there are for a model, the more indicators there are that its output is less diverse. Asterisks mark models that appear more diverse than a human text of shorter length.

The most interesting diversity scores are ones that capture differences between human and automatically produced text, without necessarily committing to an interpretation of which source is preferable. Human evaluation in future work will address this question. On the CNN/DM dataset, homogenization with BERTScore and MATTR are the two scores that detect no differences between human and model texts. BERTScore does not detect such differences on the XSUM dataset either. Compression ratio for part of speech sequences is the score that identifies the most differences between human and model-generated text. Self-repetition stands out as the only score that identifies model generated text as more diverse on the CNN/DM dataset. From this analysis, CR:POS and self-repetition emerge as prime candidates of reportable scores, while homogenization with BERTScore as perhaps not useful.

All scores detect at least one difference for a pair from the models we study. According to seven of the scores, Llama-2 generates texts that are less diverse than those from Mistral. FLAN-T5 is also marked as less diverse than StableBeluga according to four scores. Finally, four scores identify GPT-4 as less diverse than FLAN-T5; two of these are BERTScore homogenization, which we establish is perhaps not necessarily applicable and self-repetition, which marks human text as less diverse.

The XSUM dataset results in fewer notable observations. The one consistent findings is that FLAN-T5 produces that shortest and least diverse summaries, less diverse than other models and less diverse than human text. The type of input text clearly changes the behavior of the models and the diversity of text they produce.

Correlation Analysis

Here we present three sets of correlation analyses between (i) different diversity scores, (ii) the same diversity score across datasets, and (iii) diversity scores and standard reference-based evaluations.

Despite the large number of diversity scores in our list, they all revolve around $n$-gram repetition. It is of interest to know if they capture different or similar information. With this motivation in mind, we computed the correlations between every pair of scores, shown in Figure 1.

Compression ratio is highly to moderately correlated with other $n$-gram scores. The only weak correlations are with Self-BLEU and BERTScore homogenization. BERTScore homogenization and Self-BLEU are moderately correlated with each. Given the degenerate behavior of BERTScore homogenization on the analysis of summaries, reporting self-BLEU only is advisable. Finally, self-repetition is only moderately correlated with with other scores, so is informative to report as a standard score for diversity. The correlations are similar on the XSUM summaries (see Appendix 4), reinforcing the recommendation for the set of scores that should be used to capture diversity.

Diversity analysis on the CNN/DM and XSUM datasets did not indicate consistent system behavior. We further examine this mismatch, reporting in Table 2 the correlations between diversity score types values across the two datasets. Self-BLEU scores are almost perfectly correlated between the two datasets; they appear to not be affected by text source. The other scores are still moderately to highly correlated but as already observed, models are ranked differently. When reporting diversity, source of analyzed data also has to be taken into account, in addition to length.

Our guiding assumption is that output diversity and self-repetition are aspects of model behavior that are not captured by existing evaluation approaches. Here we directly test this assumption. We compute the system level correlation between the diversity scores and the traditional BERTScore and ROUGE evals, shown in Figure 2. The reference-based evaluations are only weakly correlated with the diversity metrics. Self-BLEU, however, is moderately anti-correlated with with both ROUGE-1 and BERTScore.

Truncating to Control Length

For each input for summarization, we truncate all summaries to the length of the shortest one produced by any of the sources, as a crude method to remove the influence of length on scores. The resulting scores are directly comparable across sources, listed in Tables 4 and 5.

Compression ratio and Self-BLEU scores indicate that model-produced text is less diverse than human text. BERTScore homogenization scores barely vary across sources, further supporting the recommendation that this is not a useful score to report.

On the CNN/DM dataset, Self-BLEU indicates that Llama-2 and StableLM are the most repetitive models. Compression ratio also ranks these two models as the least diverse. The results are consistent on XSUM, but for that dataset Flan-T5 is also highly ranked and the most repetitive.

The truncation approach to control for length is not practical for published research or leaderboards. Introducing a new source of texts would require recomputing the scores for other sources one may want to compare with, which is impractical and sometimes impossible when the outputs from other sources are not available. Future research will have to search for more practical alternatives.

Run-Time Considerations

When analyzing the diversity of large volumes of text, run-time considerations become relevant. Figure 3 provides insights about the feasibility of obtaining scores for large samplesRun on a single NVIDIA Quadro RTX 8000 GPU.. The compression ratio scores are fast, with text compression utilities specifically optimized for speed. Self-repetition takes longer but acceptable time. Self-BLEU and BERTScore homogenization are prohibitively slow.

Broader Applications

The guiding motivation for this work has been to develop standardized and informed approach to the analysis the diversity of text produced by LLMs. The standardization of scores will facilitate analysis in broader settings. Here we provide two examples: human writing, with and without facilitation from a LLM, and instruction tuning datasets.

Padmakumar et al. (2023) presented an analysis of human-written stories, where people wrote either by themselves or with the help of GPT-3 or GPT-3.5 Turbo. They find that using LLMs as writing partners leads to greater homogenization of the stories.

As reported by Padmakumar et al. (2023), we find that all diversity scores agree that people writing independently produce the more diverse texts (cf. Table 8). Here, story length is not an issue because the average length of stories in each setting are comparable: 375 words for writing without help, 372 words when writing with GPT-3 and 370 when writing with GPT-3.5.

2 Instruction-tuning Datasets

The quality and diversity of instructions are likely to result in more robust and capable systems (Sanh et al., 2022; Mishra et al., 2022). We analyze the diversity of five instruction-tuning datasets.

is a collection of crowdsourced instructions (Köpf et al., 2024). The data was collected under detailed guidelines and includes questions that reflect real-life situations.

Super-NaturalInstructions

A corpus comprising crowdsourced instructions that transform 200 benchmarks and intermediate evaluation results into a set of instructions and demonstrations (Wang et al., 2022).

Unnatural Instructions

An (almost) automatically created dataset, using instructions from the SuperNatural-Instructions dataset to automatically generate new instructions (Honovich et al., 2023). To increase diversity, each instruction was also paraphrased. Honovich et al. (2023) compare the diversity of instructions in Unnatural and Super-Natural Instructions with pairwise BERTScore similarities (within each dataset), and find that the similarities are much higher in Super-NaturalInstructions.

Alpaca

This dataset is created following the Self-instruct dataset (Wang et al., 2023b). GPT-3 was prompted to create instructions and demonstrations based on a seed of 175 human-written instructions. Crucially, the collection method includes a diversity filter, only including model-written instructions if their ROUGE-L similarity is less than 0.7 with an existing instruction. Length of instructions and demonstrations is also controlled for as a criterion for inclusion in the final instruction dataset.

Dolly

A set of human instructions and demonstrations, collected by Datrabricks employees (Conover et al., 2023). By design, they cover only eight classes of popular tasks: creative writing, closed and open QA, summarization, information extraction, classification and brainstorming.

In Table 9 we report diversity scores. Here datasets are ordered by size; we therefore expect that scores will be sorted in diminishing order in each column. Only deviations from this ordering are reportable. We provide details about the number of instructions and words in Appendix A.4.

Open Assistant instructions are remarkably diverse compared to the other datasets, and all diversity scores for it are more favorable than that for other datasets. Unnatural instructions are remarkable in the opposite direction, with outlier scores that are so much higher, they are likely not due to length entirely. We provide an analysis of the diversity scores with the length controlled in Appendix A.5.

Given the large dataset sizes, ranging from 15-80k data points, we do not compute the homogenization scores nor Self-BLEU, as the computation time is infeasible. For approximately 50k instructions, the estimated computation times ranged from 48 to 800 hours for these scores. This case study highlights the relevancy of the run-time analysis for computing score that we presented in the previous section.

Discussion and Recommendations

Our in-depth analyses reveal that compression ratio is an excellent score to report, easy to compute and strongly correlated with other scores used in past work. Compression ratio of part of speech sequences capture differences between human and model-generated text, so is also a good score to track. Self-repetition zeros in only on repetition of longer $n$-grams across generations and is only moderately correlated with compression ratios and is intuitively interpretable, ass desirable characteristics. Finally Self-BLEU is only weakly correlated with the previous three, so is a good complement score to report. In our analyses, we identified several drawbacks of BERTScore: it does not show differences between human and model-generated text and barely varies when adjusted for length. There is no good justification to report it.

Length of the analyzed text has to be reported alongside all these scores. When length differs, scores are not meaningfully comparable. Truncating and downsampling text is one way to produce a set of results that are intuitively comparable. Different random draws of the sample chosen to represent a dataset will likely differ in diversity; the selection may lead to unwarranted conclusions. Truncating texts prevents any possibility of discovering repetitive behavior towards the end of longer text. Future research into a principled solution for this problem is urgently needed.

Despite all this, we were able to glean meaningful insights about differences in diversity between human and model-produced text for summaries, essays and instructions.

Limitations

In the work presented here we do not explore human approaches to evaluating the diversity of collections of text. These are straightforward when the produced text is fairly short, as in judging the diversity of a set of questions generated for a given document (Sultan et al., 2020) or the diversity of possible continuation of a conversation (Tevet & Berant, 2021). Longer texts, as in the case of summaries, and larger collections, as in the case of instruction datasets are harder to judge for diversity.

An interface allowing people to explore the data, organized by stretches of repeated text ordered either by the length of the repeated string or the number of times it has been repeated, can facilitate human evaluation.

Acknowledgements

We gratefully acknowledge the National Science Foundation (RI 2211954) for supporting this work.

References

Appendix A Appendix

Table 10 show more examples of repeated sentence structures (using part-of-speech tags) from Padmakumar et al. (2023).

A.2 Summarization Prompts

Table 11 details the prompts and format used to generate the summaries for the news datasets. We follow the formats recommended provided by each model, and insert the along with the instruction for summarization.

A.3 XSUM Metrics

Figure 4 shows the correlations between all pairs of metrics for the XSUM dataset. The correlations show that compression ratio is highly to moderately correlated with other n-gram scores, similar to the findings for the CNN/DM dataset.

A.4 Instruction Datasets, Details

Table 12 shows the number of instructions, the typical length of an instruction and average number of words per instruction set. All vary, making it even harder to control for length. Truncating makes less sense here, and down-sampling the number per instructions is counter-productive given our goal to understand the diversity of the entire dataset. We do make use of these instruments given the lack of alternatives, but note that more meaningful solutions are urgently needed.

A.5 Instruction Datasets, Length Controlled

Table 13 shows scores for instructions downsampled to the size of the smallest dataset, and truncated to the length of the shortest instructions in the remaining data. Again, the Open Assistant dataset stand out as most diverse, while the Unnatural Instructions dataset is markedly less diverse than the others. Self-repetition in the related Super-Natural and Unnatural instructions is notably high. The human instructions in Dolly compare favorably with automatic instructions, especially when bearing in mind that only eight tasks are covered in it. CR:POS points to Super-natural instructions as the most diverse. We do not have a convincing explanation of why it compares so favorably against others on this score.