KIEval: A Knowledge-grounded Interactive Evaluation Framework for Large Language Models

Introduction

The landscape of artificial intelligence has been significantly reshaped by the emergence of Large Language Models (LLMs) as they have been pivotal in various natural language understanding and generation tasks (Brown et al., 2020; OpenAI, 2023; Bubeck et al., 2023). To better understand the capabilities and weaknesses of LLMs, their effective evaluation becomes increasingly essential (Chang et al., 2023; Guo et al., 2023).

Automatic evaluation methods of LLMs generally fall into two categories: static dataset-based and LLM-based evaluation (Chang et al., 2023). The former (Clark et al., 2018; Zellers et al., 2019; Hendrycks et al., 2020; Huang et al., 2023) requires evaluated LLMs to generate a short span of text containing choices or answers for pre-defined questions (Gao et al., 2021) to challenge their knowledge. The latter (Chiang and Lee, 2023), also known as LLM-as-a-judge, typically depends on LLM evaluators to evaluate the model’s outputs given predetermined questions or instructions (Zheng et al., 2023; Lin and Chen, 2023; Fu et al., 2023; Wang et al., 2023b). Despite these promising efforts, current evaluation methodologies still broadly face the bottleneck of data contamination (Schaeffer, 2023; Wei et al., 2023; Oren et al., 2023; Sainz et al., 2023; Daniele and Suphavadeeprasit, 2023), where models trained on test splits of datasets can artificially inflate benchmark performance, overestimating their real-world efficacy and even potentially misleading scientific conclusions (Zhou et al., 2023).

Recently, two primary strategies have been employed to mitigate data contamination of LLMs. The first involves identifying whether specific texts or test samples exist in the training dataset by assessing loss values Wei et al. (2023); Shi et al. (2023) or probing datasets like Common Craw Li (2023). The limitation lies in its capacity to only measure contamination levels rather than actual model performance. Meanwhile, this technique demands access to the model’s internal structure or training datasets, rendering it ineffective for proprietary LLMs. The second strategy creates dynamic evaluation samples through heuristic methods, such as graph-based processes Zhu et al. (2023), yet this is confined to particular tasks (e.g., multi-step reasoning). Currently, there is a lack of a generalized evaluation protocol capable of assessing genuine performance amidst data contamination across diverse tasks and domains for both open and closed-source LLMs.

To this end, we propose KIEval, a Knowledge-grounded Interactive Evaluation framework, where a novel LLM-powered role, named "interactor," is introduced into the evaluation process for the first time. The term "knowledge-grounded" refers to our evaluation’s starting point, which involves posing a question from an existing benchmark dataset that demands domain-specific knowledge. By "interactive," we mean the evaluation process delves deeper with structured and dynamic multi-round dialogues—tailored by the proposed interactor—to explore knowledge related to the initial question. These technical designs inherently provide our evaluation framework with two distinct merits.

Contamination-Resilient: KIEval marks a departure from conventional approaches that evaluate a model’s capability in responding to static questions. Dynamic multi-round interactions allow us to distinguish whether a model’s answer stems from a simple recall of benchmark answers or reflects a sound understanding to apply knowledge in problem-solving.

Generalized and Scalable: Leveraging the capabilities of advanced LLMs as interactors renders our evaluation method universally applicable and eliminates the need for additional human efforts. Meanwhile, by reusing high-quality benchmark datasets as a foundation for domain knowledge, KIEval enables efficient scalability across diverse domains, tasks, and languages without significant resource expenditure.

We validate KIEval’s alignment with humans and compare it with previous evaluation methods. Our experiments show that KIEval achieves a high Pearson correlation coefficient of 0.81 with human scores, underscoring KIEval’s proficiency in reflecting human preferences in our settings compared to previous evaluation methods. We also analyze KIEval’s correlation with static dataset-based benchmarks, identifying that notable disparities in performance could signal data contamination.

Overall, our core contributions are three-fold:

A novel dynamic evaluation protocol. KIEval pioneeringly evaluates LLMs through dynamic multi-round interactions to mitigate data contamination. By seamlessly integrating with existing datasets as knowledge sources, KIEval can cost-effectively assess knowledge memorization and generalization across domains and tasks.

Extensive evaluation of popular LLMs. We conduct thorough experiments and analysis with seven leading LLMs across five datasets with KIEval, assessing their generative abilities and domain knowledge, confirming the susceptibility of current evaluation methods (e.g., static dataset-based and LLM-based evaluations) to data contamination.

New insights into data contamination. Our investigation reveals the incompetence of data contamination in improving LLMs’ genuine understanding and generalization, with current detection methods unable to identify contamination in the fine-tuning phase.

We release all necessary code and data for reproducing our method and the compared baselines.We release all materials at https://github.com/zhuohaoyu/KIEval.

Related Work

Human evaluation approaches manually design experiments and tests (Novikova et al., 2017; Bommasani et al., 2023). While it provides insights into human-model interaction, it faces challenges due to the subjectivity and inconsistency of human judgments (Chang et al., 2023). Moreover, it is resource-intensive in terms of time and cost, limiting its feasibility for large-scale assessments (Karpinska et al., 2021).

Static dataset-based approaches assess LLMs focused on domain-specific questions or tasks using pre-defined static datasets. Typical evaluation tasks include solving single or multiple-choice problems (Clark et al., 2018; Hendrycks et al., 2020; Huang et al., 2023) and question answering (Lin et al., 2021; Cobbe et al., 2021), these tasks require LLMs to generate short spans of text containing answers to the questions (Gao et al., 2021). The performance of LLMs is measured by their ability to correctly answer or perform these tasks.

LLM-based evaluation, utilizing one strong LLM (Brown et al., 2020; OpenAI, 2023) to assess others, is a recent approach that often employs pairwise comparisons to identify nuanced differences in model outputs, addressing the challenge of determining clear model superiority (Wang et al., 2023b; Zheng et al., 2023). This method bridges the gap between human and dataset-based evaluations by focusing on generative abilities. However, this approach has limitations, including reliance on fixed templates (Zheng et al., 2023), instructions (Wang et al., 2023b; Li et al., 2023), or multi-round chat datasets (Fu et al., 2023; Lin and Chen, 2023), limiting its scope in capturing diverse domain knowledge and real-world applicability. It also faces contamination risks, as training on outputs from a strong LLM can inflate results, as noted in work from Daniele and Suphavadeeprasit (2023) collect data from MT-Bench (Zheng et al., 2023) as training data while AlpacaEval (Li et al., 2023) contains evaluation set from various instruction-tuning dataset. Additionally, studies indicate these LLM evaluators might be biased (Zeng et al., 2023; Wang et al., 2023a, b).

2 Addressing Data Contamination of LLMs

Data contamination refers to the inclusion of information in the training set of models that provides insights into the test set of a benchmark dataset, and then evaluated in the same benchmark. Recently, the AI community has become increasingly concerned (Schaeffer, 2023; Zhou et al., 2023; Oren et al., 2023) about data contamination in LLMs. Detecting data contamination, a form of Membership Inference Attack (MIA), poses challenges for large language models (LLMs) due to their training on vast corpora and the difficulty of conducting ablation studies (Shi et al., 2023). To detect such contamination of LLMs, Wei et al. (2023) suggested comparing average loss values between training and test datasets, while Shi et al. (2023) introduced Min-K% Prob based on loss values to identify texts used in training. Our experiments show these methods are effective for pre-training but not for detecting contamination during fine-tuning. Zhu et al. (2023) leveraged DAG to dynamically generate evaluation data in reasoning tasks to avoid contamination. In comparison, KIEval only requires access to output texts of evaluated models and detects data contamination through evaluating its ability to generalize and utilize knowledge as well as generative ability, which requires a deeper understanding of knowledge instead of mere memorization of the answers. Moreover, our experiments suggest that KIEval is resilient to data contamination, offering a reliable means to discern whether models have been trained on test sets. This makes it a valuable tool for complementing traditional benchmarks, providing a more nuanced understanding of a model’s exposure to and handling of contaminated data.

Methodology

KIEval involves a series of iterative interactions, as depicted in Figure 1. KIEval is engineered to dynamically evaluate the conversational abilities of LLMs through interactive dialogues focusing on domain-specific topics that challenge LLMs’ generative ability and in-depth generalization of knowledge. It simulates realistic conversation flows, offering a dynamic alternative to the static question-answer format of traditional benchmarks.

KIEval orchestrates an evaluation where an LLM, referred to as the candidate (the model under evaluation), must understand and respond to an evolving series of questions. These question prompts are generated by an interactor model, designed to challenge the candidate with contextually rich scenarios. The responses from the candidate are then assessed by an evaluator model, which scrutinizes the output for factual accuracy, relevance, and coherence. The interactor and evaluator are both strong LLMs (e.g., GPT-4, Gemini, Claude 2, LLaMA2-70B-chat, etc.) as the standard practice of LLM-based evaluation protocols.

The design of KIEval emphasizes the importance of reproducibility and consistency in LLM evaluations. By employing separate models for the interactor and evaluator roles, KIEval ensures that the dialogue context remains consistent across different evaluations, as it is fair for the same conversation to be assessed by various evaluators or the same evaluator with different seeds, facilitating a voting strategy to ensure consistent evaluation results. To achieve reproducibility, KIEval utilizes deterministic outputs from LLMs, such as the latest GPT-4 model with temperature sampling disabled and a fixed seed or deploying local models as evaluators. This guarantees identical responses in every run. Due to space limits, we show the complete system prompts in Appendix K.

2 Interactive Evaluation Procedure

The interactive evaluation procedure can be described by Algorithm 1 and the complete implementation can be found in our repository. In LLM-based benchmarks, we hypothesize that the evaluator ($\mathcal{M}_{E}$) models, given their advanced capabilities, can reliably evaluate the performance of less sophisticated candidate models ($\mathcal{M}_{C}$) (Zheng et al., 2023; Zeng et al., 2023). Nevertheless, their applicability as definitive standards is not without limitations, especially when confronting arduous benchmarks. To counteract this, we test the evaluator models against benchmark datasets and sample a fixed number of questions they answer correctly, to ensure the validity of their judgments.

3 Evaluation Metrics

KIEval implements a scoring system to quantitatively grade the performance of candidate LLMs in different aspects. Responses are rated on a definitive scale from 1 to 4 for each aspect, where 1 and 4 denote ‘Poor’ and ‘Strong’ performance, respectively, as detailed in Table 1. These scores are intended to be definitive to encourage decisive evaluations and are accompanied by comments for interpretability and insights into each score.

We then calculate the KIEval score, which quantitatively measures the results given by the evaluator model, emphasizing sustained and high-quality conversations. Formally, the KIEval score for single-turn scores $s_{0},s_{1},\ldots,s_{n}$ in $n$ rounds can be computed as:

where the decaying weight $w_{i}=\exp(-\frac{i}{n})$ placing more emphasis on early turns of the conversation. The normalization ensures a bounded KIEval score, with $1.0$ indicating perfect performance across all rounds. In addition to these metrics, KIEval incorporates an early stopping mechanism within the evaluative process. The evaluator model ($\mathcal{M}_{E}$) possesses the discretion to prematurely end the conversation if the candidate’s response is egregiously inadequate. Criteria for early termination include significant deviations from the topic, empty responses, unpermitted role shifts, and hallucinatory content. We adopt this strategy to measure how well the candidates maintain a meaningful conversation. We further examine the effectiveness of these techniques through an ablation study, with detailed experiments and results available in Appendix D.

Experiments

In this section, we conduct experiments designed to rigorously test the KIEval framework. Our objectives are threefold: (1) to evaluate the generative performance and generalizable knowledge of popular large language models on KIEval using existing benchmark datasets; (2) to assess the impact of data contamination on model performance, specifically examining whether such contamination leads to mere memorization or contributes to genuine understanding and generalization; and (3) to determine the alignment with human, reliability, and effectiveness of KIEval.

Experiment Setup. We select GPT-4We use gpt-4-1106-preview from OpenAI’s official API for all experiments, including MT-Bench (0.2.32) and AlpacaEval (0.3.6). (OpenAI, 2023) to be both the evaluator and interactor model by feeding it corresponding prompts with a fixed seed to ensure deterministic outputs. We select 200 samples for each dataset, allowing a maximum of 5 rounds of conversation. The candidates’ performance is assessed using the KIEval framework, which evaluates responses based on accuracy, logic, relevance, coherence, and conciseness. We also report dataset-based benchmark accuracies in 5-shot settings and LLM-based benchmark scores from AlpacaEval (Li et al., 2023) and MT-Bench (Zheng et al., 2023) in comparison, as depicted in Table 2.

In this experiment, we utilized five popular LLM benchmark datasets: ARC-Easy and ARC-Challenge (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2020), and C-Eval (Huang et al., 2023). For candidate models, we selected a diverse set of 7 LLMs: including proprietary model GPT-3.5 (Brown et al., 2020) with API access and open-access foundation models: Llama 2 (Touvron et al., 2023b) 7B, 13B, 70B; Mistral-7B (Jiang et al., 2023); Yi-6B-chat (01.AI, 2023); MPT-7B (MosaicML, 2023).By default, we use the ‘chat’ versions of Llama2, Yi, and MPT models and the ‘Instruct’ version of Mistral model. Detailed introduction of these datasets and models can be found in Appendix B.

Referencing Table 2, we observe the following trends: GPT-3.5 demonstrated consistently high performance across all datasets, particularly excelling in KIEval scores, which indicates strong contextual understanding and response generation. LLaMA2 70B showed competitive results, achieving only a marginal gap from GPT-3.5 on ARC-E, ARC-C, HSwag and even surpasses GPT in MMLU when measured by dataset accuracies, but we can significantly observe a larger gap between these two models with KIEval metrics in all datasets which is also observed by MT-Bench results as reported in Table 2. This suggests that traditional benchmarks may underestimate the difference in performance between LLMs as these benchmarks only let models generate a short span of text to evaluate which focus on testing understanding ability. Thus it is hard for these benchmarks to accurately reflect performance gaps in generative tasks.

The results from different aspects visualized in Figure 2 benefits us in evaluating model capabilities more comprehensively. We observe that most models exhibit relatively strong performance in terms of relevance and could generate coherent responses. Larger models generally perform better in benchmarks, but it is notable that LLaMA2 70B does not perform well in generating concise responses, compared to its smaller counterparts. Although MPT performs weakly in accuracy, its ability to generate concise responses deserves a closer look at its instruction-tuning data.

One interesting finding is that Yi-6B performs unexpectedly well in all benchmark dataset accuracies, especially with it surpasses GPT-3.5 and all other models by a large margin of over 20% in the C-Eval dataset while exhibiting a similar performance of LLaMA2 70B in other datasets. However, Yi-6B’s KIEval score is very similar to LLaMA2 7B and in the range of other 7B models, while it only performs marginally better in the Chinese dataset C-Eval. This raises our concern over potential data contamination in Yi-6B.

To better understand the correlation of KIEval and static dataset-based benchmarks, we provide a detailed analysis in Appendix C.

2 Resilience to Data Contamination

In this subsection, we show that existing static dataset-based and LLM-based evaluation approaches are prone to data contamination while KIEval is resilient to data contamination. Additionally, we test existing contamination detection methods and point out their challenges.

Contamination on static dataset-based evaluation. We train two models on the test sets to introduce contamination in the pre-training (‘PT-Cheater’) and supervised fine-tuning (‘SFT-Cheater’) phases using un-tuned LLaMA-2 7B as the backboneTraining details including hyperparameters and hardware settings can be found in Appendix F.. For PT-Cheater, test set contents are integrated into the pre-training set. Subsequently, the model undergoes fine-tuning with the ShareGPT (Eccleston, 2023), a commonly used instruction-tuning dataset, to develop chat functionalities. Conversely, the SFT-Cheater replicates this process but adapts the test data to the SFT format. As a control, we also train the backbone solely with ShareGPT (‘Normal’), devoid of contamination, ensuring uniform training conditions across all models. From results in Table 3, it is clear that the accuracies for benchmarks are significantly boosted, by a large margin of over 45%, suggesting a susceptibility to data contamination. However, when faced with KIEval, the cheater models perform slightly worse than ‘Normal’ model, not positively affected by data contamination. The average rounds of valid conversation are lower in the cheater models, from the reasons specified by Figure 3, contaminated models tend to go off-topic of the conversation, repetitively stick to the incorrect knowledge making the conversation meaningless to continue. We can infer from this result that training models on test sets does not bring generalizable domain knowledge, instead, only contributing to mere memorization of knowledge from test sets.

Contamination on LLM-based evaluation. We also find existing LLM-based evaluations vulnerable to data contamination, due to their reliance on static templates. We train the fine-tuned model (‘Normal’) with MT-Bench input templates and GPT-4 outputs using only 80 samples and test it against MT-Bench and KIEval. Table 4 reveals that contamination training notably inflates the MT-Bench score by 1.79, a surge over 45% compared to the baseline, while ARC-Challenge accuracy remains stable and KIEval score slightly decreased.

Challenges in Contamination Detection. We evaluate the efficacy of current data contamination detection strategies, notably Skywork (Wei et al., 2023) and Min-K% Prob (Shi et al., 2023), which identify training data leakage through loss metrics as introduced in Related Work. We sampled 200 instances each from the trainset and testset of ARC-Challenge with contamination labels and tried to classify each instance with Min-K% Prob. We report AUC to measure its effectiveness. Table 3 demonstrates their capability to detect leaked data in the pre-training phase effectively, as the difference of average loss is significantly higher and Min-K% AUC reaches 0.89. However, both methods fail to identify contamination during SFT, with slight differences in loss values and Min-K% Prob AUC near random. We hypothesize that this may be due to the fine-tuning process only supervising the output sequence which is a short span containing the answer. This enables easy recall of the answer, without significantly impacting average loss values or Min-K% Prob values. This discrepancy underscores the ineffectiveness of loss-based metrics in discerning data contamination during SFT phase. Conversely, by leveraging KIEval, a correlation between KIEval scores and dataset accuracies emerges, suggesting the potential of KIEval in distinguishing between generalized knowledge application and mere data regurgitation for contamination detection.

3 Meta-Evaluation of KIEval

Meta-evaluation is essential for validating the practical utility of any evaluation framework. In this section, we assess KIEval’s alignment with human judgment and compare its performance against existing evaluation methods. Additionally, we conduct a cost analysis focusing on computational resources and API usage in Appendix E to validate KIEval’s cost-effectiveness and scalability.

Human Evaluation: To ascertain KIEval’s alignment with human preferences and its comparative effectiveness against prior methods, we collected a sample of multi-turn conversations generated by KIEval and compared the correlation between different evaluators and human-annotated scores. Specifically, we sampled 100 sets of conversations from all 5 datasets and across 7 candidate models and converted the multi-turn conversations to single-turn format to evaluate only one round of interaction to align with our compared baselines. Three human experts were asked to independently rate the responses of different models on a scale from 1 to 4, and we cover human annotation details in Appendix G. The Inter-Annotator Agreement (IAA) was measured by averaging Cohen’s Kappa coefficients for each annotator pair, yielding an IAA of 0.624. This indicates substantial agreement among the annotators, a significant achievement for the complexity of the task at hand. The average score for each instance was then calculated and used as the human score for that response.

Following the meta-evaluation in G-Eval (Liu et al., 2023b), we computed Pearson, Spearman, and Kendall-Tau correlation coefficients to gauge the agreement between different evaluators’ scores and human ratings. A detailed introduction of these evaluated baselines is provided in Appendix A due to page limitations.

As shown in Table 5, traditional reference-based evaluators align poorly with human judgments in open-ended conversations, due to their reliance on limited reference texts which cannot encompass the variety of valid responses. While the LLM-based evaluator MT-Bench shows commendable alignment with human preferences, its applicability is somewhat limited by its design, which is tailored to a predefined set of instructions and responses. In contrast, KIEval demonstrates a robust correlation with human preferences, underscoring its efficacy in evaluating dynamically generated, open-ended conversations.

Ablation Study: We further examine the effect of KIEval’s main components through an ablation study, the experiments and results are presented in Appendix D.

Conclusion

KIEval provides a dynamic evaluation and analysis of LLMs across various domains, evaluating generative abilities and domain knowledge through structured conversations instead of relying on fixed templates or instructions, reducing the risk of data contamination and enhancing the reliability of evaluations, while preserving alignment with human preference. Overall, our findings suggest several key insights:

Static dataset-based benchmarks may not capture the full extent of performance disparities among LLMs, such datasets could potentially underestimate these differences.

Training models on test splits of benchmark datasets primarily improves recall of answers rather than a genuine enhancement in knowledge comprehension or problem-solving abilities, underscoring the impact of data contamination.

Detecting data contamination, particularly for the fine-tuning phase of LLMs, might be challenging for existing methods. We propose a paradigm shift from only detecting exposure to specific training texts towards evaluating the models’ underlying rationale and depth of knowledge comprehension.

We believe that KIEval will serve as a valuable tool for researchers and practitioners alike, aiding in the development of more robust, versatile, and ethical AI systems.

Limitations

Our method, while insightful, operates under the assumption that LLMs can accurately evaluate the capabilities of less sophisticated models. However, the reliability of LLMs as universal evaluators is not without limitations, particularly when faced with complex benchmarks or assessing more advanced models. For certain evaluation tasks, such as mathematics problem-solving, coding, and fact-checking, depending solely on LLM evaluators may be insufficient. Furthermore, these evaluators may introduce additional biases into the assessment process. As these limitations can also be applicable to other current LLM-based evaluators, future research could explore a hybrid evaluation strategy that combines task-specific methods with LLM evaluators to achieve more nuanced and accurate assessments.

Another limitation concerns the scope of our work. Our focus is on evaluating instruction-tuned generative models with conversational abilities, excluding those designed solely for natural language understanding (NLU) tasks without generative capabilities or base models lacking instruction-following capabilities. We can assess base models by instruction-tuning them using the exact same datasets and settings, operating under the hypothesis that employing identical data for training different models results in a fair comparison. Future research should delve more deeply into the evaluation of base models, scrutinizing the impact of instruction-tuning on their performance.

References

Appendix A Baseline Evaluators

In our experimental framework, we compare KIEval with prevalent evaluators in open-ended dialogue evaluation, following Liu et al. (2023b), alongside MT-Bench, which epitomizes the application of Large Language Models (LLMs) in evaluation processes. To compare reference-based methods with reference-free approaches and our method, we use GPT-4 to generate references.

METEOR (Banerjee and Lavie, 2005), a reference-based evaluation metric, utilizes unigram matching between generated outputs and reference texts crafted by humans to assess performance across a variety of Natural Language Generation (NLG) tasks, including machine translation and dialogue generation.

ROUGE (Lin, 2004) comprises a suite of metrics for reference-based evaluation, facilitating the comparison of automatically generated summaries or translations against one or more human-crafted reference summaries or translations.

BERTScore (Zhang et al., 2019), another reference-based evaluation metric, employs contextual embeddings from BERT to measure cosine similarity between words in candidate and reference sentences. Demonstrated to align well with human judgment at both the sentence and system levels, BERTScore calculates precision, recall, and F1 scores, offering valuable insights for various NLG tasks.

MT-Bench (Zheng et al., 2023), a LLM-based, reference-free evaluation approach, harnesses cutting-edge LLMs to assess model outputs. It features a series of open-ended questions designed to test a model’s capabilities in engaging in conversation and instruction-following abilities. As MT-Bench is similar to AlpacaEval (Li et al., 2023), PandaLM (Wang et al., 2023b) and G-Eval (Liu et al., 2023b) but being a more popular option, we select MT-Bench without compromising on the breadth of our evaluation. In our meta-evaluation experiment, we use gpt-4-1106-preview as the evaluator and use the single-answer grading mode of MT-Bench.

Appendix B Datasets

We use the following datasets in our experiments, for statistics and used splits, please refer to Table 6.

ARC-Easy and ARC-Challenge (Clark et al., 2018): Both are subsets of the AI2 Reasoning Challenge, a benchmark for assessing a model’s reasoning and understanding in science questions. ARC-Easy contains simpler questions, while ARC-Challenge includes more complex ones.

HellaSwag (Zellers et al., 2019): challenges models to complete realistic scenarios in text, testing common sense and predictive abilities.

MMLU (Hendrycks et al., 2020): A comprehensive English examination composed of multiple-choice questions encompassing a wide array of disciplines. This extensive test includes subjects ranging from humanities and social sciences to hard sciences, alongside other essential areas of knowledge. It encompasses 57 distinct tasks, covering fields such as elementary mathematics, US history, computer science, law, and beyond.

C-Eval (Huang et al., 2023): A comprehensive Chinese evaluation composed of 13948 multi-choice questions spanning 52 diverse disciplines and four difficulty levels.

Appendix C Correlation Analysis of KIEval and Dataset Benchmarks

To further investigate the correlation between dataset-based benchmarks and KIEval, we use regression analysis as shown in Figure 4. We also leverage the Pearson correlation coefficient to provide quantitive analysis in Table 7. The results revealed a significant positive correlation between KIEval scores and dataset-based benchmark accuracies. This correlation underscores KIEval’s alignment with traditional evaluation methods. However, we also bring new insights that traditional benchmarks do not offer: while dataset-based benchmarks effectively assess LLM knowledge under contamination-free conditions, their results are easily inflated in the presence of data contamination. In contrast, KIEval exhibits a lower susceptibility to these issues. Visual analysis offers additional perspective by contrasting model performances as per benchmark accuracies and KIEval scores. Models significantly above the regression line suggest capabilities beyond those captured by traditional benchmarks. In this scenario, traditional benchmarks are not sufficiently challenging to effectively differentiate the stronger models from others, nor do they accurately represent the generative capabilities of these models. It is evident that GPT-3.5 is included in this category. Conversely, models falling below the regression line, exhibiting high benchmark accuracy but low conversation quality, suggest limited real-world applicability, potentially indicative of data contamination. Interestingly, the visualization shows that not only does our simulated SFT Cheater model fall into the outlier category below the regression line, but Yi-6B also exhibits similar behavior.

Appendix D Ablation Study of KIEval Components

This study assesses the impact of the decaying weight scoring and the early stopping mechanism of KIEval through an ablation analysis. Employing the same set of KIEval-generated conversations used in our meta-evaluation, we explore four distinct configurations of the KIEval framework. Specifically, we investigate the influence of the weighted scoring by replacing the decaying weight with a constant value, effectively equating the multi-round score to the mean of single-turn scores. Additionally, we examine the consequences of omitting the early stopping mechanism, thereby allowing conversations to proceed unabated until their conclusion. We then compare the correlation coefficients between these variants and human scores. As indicated by the data in Table 8, the exclusion of either feature results in a notable decline in performance, underscoring their respective contributions to the model’s efficacy.

Appendix E Cost and Scalability

Assessing KIEval’s scalability requires a thorough evaluation of overall costs. Our method employs a strong LLM accessed via API, with expenses based on input and output token lengths. Table 11 details the average token count per model evaluation across diverse datasets. Additionally, the average GPU expenditure for single model evaluations on NVIDIA A100 GPUs is provided in Table 10. Financially, deploying GPT-4 in both interactor and evaluator roles within KIEval incurs a cost of around 27 USD for each model evaluation, comprising 1000 interaction rounds. Importantly, due to our adoption of single-answer grading over pairwise comparison (Wang et al., 2023b; Zheng et al., 2023), costs increase linearly rather than quadratically with the number of models evaluated. For a comprehensive understanding of the cost implications at scale, we present a detailed estimation in Table 9.

Appendix F Experiment Details

In this section, we detail the experimental setup employed to facilitate the reproduction of our results. The entire codebase used in our experiments has been made publicly available, ensuring transparency and ease of verification for our findings.

Codebase and Dependencies: Our experiments leverage the LLaMA-Factory projectAvailable at https://github.com/hiyouga/LLaMA-Factory/., a framework designed to streamline the training of large language models. We use Huggingface Transformers (Wolf et al., 2019) library and the Deepspeed (Rasley et al., 2020) Zero-3 optimizer (Rajbhandari et al., 2021), forms the backbone of our computational experiments.

Training Configuration: For the training process, we have configured the learning rate to 2e-5, employing a cosine learning rate scheduler. Our hardware setup consists of 4 NVIDIA A100 GPUs, and we’ve set per-device batch size to 1, coupled with a gradient accumulation step of 4. We use full-parameter training for 4 epochs in all our experiments, including training models with data contamination during pre-training and fine-tuning.

Appendix G Human Annotation Details

For human annotation in our work, all annotators are authors of this paper who previously have not accessed the outputs of models in our experiments and volunteer to contribute. All three annotators agree on how the data would be used. Since the data to be annotated come from open-source datasets and popular LLMs, ethical concerns are not applicable. We provide guides for each annotator and for each annotator, we give them a unique URL to our annotation platform built with Gradio, and give them instructions: ‘You are given some conversations between a candidate model and a interactor model. Please score the response of the candidate model with integers from 1 to 4, following our scoring guide. Your score should be definitive, and consider the response’s factual accuracy, logical structure, language conciseness, and coherence.’

We measure the agreement of our annotators by calculating Cohen’s Kappa as Inter-Annotator Agreement, results can be found in Table 12. We reach an average IAA of 0.624, indicating a substantial agreement among our annotators.

Appendix H Potential Risks

While KIEval advances the evaluation of Large Language Models (LLMs), it is not without potential risks. Primarily, reliance on strong LLMs as evaluators could inadvertently propagate existing biases or limitations inherent in these models. The computational and financial costs associated with using high-performance LLMs for continuous evaluations could be a barrier to widespread adoption, particularly for researchers with limited resources.

Appendix I Use of AI Assistants

In this work, we use GitHub Copilot to assist coding, and GPT-4 to correct grammatical errors.

Appendix J Complete Experiment Results

We share the complete experiment results from all 5 datasets with 7 models, evaluated with KIEval and benchmark accuracies in Table 13, 14, 15, 16, 17.

Appendix K Complete Prompt

The system prompts for interactor, candidate and evaluator models are given in Figure 5.