EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteria

Introduction

Large Language Models (LLMs) have catalyzed the creation of a wide array of novel applications. Composed of billions of parameters and trained on billions of tokens, LLMs can interpret a natural language description of a task, a prompt, and generate coherent human-like outputs for diverse purposes (Brown et al., 2020; OpenAI, 2023; Liu et al., 2023c) (e.g., summarization (Wu et al., 2022), dialogue (Wei et al., 2023), story writing (Chung et al., 2022)). By composing a prompt, developers and researchers (i.e., prompt designers) can guide LLMs to perform novel tasks that satisfy desired requirements and support specific application settings. For example, HCI researchers have leveraged the generative capabilities of LLMs to ideate possible journalistic angles for a given event (Petridis et al., 2023), generate questions to quiz children about information they learned (Lee et al., 2023), or simplify research papers into plain language (August et al., 2023).

Although prompt designers can easily bootstrap AI-based applications by simply composing a prompt, developing a prototype into a polished application that consistently produces high-quality outputs requires more dedicated effort. As LLMs are non-deterministic and even partial changes in a prompt can significantly influence generated outputs (Lu et al., 2021; Liu et al., 2021), designers need to iterate on their prompts multiple times to achieve satisfactory results (Jiang et al., 2022; Strobelt et al., 2023; Zamfirescu-Pereira et al., 2023b, a; Wu et al., 2022; Liu et al., 2023c). In this iterative process, designers test their prompt with sample inputs (e.g., paragraphs to summarize), inspect the generated outputs to identify areas for improvement, revise their prompts (e.g., change structure, wording, content), and repeat. When designers adopt LLMs for more open-ended generative tasks, however, evaluating outputs becomes significantly more challenging as no automatic metrics can adequately encode and measure the subjective quality of outputs (Clark et al., 2021). Due to the lack of suitable automatic metrics, generative tasks are typically evaluated by human annotators or experts (Gehrmann et al., 2023), but these can be impractical during early development stages when designers need to quickly iterate on prompts.

To understand how evaluation challenges affect the development of LLM-based applications, we conducted formative interviews with 8 prompt designers (e.g., developers, and researchers in HCI and ML) to understand how they iterate on and evaluate their prompts. Our interviews revealed that designers considered multiple criteria that were unique and specific to their applications when evaluating outputs from their prompts. Due to the novelty of these criteria and the significant cost of recruiting annotators, however, designers had to manually evaluate their prompt outputs themselves. As this manual and multi-faceted evaluation of outputs incurred a significant cognitive load, designers could only evaluate small batches of outputs and only on a subset of their criteria. As a result, when they refined their prompts, designers could not fully verify how their refinements had affected output quality or identify where further refinements were needed.

Based on these findings, we introduce EvalLM to facilitating prompt iterations by supporting the evaluation of outputs on user-defined and application-specific criteria (e.g., measuring “Object Familiarity” in scientific analogies for children). Instead of focusing on the low-level task of assessing generated outputs, EvalLM shifts designers’ focus to the higher-level process of refining prompts and criteria—representations of their plans and requirements. Inspired by the methodology for developing and validating psychometric scales (Robinson, 2018; Boateng et al., 2018) and techniques for LLM-based evaluations (Zheng et al., 2023; Ye et al., 2023; Liu et al., 2023a), EvalLM employs an LLM as both (1) an evaluation assistant, which evaluates outputs on the defined criteria, and (2) a criteria reviewer, which revises the defined criteria. To aid users in revising their prompts and criteria, the evaluation assistant explains its assessments, allowing the user to identify where prompt outputs fell short or to identify where the assistant’s interpretation of criteria misaligned with their own. Furthermore, the criteria reviewer analyses the user’s criteria to identify revisions that can lead to evaluations of outputs on more specific and fine-grained dimensions. Through iterations of this collaborative process, designers co-evolve their prompts and criteria, where prompts improve to satisfy criteria and criteria improve to discern the quality of prompts—ultimately leading to more polished applications.

To understand how prompt designers adopt automatic evaluations during prompt iterations, we conducted a within-subjects study (N=12) where participants improved and evaluated prompts for novel tasks proposed by recent HCI work. In the study, participants used both EvalLM and a baseline where they manually evaluated outputs—emulating designers’ current practice. Our study revealed that EvalLM helped participants “debug” their prompts by allowing them to quickly identify areas for improvement, and the evaluation assistant’s explanations served as feedback by helping participants think about how to make these improvements. As a result, we observed that participants reached satisfactory prompts more efficiently as they tested 59% fewer changes compared to when they did not have evaluation assistance. As EvalLM also facilitated criteria revision, participants felt higher satisfaction regarding the quality of their criteria—suggesting that these criteria could be valuable during human evaluations. Overall, these findings suggest that EvalLM can fill the current gap between application development and deployment by assisting designers to iterate on prompts until a stage where they have the confidence to commit resources for more robust human evaluations.

This work presents the following contributions:

Qualitative findings from interviews with prompt designers ($N=8$) that revealed how the effort of manually evaluating outputs on multiple, task-specific criteria can inhibit designers from making informed decisions during the iteration process.

EvalLM, an interactive system that aids users in revising prompts and verifying the effect of revisions by employing an LLM-based evaluation assistant to assesses outputs on user-defined criteria, and a criteria reviewer to refine these criteria to assess more specific and detailed dimensions of outputs.

Findings from a user study ($N=12$) that demonstrated how EvalLM can aid designers in debugging their prompts and ideating on strategies to more effectively revise their prompts.

Related Work

This work aims to support the iteration of LLM prompts for novel generative tasks by supporting interactive evaluation of outputs. To understand this space, we review literature in (1) prompt design challenges and related support, (2) natural language generation, and (3) interactive evaluation in broader machine learning.

Although LLMs facilitate the development novel AI-based applications without the need for data collection or model training, designing satisfactory prompts can be an arduous task (Liu et al., 2023c). The specific format, phrasing, content, examples, or even the order of examples used in prompts can significantly affect performance (Lu et al., 2021; Liu et al., 2021; Rubin et al., 2021; Zhao et al., 2021; Liu et al., 2023c). However, as the space of possible natural language instructions is near infinite, designers need to test as many possibilities as possible to identify high-performing prompts (Zamfirescu-Pereira et al., 2023a; Liu et al., 2023b). To help designers identify effective prompts, researchers have proposed various tools that facilitate prompt design. For example, AI Chains (Wu et al., 2022) helps users decompose complex tasks into a chain of prompts that can be tested individually, and Kim et al. (Kim et al., 2023b) proposed a design framework for interfaces that support end-users to testing and experimentation with prompts. To automate testing, various systems (Strobelt et al., 2023; Wu et al., 2023; Mishra et al., 2023) allow users to create prompt variants and automatically compare their performance. However, while these prior approaches focused on classification tasks, evaluating performance in open-ended generative tasks can be more complex as outputs need to be examined on subjective criteria, which typically requires human effort. Building on these systems, our work proposes a human-AI collaborative system where users define subjective evaluation criteria and an LLM automatically assesses outputs on these criteria to provide insight into the performance of prompts and surface necessary improvements.

2. Natural Language Generation and Evaluation

Natural language generation (NLG) is the family of NLP tasks where the goal is to generate text that satisfies a communicative goal (e.g., summarize a document) while possessing several desired qualities (e.g., fluent, coherent) (Gehrmann et al., 2023; Nallapati et al., 2016; Gopalakrishnan et al., 2023; Fan et al., 2018). While recent years have shown significant progress in NLG, especially due to LLMs, a constant obstruction to progress has been the difficulty of evaluating these tasks (Khashabi et al., 2021; Dou et al., 2021; Xiao et al., 2023). Unlike classification tasks where performance is measured by comparing a predictions to a ground-truth label, generation tasks are ill-posed—i.e., multiple dissimilar outputs can be equally valid. While researchers have proposed automatic metrics that compare outputs to several ground-truth references (e.g., BLEU (Papineni et al., 2002), ROUGE (Lin, 2004)), the space of valid outputs in open-ended generative tasks can be overwhelmingly vast, making it nearly impossible to create sufficiently comprehensive references sets. Thus, human evaluations where annotators rate or rank generated text have become the golden standard in the domain (Gehrmann et al., 2023). However, the cost and effort involved in recruiting human annotators can make this type of evaluation prohibitive during earlier development stages. As an alternative, recent work (Li et al., 2023; Zheng et al., 2023; Fu et al., 2023; Liu et al., 2023a; Ye et al., 2023; Chiang and yi Lee, 2023) has employed LLMs to simulate annotators and automatically evaluate outputs on their general quality or a pre-defined set of criteria—demonstrating agreement with human evaluations on par with the level of agreement between human evaluators (Zheng et al., 2023). Inspired by these approaches, we incorporate LLM-powered simulated evaluators to support interactive evaluation of LLM prompts on user-defined criteria, and investigate how users interact with these evaluations to refine prompts.

3. Interactive Machine Learning Evaluation

Evaluation is fundamental to the development of machine learning models for real-world applications. Beyond assessing performance on a single metric, practitioners (e.g., developers, researchers, engineers) assess more fine-grained model behaviors to identify flaws and potential improvements (Ribeiro et al., 2020). To support fine-grained assessments, prior work has introduced various systems that allow practitioners to interactively evaluate outputs. For example, Zeno (Cabrera et al., 2023), the What-If Tool (Wexler et al., 2020), and Errudite (Wu et al., 2019) help practitioners to identify slices or subsets of data that may reveal distinct model failures. Beyond testing on existing input data, Polyjuice (Wu et al., 2021) and AdaTest (Ribeiro and Lundberg, 2022) allow practitioners to generate potentially challenging input data to test a model’s behavior. To aid practitioners to resolve issues after models are deployed, Angler (Robertson et al., 2023) combines online and offline data to help practitioners prioritize performance issues, and Deblinder (Cabrera et al., 2021) allows practitioners to collect and analyze model failure reports from crowdworkers. Our work expands on these ideas by allowing practitioners to interactively evaluate prompt outputs on fine-grained criteria with the aid of an LLM-based evaluator, which could simulate the feedback provided by end-users.

Formative Interviews

To understand current practices and challenges when evaluating and iterating on LLM prompts, we conducted interviews with prompt designers. These interviews focused on understanding how prompt designers evaluate performance during early development stages and how these evaluations inform their refinement of prompts.

We recruited 8 prompt designers through posts on online forums within our institution and word-of-mouth. These participants held various roles related to generative applications, and came from both academia and industry: 2 graduate students in HCI (1 MS, 1 PhD), 2 MS students in ML/NLP, 2 research scientists at a large company, 1 data scientist at a startup, and 1 startup CEO. All of the participants mentioned working on at least one project where they designed prompts for a novel generative applications. Their applications covered diverse contexts: social media, document question-answering, image captioning, writing, teaching, and intelligent assistants.These are described broadly to preserve confidentiality. The length of their experiences with intensive prompt engineering ranged from 4 months to more than 1 year. Participants were compensated with approximately 45 USD (60,000 KRW) for the 1-hour interview. The interviews were conducted in a semi-structured format, and were recorded, manually transcribed and coded through a thematic analysis.

2. Findings

All of the designers mentioned working on applications for which they defined novel generation tasks. These tasks were novel as they (1) introduced new requirements to pre-existing generation tasks, or (2) were not analogous to any pre-existing tasks, according to participants. To develop the prompts for these applications, all of the designers first composed an initial prompt based on a preliminary set of expectations and requirements and, then, iteratively evaluated and refined their prompt to guide the LLM to better meet these expectations.

All of the designers mentioned how, at each iteration step, they tested their prompts on sample inputs and then manually evaluated the outputs themselves. Designers mentioned that they had to evaluate manually since they considered aspects that were subjective (P1-3, P5-8) and specific to their task (P1-8), meaning that there were no existing automated metrics to measure these aspects. Furthermore, since they were still in the development stage, recruiting annotators would be prohibitively expensive and would slow down the iteration process (P1-2, P4). However, all of the designers mentioned how manual evaluation could be demanding and time-consuming and, thus, they only tested prompts on a small set of sample inputs (i.e., one to three samples) at each iteration step—which was still taxing especially with lengthier outputs (P1-2).

2.2. Evaluation is Multi-Faceted

Due to the complexity of the designers’ intended applications, performance or the quality of the outputs could not be determined with a single criterion. Instead, designers considered multiple criteria or factors simultaneously when examining outputs, which made evaluation significantly more challenging (P1, P3-5). P5 mentioned how they had to carefully examine the outputs to “catch all the subtleties”. As this involved significant cognitive load and effort, designers described various ways in which they handled this multi-faceted evaluation. For example, four designers (P2, P4-5, P7) said that they only focused on the most important criteria for their task, and two others (P5-6) simplified assessments to assigning binary ratings on each criterion—i.e., whether the criterion was satisfied or not. Alternatively, P4 resorted to evaluating and refining one criterion at a time, but P7 noted that this method did not work for them as refining the prompt for one criterion led to the prompt failing at previously “resolved” criteria. Thus, while designers ideally wanted to evaluate holistically on multiple criteria, the effort required could lead them to only partially evaluate outputs.

2.3. Evaluation is Dynamic

Several designers described how they started the prompt design process with an initial set of evaluation criteria based on the intended goals for their application (P1-3, P5) and prior work (P2-5). Additionally, designers also expanded and transformed their criteria in each prompt iteration. By examining outputs, they identified additional criteria to consider as they observed flaws in the outputs, which they had previously not expected, or because they recognized other aspects that they wanted to improve on (P1-2, P4, P7-8). Beyond adding criteria, designers also mentioned how they had to concretize their criteria by determining how they should be evaluated. However, as these criteria could be subjective, it could be challenging to define what “success” meant for each criterion (P4-7). For designers who worked in teams, they mentioned how they would concretize the criteria by discussing with team members who could provide different perspectives (P4-5).

2.4. Evaluation to Refinements

Through the evaluations, designers identified what criteria the generated outputs failed to satisfy, and they attempted to refine their prompts to improve on these dimensions. However, most designers (P1-7) mentioned how they were unsure about how they should revise their prompts—a well-known challenge with LLM prompts (Zamfirescu-Pereira et al., 2023a). Designers mentioned how they had no alternative, but to simply test different changes and to manually evaluate outputs again to check the effect of these changes. As this involved significant effort, designers mentioned that they could struggle to verify how much a revision improved quality on specific criteria (P3-4, P6). Due to the overall complexity of the evaluation-refinement process, designers also mentioned how they would fail to record all of their prompt revisions and their effects, which prevented them from learning from previous attempts and tracking their progress (P4-5, P7).

Design Considerations

To support prompt iteration, we must help designers to efficiently evaluate various outputs on various criteria, while also allowing them to flexibly define these criteria. Recent work (Ye et al., 2023; Liu et al., 2023a) has demonstrated that LLMs can evaluate outputs on various subjective criteria when provided with natural language descriptions of said criteria. By leveraging LLMs as evaluation assistants, we could enable designers to efficiently evaluate larger samples on diverse criteria of their own design. However, for designers to effectively use LLMs as evaluators, they must be able to define these criteria and verify that the corresponding evaluations align with their expectations. To scaffold these interrelated processes, we take inspiration from the process of developing and validating psychometric scales.

A psychometric scale (Robinson, 2018) is a set of questions that collectively measure a variable (e.g., behavior, feeling, or action) that cannot be assessed directly. For example, the NASA task load index (NASA-TLX) questionnaire, which is frequently used in HCI, measures a person’s perceived workload (Hart and Staveland, 1988). Similar to these scales, evaluation criteria for generative tasks collectively measure the quality of outputs—a variable that cannot be directly measured. To support the construction of evaluation criteria, we adapt the three main stages in the development process of scales (Robinson, 2018; Boateng et al., 2018): item development, scale development, and scale evaluation.

This stage involves designing the items or questions to include in the scale. Literature suggests combining both inductive (e.g., data analysis) and deductive (e.g., literature review) methods to design questions for psychometric scales. After designing the items, the items are frequently assessed by external judges to determine whether they are relevant and clearly defined.

1.2. Scale Development

Then, a scale is administered to a small set of respondents and their responses are used to assess the scale’s quality through multiple methods, including cognitive interviews and item reduction analysis. In cognitive interviews, respondents are asked to verbalize their mental process as they respond to the scale, which reveals whether items are producing the intended data and if there are any unclear items. In an item reduction analysis, responses are analyzed to determine whether there are items that assess similar variables and, thus, could be removed.

1.3. Scale Evaluation

Finally, scales are administered to a larger set of respondents to test their validity and reliability. Validity tests compare responses on the scale with ground-truth measurements of the variable or other similar variables. Reliability tests check whether the same respondent provides consistent responses at different points in time (i.e., test-retest reliability) and whether multiple respondents agree in their responses about the same subject (i.e., inter-rater reliability). In our work, we only take inspiration from reliability tests as designers do not have the ground-truth measurements needed for validity tests.

2. Design Goals

Based on the challenges identified from our interviews and insights from the literature on psychometric scales, we distill the following design goals:

DG1: Automate evaluation of generated outputs according to user-defined criteria. An automatic evaluation assistant can reduce effort by providing an initial assessment of outputs that designers can then verify. By defining their own criteria, designers can align the assistant’s assessments with their own expectations and requirements.

DG2: Facilitate inspection of automatic evaluations through explanations. Similar to cognitive interviews in the development of psychometric scales, the automatic evaluator should explain and justify its evaluations so that designers can inspect whether the evaluations align with their expectations.

DG3: Allow for the definition of criteria based on output data and prior literature. As revealed by our interviews, prompt designers also employ both inductive and deductive methods when defining evaluation criteria as they envision new criteria while assessing outputs but also adapt criteria from prior work.

DG4: Review the user-defined criteria to identify potential revisions. Inspired by how scales are revised through reviews from external judges and analytical methods (e.g., item reduction), a system can review designers’ criteria to identify potential revisions, which could increase the effectiveness of subsequent evaluations.

DG5: Surface unreliable evaluations during large-scale evaluations. To gain a more comprehensive understanding of their prompt’s performance, designers may evaluate their prompt on large samples. As designers might not be able to verify each evaluation, a system can identify inconsistent or unreliable evaluations that designers can focus on checking.

DG6: Aid designers in tracking and comparing the effect of prompt changes. As stated in the interviews, it can be challenging to understand the effect of each prompt change. By helping designers track how changes affect performance in the automatic evaluations, designers can make more informed iteration decisions.

EvalLM

Based on these design goals, we present EvalLM, an interactive system for iterating on prompts by evaluating multiple outputs on multiple user-defined criteria. While designers typically compose prompts and assess outputs to provide feedback to themselves, EvalLM transforms this into a collaborative process where a designer iteratively refines prompts and criteria based on feedback from an LLM (DG1). Specifically, the system employs an LLM as both an evaluation assistant and criteria reviewer. The evaluation assistant judges prompt outputs according to the user’s definitions of criteria and explains its evaluations, allowing the user to identify where their prompts fall short or where their criteria may be unclear (DG2). The user can define and revise criteria at any moment of time (DG3) and, to facilitate this process, the criteria reviewer analyzes the user’s criteria to recommend potential revisions that can enhance the detail and specificity of the evaluations (DG4). The main screen of EvalLM consists of three panels (Fig. 2): the generation panel (left), the data panel (middle), and the evaluation panel (right).

To illustrate the interactions and mechanisms in EvalLM, we walk through an example of an ML practitioner, Emily, who is designing prompts for a novel task of generating examples that help young children understand complex scientific phenomena.

In the generation panel, the user writes the overall instruction for their task (Fig. 2A) and designs two prompt templates (Fig. 2B). EvalLM is designed for comparing prompts as this enables designers to compare the performance of different prompt variations, or to compare prompts before and after specific edits (DG6). Furthermore, prior work has found that it is easier for both humans (Gehrmann et al., 2023; Li et al., 2019) and LLMs (Zheng et al., 2023; Bai et al., 2023) to compare model outputs than to rate a single output. For each prompt template, the user can compose the system and user prompt (Fig. 4D-E). In either of these, the user can add the tokens {{instruction}} and {{input}}, which are replaced with the instruction and the content of an input sample when the prompts are used to generate outputs. By composing the overall instruction separately, users can re-use the same base instruction across prompt templates, and test the effect of adding more information to a prompt or changing its format. To test and compare different prompt ideas, the user can create more prompts (Fig. 4B), name them, and switch between them as they desire (Fig. 4C).

In the Instruction, Emily describes her task: writing an example to help explain a piece of scientific information to a young child. To test the effect of different prompting “tricks”, Emily first creates a basic prompt that follows a simple form-like format, and creates another prompt with sub-headers, additional context, and a system prompt that instructs the LLM to act as a teacher.

1.2. Sampling Inputs and Generating Outputs

To test their prompts, the user can upload their own input dataset and then sample inputs from this dataset (Fig. 2C). Users are provided with two ways for sampling inputs: (1) manual, which opens a panel where the user can browse through their dataset and choose samples, and (2) diverse, which automatically samples distinct data points (details in §5.3). When the user samples inputs, these are shown as rows in the data panel, where the first column shows the content of the input sample (Fig. 5A). Then, the user can click on Run Prompts to generate outputs for each of these sampled inputs with each of their prompts. The outputs from each prompt are shown side-by-side in the data row (Fig. 5B).

1.3. Defining Criteria

In the evaluation panel, the user can define and manage their evaluation criteria (Fig. 2D). The user defines new criteria by writing its name and providing a description for the criteria, which can explain what the LLM should assess when evaluating that criteria or describe the characteristics that an output must possess to satisfy that criteria (Fig. 4). Instead of defining their own criteria from scratch, the user can also browse through the Criteria Dictionary to select from criteria that were defined in prior work (DG3). This dictionary can serve as a starting point by providing initial criteria descriptions that the user can adapt to their context, or as inspiration by helping users consider other aspects to evaluate.

As Emily knows that explanations to children should only contain familiar words and information, she creates a criteria called “Familiarity” to assess whether generated examples use language and situations that a child can understand. Then, to decide on what else to evaluate, Emily browses through the dictionary and finds the “Faithfulness” criteria, which checks that summaries are devoid of factual errors (Krishna et al., 2023). As it is also important to verify that examples are faithful to the scientific information, Emily adds and edits the criteria to fit her context.

1.4. Revising Criteria: Refine, Merge, and Split

To improve the quality of their evaluations, users may need to iteratively revise their criteria. To help users identify potential improvements in their criteria (DG4), EvalLM provides the Criteria Review Tool that checks the criteria on their (1) clarity and relevance, (2) redundancy, and (3) granularity. The tool automatically reviews the user’s criteria and, if it identifies criteria that could be improved, the system displays badges next to the criteria name (Fig. 4C) that the user can click to see the recommended improvements (Fig. 4D). Similar to how external judges assess psychometric scales, the tool identifies criteria that are unclear or irrelevant, and suggests how to refine the criteria to increase their clarity and relevance with the user’s task. Inspired by item reduction analysis, the tool also identifies criteria that may be redundant and suggests how these could be merged into one single criteria. Finally, as prior work has demonstrated that human evaluators are more accurate when evaluating fine-grained criteria (Gehrmann et al., 2023; Krishna et al., 2023), the review tool identifies criteria that may be coarse-grained and suggests how to split them into multiple fine-grained criteria. While the user can manually activate the criteria review tool, it also runs automatically if the user has not modified their criteria for a certain period of time.

As she was reading through the input samples, Emily notices that the review tool has suggested that she splits the “Familiarity” criteria into “Language Simplicity” and “Relatability”. She agrees that, while these suggested criteria are related, they assess different aspects of the outputs. She decides to add these two suggestions and remove her previous “Familiarity” criteria.

1.5. Evaluating Outputs

After generating outputs and defining their criteria, the user can click on Auto-Evaluate to automatically evaluate the output pairs on each of the criteria (DG1). For each criterion, the evaluation assigns each output in a pair with a score out of 10 and, based on these scores, the interface presents which output was better at satisfying that criterion or if there was a tie (Fig. 5C). If the user wants to understand the evaluation results, they can click on the criteria name to view the explanation for that evaluation and the scores assigned to each output (Fig. 5D). To help the user verify the explanations without dedicating substantial effort into fully reading the outputs (DG2), the interface highlights fragments from the outputs (Fig. 5E) that the evaluation assistant focused on more when evaluating that criterion.

To provide a bigger picture on the performance of the prompts, the interface also displays statistics of the evaluation results (Fig. 2E). For each criterion, the interface presents the proportion of samples where each prompt “won” and the proportion of “ties”.

After generating and evaluating outputs on three samples, Emily checks the overview of results and sees that her improved prompt won the most in all three criteria. However, she notices that there was one sample where the original prompt won in the “Language Simplicity” criteria. To check why this happened, she looks for the sample and opens the explanation for that evaluation, which describes that the output from her prompt used more complex terms. To improve on this, Emily adds an additional requirement to her prompt that instructs the LLM to avoid using complex terms and to always simplify them first.

As the evaluations are conducted by a non-deterministic LLM, the evaluation results may differ in every run. To increase their assurance on the evaluation results, the user can increase the number of evaluation trials that are performed. If the number of evaluation trials is set to three, the interface evaluates each output pair three times and decides on the “winner” for each criteria based on which output won the most number of trials (i.e., majority vote). The user can check the evaluations for each of these trials by opening the explanation box and using the carousel at the bottom (Fig. 5F).

1.6. Additional Features: History and Validation

To help the user keep track of their iterations, EvalLM automatically records all evaluations (DG2). The user can click on the “History” button (Fig. 2E) to view a visualization of their evaluation history, and look back at prompts and criteria used in those evaluations (Fig. 7). Additionally, the system allows users to store generated samples into a validation set where they can annotate their own ground-truth evaluations. With a populated validation set, the user can click on Validate Criteria (Fig. 2F) to evaluated these samples and check how accurately the automatic evaluator predicts the user’s own ground-truth evaluations.

1.7. Experimenting on Larger Samples

After designers have developed their prompts and criteria, they may wish to verify their prompt’s performance by testing on significantly larger samples. EvalLM provides an Experiment screen (full figure in Appendix A), where the user can automatically generate and evaluate larger samples and check the reliability of the evaluations. In this screen, designers can increase the number of evaluation trials and select another LLM as an alternative evaluator (e.g., ChatGPT or PaLM 2 (Anil et al., 2023)). When the designer runs an experiment, the system automatically samples diverse inputs, generates outputs with these inputs and the chosen prompts, and then evaluates the outputs on the chosen criteria for the configured number of trials. If an alternative evaluator was selected, this LLM is used to evaluate the same outputs on the same criteria for the same number of trials. The Experiment screen shows two additional statistics (Fig. 7): test-retest reliability or the consistency of evaluations between trials, and inter-rater reliability or the consistency between the evaluations by the system’s LLM (i.e., GPT-4) and the alternative evaluator. These statistics are shown as stacked bar charts that present the proportion of samples where evaluations were consistent or not, and the user can click on a bar to only display these unreliable cases (DG5).

After multiple iterations, Emily moves to the Experiment screen to choose between two promising prompts. After running an experiment with 40 samples and ChatGPT as the alternative evaluator, Emily sees that her first prompt excels at “Language Simplicity” while the second outperforms in “Scientific Accuracy”, representing a trade-off between the criteria. Additionally, she notices that the evaluators disagree often in “Language Simplicity” so she clicks on that bar to browse through these cases. By browsing through, Emily notices that the evaluation assistant may be unreliable as it also assessed sentence complexity for this criteria. As she wants it to focus only on vocabulary, she changes the criteria to “Simple Vocabulary”.

2. Prompting Techniques

Our interface is powered by two LLM-based components: automatic evaluation, and criteria review. In this section, we describe the design of these prompts and we include the full prompts in Appendix B.

The main goal of EvalLM is to evaluate prompt outputs by comparing them on their ability to satisfy a set of user-defined criteria. To this end, we designed our evaluation prompt by adapting the prompt design from two state-of-the-art approaches for LLM-based evaluation: LLM-as-a-judge (Zheng et al., 2023), which compares model responses on their overall quality, and FLASK (Ye et al., 2023), which rates a single model response on how it performs on multiple “skills” or criteria that are described in natural language. Our prompt takes as input a task instruction, an input sample, a pair of outputs, and a list of criteria descriptions. Then, our prompt instructs an LLM to evaluate the output pair on each criterion by (1) explaining how the outputs satisfy the criterion, extracting evidence fragments from each output, and then providing each output with a score out of 10. To design this prompt, we considered alternative approaches for each of these components.

Instruction and Input: We only include the overall instruction that is shared between the generation prompts to provide the evaluator LLM with context about the task while keeping its evaluations prompt-agnostic to limit potential bias due to the content or wording of each generation prompt.

Output Ordering: Prior work has found that LLM-based evaluations have positional bias (Wang et al., 2023a) where they frequently favor the first output. While this work recommended evaluating each candidate in each position and aggregating these evaluations, as this would introduce additional cost and delays, our system instead alternates the positions of outputs every time it evaluates.

Criteria Descriptions: Similar to Ye et al. (Ye et al., 2023), the criteria are added to the prompt as a list of lines of the form: “name of criterion: description of the criterion.” However, in their work, they also designed scoring rubrics for each criteria that describe the meaning of each numeric score. In contrast, we opt for only including the criteria descriptions as designing rubrics may require excessive effort from users and, according to our preliminary tests, automatically generated rubrics could negatively affect evaluation results.

Explanation: The prompt instructs the LLM to provide an explanation where it compares and contrasts between the outputs as this explanation is later shown to users, but also because this could elicit reasoning from the LLM and increase performance (Wei et al., 2022). While we considered an initial design where the LLM was instructed to describe the performance of each output separately as this could lead to more in-depth examinations, initial tests and pilot studies revealed that this produced repetitive and less useful explanations.

Evidence: To help users associate the evaluation explanations with the outputs, the prompt instructs the LLM to extract fragments from the outputs that are relevant to its explanation and evaluation. While we considered a design where the LLM first extracted evidence and then cited these in its explanation, we observed that the resulting explanations would simply list these fragments without providing in-depth explanations of how the outputs satisfied the criteria.

2.2. Criteria Review

To automatically review the user’s criteria and provide suggested revisions, we designed three prompts for each type of supported review: refining, merging, and splitting. Instead of designing one prompt that conducts all of these reviews, we designed separate prompts for each review as certain criteria may require multiple revisions and we wanted to allow users to flexibly choose between these. These three prompts follow the same general design: given a task instruction and a set of criteria, the LLM is instructed to review each criterion and identify any faulty ones, explain how these can be revised, and then return new criteria that result from these revisions. The three prompts differ in terms of what determines a criteria to be faulty and how they should be revised:

Refining: Identifies criteria that are confusing or imprecise, and revises these to be clearer and more specific.

Merging: Identifies criteria that may measure similar aspects, and combines these into one joint criterion.

Splitting: Identifies criteria that are excessively broad and consider multiple unrelated aspects, and divides these into more specific and mutually exclusive criteria.

Additionally, all of the prompts instruct the LLM to ensure that the suggested criteria (1) are clear and concise, following the requirements of psychometric scales (Robinson, 2018), and (2) do not remove or add new information. Additionally, as LLMs tend to be overeager to follow instructions, which could lead to excessive revisions of criteria, the prompts explicitly mention that it is possible for all of the criteria to be satisfactory and not require any revisions.

3. Implementation Details

We implemented the front-end of EvalLM using TypeScript, ReactJS, and CSS. The back-end was implemented as a Flask server and the OpenAI APIhttps://platform.openai.com/ for all LLM components. In terms of the LLM settings, we set the temperature to 0.3 for all components. The automatic evaluation and criteria review tool used the gpt-4-0613 model and, as LLMs are prone to self-enhancement bias where it rates its own outputs highly (Zheng et al., 2023; Li et al., 2023), we use gpt-3.5-turbo-0613 when generating outputs. Finally, to support diverse sampling of inputs, we automatically cluster samples in the uploaded datasets by embedding data points using the OpenAI API with the text- embedding-ada-002 model, and then clustering these embeddings using the KMeans algorithm in the scikit-learnhttps://scikit-learn.org/ library. When the user chooses to sample diversely, the system chooses data points from distinct clusters.

Technical Evaluation

We conduct a small-scale technical evaluation of our LLM-based evaluation approach to understand how performance is affected with more task-specific criteria, and to gain a more in-depth understanding of the LLM’s explanations for its evaluations.

While prior work in NLP has assessed the performance of LLM-based evaluations (Zheng et al., 2023; Ye et al., 2023; Liu et al., 2023a), these focused on evaluating outputs on overall quality or based on pre-defined criteria. As our work employs LLMs to evaluate task-specific criteria, we conduct a technical evaluation to assess how this affects evaluation performance.

We assess LLM evaluations by comparing them to human evaluations in the MT-Bench dataset (Zheng et al., 2023). This dataset presents 80 user requests of diverse categories (e.g., writing, roleplay, math, coding) and the responses from various LLMs to each request. These responses are paired and, for each pair, the dataset provides votes from one to three human annotators on what response was better or if there was a tie. For our evaluation, we selected 19 requests in the writing and role-playing category as they involve the most subjectivity. As our evaluations focus on prompts rather than requests, we decompose these requests into a prompt-input format. We include examples of the original requests and their prompt-input versions in Appendix C, and the full data in the Supplementary Materials.

1.2. Conditions

We compare LLM evaluations in three conditions:

Overall-Quality: We adopted the prompt from LLM-as-a-judge (Zheng et al., 2023) that compares outputs on overall quality.

General-Criteria: We used our evaluation prompt with general criteria from FLASK (Ye et al., 2023). In this work, they instruct an LLM to select three criteria (out of their set of 12) that are the most relevant to a given request. We followed their approach to select three criteria for each of the 19 tasks.

Specific-Criteria: We used our evaluation prompt with criteria that were automatically adapted for each task. For each task, we start with the same criteria from the General- Criteria condition, but automatically split and refine them using our criteria review prompt (i.e., all automatic suggestions are taken).

For all evaluations, we use the gpt-4-0613 model with temperature set to 0 for reproducibility. Additionally, due to the positional bias of LLMs, we run evaluations twice with each output in each position and then average the scores for each criterion.

1.3. Measures

To aggregate the criteria-wise evaluations into a single vote, we determine the vote for each criterion, and then calculate the majority vote across the criteria. Following LLM-as-a-judge, we calculate the agreement between human and automated evaluations based on two cases: (1) if the majority of human evaluators agreed on a vote, then we count an agreement if the LLM evaluation agrees with this majority vote, or (2) if there was no majority vote between human evaluators, then we calculate the proportion of annotators that the LLM evaluation agreed with. Also, we calculate the Fleiss’ kappa between the LLM evaluations and the majority vote of human annotators.

1.4. Results

Overall, Specific-Criteria had the highest agreement and correlation with human annotators and General-Criteria had the lowest (Table 1). By qualitatively reviewing cases, we observed that the Specific-Criteria condition could lead to more balanced evaluations compared to the other conditions. For example, when assessing the quality of generated travel blogs, Overall-Quality only assessed how many diverse attractions were covered, while Specific-Criteria also assessed the level of detail in which attractions were covered. Additionally, compared to General-Criteria, we observed how Specific-Criteria assessed various specific requirements that were posed by the prompt, enabling it to better capture how well the outputs followed the prompts. We also found cases where Specific-Criteria was less successful as each criterion was given equal weighting but certain criteria should hold precedence—e.g., not providing an incorrect medical diagnosis should be more important than the readability of an output. These preliminary findings suggest that instructing LLMs to evaluate more fine-grained and specific criteria can increase their ability to assess the alignment of outputs with instructions.

2. Evaluation Explanations

Additionally, we assess the quality of the LLM-generated explanations. Prior work did not assess the quality of the LLMs’ explanations during evaluations as their purpose was only to induce chain-of-thought (Wei et al., 2022).

We sampled two evaluations by the Specific- Criteria condition for each of the 19 tasks, resulting in a total of 38 output pairs evaluated and 194 criteria-wise evaluations. Each evaluation was annotated for the presence of any errors regarding five criteria: (1) logical (i.e., the explanation presents logical and coherent arguments and justifications), (2) faithful (i.e., the explanation does not hallucinate content that does not exist in the outputs), (3) independent (i.e., the explanation does not assess other criteria or aspects not described in the evaluation criterion), (4) evidential (i.e., the evidence extracted is relevant to the explanation), and (5) score aligned (i.e., the final score aligns with the explanation provided). We recruited two annotators, who had previous experience grading written assignments, and instructed them to mark these errors even if only part of the explanation presented the error. Since there is a large class imbalance (i.e., most explanations have no errors), we considered an explanation to have errors if at least one of the evaluators marked an error.

2.2. Results

Overall, we found that the explanations were mostly free of issues: 91.4% of the explanations were logical, 99.1% were faithful, 84.2% were independent, 100% provided relevant evidence, and 98.6% were aligned with their scores. We qualitatively reviewed erroneous cases and observed that the LLM evaluations frequently failed to produce independent explanations as they would contrast outputs on their level of detail despite the criterion not assessing this. In terms of logic, the evaluations struggled to assess creativity and could be too superficial in their interpretations. For example, a story was considered novel for describing a character as “the last surviving member of a wizard family”, and news article headlines were considered to address ethical dilemmas since they explicitly included the phrase “ethical concerns”. For faithfulness, there were cases where the evaluations could not accurately measure the length of the outputs. Finally, for score alignment, there were cases where the explanations did not mention that one output was better than the other but provided one output with a higher score. These results show that GPT-4 is able to produce relatively sensible explanations for its evaluations, but may at times be limited by the model’s bias towards detail and its logical capabilities.

User Study

To understand how the EvalLM affects the prompt iteration process when compared to following the current practice of designers, we conducted a within-subjects study where we compared EvalLM to a baseline where participants manually evaluate outputs. In this study, we aimed to answer the following research questions:

RQ1. Can EvalLM aid designers in deciding on how to revise their prompts and verifying the effectiveness of these revisions?

RQ2. How do designers define their own criteria for given generation tasks and how does the EvalLM’s criteria review tool support revisions on these criteria?

RQ3. How do designers interpret and gauge their trust in the evaluations by EvalLM?

We recruited 12 participants through posts on online forums within our institution. All of these participants reported having extensive experiences with prompting: nine had designed prompts for research-based applications, one designed prompts for toy projects, and two described frequent use of LLMs for productivity. Regarding the length of their experiences, four had between 1 to 3 months of experience, three had 3 to 6 months of experience, four had 6 to 12 months of experience, and two had 1 to 2 years of experience. Participants were compensated with approximately 60 USD (80,000 KRW) for the 2-hour study.

1.2. Conditions

During the study, participants designed prompts and evaluated their outputs for two given tasks. For each task, participants either used EvalLM in one of two conditions: Assist or Manual. The Assist condition was the full EvalLM interface, while the Manual condition was the EvalLM interface without the evaluation assistant or the criteria review tool. In the Manual condition, participants defined their own criteria or selected from the dictionary, and then evaluated each data sample by choosing which output won for these criteria. This condition supports designers’ common practices—according to our formative interviews—where they copy generated outputs into a spreadsheet to manually check which criteria were satisfied by each output.

1.3. Tasks

All of the participants designed prompts for the same two tasks. As our work focuses on supporting the design and evaluation of prompts for novel tasks, we adapted tasks from recently proposed LLM-powered HCI systems. Specifically, we chose the following two tasks: (1) write an example that can explain a piece of scientific information to a young child—based on Lee et al.’s DAPIE system (Lee et al., 2023)—and (2) ideate a list of alternative angles for a news story—based on Peridis et al.’s AngleKindling system (Petridis et al., 2023). We chose these two tasks as they target a specific user population (i.e., children and reporters) and no significant expertise is needed to understand the task outputs. Additionally, these two tasks are different in terms of their goal (i.e., explaining vs. brainstorming), and how the output relates to the input (i.e., transforming vs. expanding on the information).

1.4. Procedure

Participants were asked to sign the informed consent form prior to the study. After a brief introduction to the study, participants answered a pre-task survey. After a walkthrough of the first interface, participants used this interface to design a prompt for the first task.The order of conditions and tasks were counterbalanced to mitigate ordering effects. Participants were asked to envision themselves as a developer at a startup that is building an application that performs the given generation task. Their goal was to design a prompt that performed better than an initial prompt that their team had designed and demonstrate its performance on diverse data samples. Participants could flexibly decide on the criteria that they would be improving and evaluating, but were asked to ensure that their final criteria set was: (1) exhaustively comprehensive (i.e., assess all factors that are important for the task), (2) mutually exclusive (i.e., minimal redundancies between criteria), and (3) clear (i.e., clearly described for others to understand what is assessed). After this explanation, participants performed the first task for 35 minutes. After the task, they responded to a post-task survey and, through a semi-structured interview, we asked them about how they defined their criteria, evaluated outputs, and revised their prompts during this task. Then, participants were provided with a walkthrough of the second interface and used this interface to perform the second task for 35 minutes. After the task, participants responded to the post-task survey and we concluded the study with a semi-structured interview about participants’ experiences in the second task, and the differences between their experiences in the first and second task.

1.5. Measures

For qualitative data, we transcribed the semi-structured interviews and coded them through a thematic analysis. For quantitative data, we analyzed participants’ responses to the two post-task surveys. These surveys asked participants to rate, on a seven-point Likert scale, their self-confidence in designing prompts and evaluating outputs, and their self-perceived experience interacting with the system (e.g., how successful and collaborative it was) based on questions from Wu et al. (Wu et al., 2022). We also asked participants to self-rate their final criteria on how exhaustively comprehensive, mutual exclusive, and clear they were. Finally, participants rated their self-perceived workload on five questions from the NASA-TLX questionnaire, excluding the “Physical Demand” question. We include the detailed survey questions in Appendix D. For these Likert scale ratings, we analyzed them through the non-parametric Wilcoxon signed-rank test. Additionally, we analyzed participants’ interaction logs to measure the number of (1) unique prompts tested, (2) criteria changes (e.g., edit, add, delete), and (3) unique outputs evaluated, where partial evaluations (i.e., only a subset of criteria were evaluated) were counted equally. We also analyzed how participants created and revised their final set of criteria. For all these measures, we conducted a Shapiro-Wilk test to determine if the data was parametric or non-parametric, and then used a paired t-test (if parametric) and a Wilcoxon signed-rank test (if non-parametric).

2. Results

Our results demonstrated that EvalLM allowed participants to “debug” their prompts by quickly checking the effect of prompt changes on multiple criteria and samples. Beyond checking the effect of revisions, we observed that EvalLM provided “feedback” to participants, which helped them ideate on how to revise their prompts. Regarding criteria, EvalLM encouraged participants to revise their criteria more, which in turn allowed them to consider diverse values and perspectives around the given tasks. In this section, we describe findings on how participants evaluated outputs and revised their prompts in §7.2.1 and §7.2.1 (RQ1), how they defined their criteria in §7.2.1 (RQ2), their trust for the evaluation assistant in §7.2.1 (RQ3), and their overall perceived workload in §7.2.1. For each of these, we first describe relevant quantitative findings and then qualitative insights.

Overall, participants had higher self-confidence in their ability to evaluate prompt outputs with the Assist condition (Assist=6.71 ± 0.40, Manual=4.96 ± 0.72, z=9.23, p¡0.001). Participants revealed that this was due to how the Assist condition shouldered the burden of examining outputs, which enabled them to evaluate prompts more comprehensively. Specifically, with Assist, participants were able to evaluate a larger number of unique outputs (Assist=20.42 ± 14.46, Manual=10.08 ± 6.27, z=8.00, p=0.03). Several participants noted how this helped them look at the “bigger picture” (P1) and understand “how [a prompt] will work at a larger scale” (P5).

By facilitating evaluations, the Assist condition also allowed participants to evaluate more criteria and assess their prompts’ performance on diverse dimensions, without worrying about the cost involved. For example, P9 mentioned that they were encouraged to “think of more aspects that they wanted to evaluate” and P2 mentioned that they “just kept criteria […] to check them just in case.” In contrast, due to the effort involved, participants would frequently only evaluate samples on a subset of their criteria when using Manual—on average, 46.7% (SD=31.2%) of all evaluated samples were partially evaluated. Besides evaluating outputs for them, several participants also mentioned how the Assist condition made it easier for them to evaluate the outputs themselves. For example, both P4 and P6 mentioned how it could be challenging to “tell what is different by simply looking at [outputs]” (P5) and “to see why one output might be better or not” (P4). However, as the evaluation assistant explained its evaluations and highlighted sections of the outputs, it was “easier to tell outputs apart” (P6) and actually assess these outputs.

As the Assist condition supported faster evaluation of prompts on various samples and criteria, participants mentioned how they used the evaluation assistant as a “debugger” (P8). The evaluation assistant helped participants to check “what the prompt is already satisfying” (P8) and “where it’s lacking” (P9). This in turn allowed participants to decide on what aspects need the most improvement and focus their iterations on them. For example, P11 mentioned, “The [system] tells me that my prompt is worse on simplicity so that means that my prompt is not communicating this clearly and I should focus on fixing this.” Then, participants mentioned how they could check that these revisions had the desired effect by evaluating the outputs generated from the revised prompts.

2.2. Revising Prompts

Besides aiding participants in checking revisions, the Assist condition also helped participants in thinking about how to revise their prompts. Participants felt that the Assist condition helped them think about how to complete the task better (Assist=6.83 ± 0.39, Manual=5.67 ± 1.15, z=0.00, p=0.01) and that they were collaborating with the system to design the prompts (Assist=6.58 ± 0.51, Manual=4.58 ± 1.98, z=1.50, p¡0.01) (Fig. 8, left). Additionally, we found that participants with the Assist condition reached a similar level of satisfaction in their prompts (Assist=5.67 ± 1.61, Manual=5.17 ± 1.47, z=20.00, p=0.24) (“Match Goal” in Fig. 8) by testing a smaller number of prompt variations (Assist=5.00 ± 3.13, Manual=8.42 ± 4.76, z=1.50, p=0.01). While participants also reported a higher self-confidence on their ability to design and improve prompts with Assist, this difference was not significant (Assist=5.63 ± 1.03, Manual=5.00 ± 0.95, z=11.00, p=0.17).

In the Manual condition, participants mentioned how they felt like they were “the only brain thinking about [the task]” (P10). In contrast, as the Assist condition explains its evaluations, participants felt that the system was providing them with “feedback on what to improve” (P2) and could help them set the “direction” for their prompts (P7, P9). Several participants found the explanations to be particularly useful as they presented them with “diverse opinions” (P1) that they “couldn’t think about” (P6). For example, P5 purposefully increased the number of evaluation trials and checked the explanation for each trial to “see all the differences [in opinion] and then change my prompt to satisfy all of these.” In this sense, the Assist condition could allow participants to gain a more holistic understanding of how to improve their prompts. In turn, this understanding could help participants to make more effective prompt revisions—as reflected by the finding that participants tested less prompt changes in the Assist condition.

These explanations, however, could also lead to participants feeling less in control and more aware of their prompt’s weaknesses. Out of all participants, only P2 mentioned preferring the Manual condition. She described how she incorporated her “own diverse ideas” into the prompt in Manual, but, in Assist, she “kept worrying about the evaluations” as she focused on incorporating feedback but, at times, not seeing improvements in the evaluations. Even participants that preferred the Assist condition mentioned how the explanations could occasionally be limited as evaluations would frequently result in ties or provide high ratings to both outputs and they “wouldn’t know what to improve.” Since the feedback focused on the outputs rather than the prompts, several participants also noted how they would have preferred if the Assist condition would “automatically suggest ways to improve prompts.” As an alternative solution, P3 and P10 mentioned how they would add “keywords” from the explanations into their prompts.

2.3. Defining Criteria

Participants had a similar number of final criteria at the end of the tasks (Assist=4.25 ± 1.29, Manual=4.00 ± 1.28, t=10.50, p=0.55). In terms of their evaluation criteria, participants felt that their criteria in the Assist condition were more comprehensive (Assist=6.00 ± 1.48, Manual=4.75 ± 1.48, t=2.07, p=0.06), although with marginal significance, and clearer (Assist= 6.42 ± 0.67, Manual=4.92 ± 1.44, t=3.59, p¡0.01) than in the Manual condition (Fig. 8, right). Additionally, participants made more changes to their criteria with the Assist condition (Assist=22.67 ± 9.17, Manual=13.33 ± 8.64, t=2.29, p=0.04). For participants’ final criteria, there was no significant difference in the proportion of criteria created from scratch (Assist=30.1% ± 34.5%, Manual=16.67% ± 18.8%, w=15.0, p=0.37) or through the criteria dictionary (Assist= 77.78% ± 20.5%, Manual=69.93% ± 34.50%, w=22.0, p=0.95). However, in the Assist condition, a significantly higher proportion of the dictionary-based criteria had been edited (Assist=75.8% ± 24.1%, Manual=17.4% ± 34.5%, w=4.000, p¡0.01)—revealing participants’ intent to adjust these criteria to the task. Figure 9 shows how participants reached their final criteria in each condition. All the criteria created during the study are included in the Supplementary Materials.

On average, participants took at least one suggested improvement from 31.3% (SD=23.5%) of the automatic criteria reviews, and 78.6% (SD=20.4%) of each participant’s final criteria were revised through suggestions. Similar to how the evaluation assistant helped participants think of how to improve their prompts, the criteria review tool helped participants think of what to evaluate when they were “too busy working on the prompt” (P4). Participants mentioned how the tool “suggested [criteria] that [they] couldn’t think about” (P11) and helped them when they “didn’t really know what [they] need” (P5). For example, the review tool suggested splitting P11’s “Engagingness” criteria into “Simplicity” and “Creativity” which helped them realize the dependencies but also differences between these aspects (Tab. 2). Furthermore, participants used the review tool to refine the “questionable” (P1) criteria that they composed in order to match the quality of the criteria defined by prior research. For example, P1 used the suggestions to further refine and elaborate on their “Explainability” criteria (Tab. 2),. Through these system-supported revisions, participants felt that they had increased the overall quality of the criteria by making them more “concrete” (P1), “easy to understand” (P8), “sharper”, and “consistent” (P3). Participants noted how these improved sets “could be useful when talking with others” (P1) or even when manually evaluating outputs (P8).

2.4. Trust

While a few participants expressed that they might “rely too much on the system” (P3), we observed that participants held a healthy level of skepticism about the evaluation assistant—the mean rating for trust was 4.91 (SD=1.51) out of 7. Participants mentioned how they would check the evaluation explanations and highlights to verify that the evaluations were “reasonable” (P11). Adding on this, P9 mentioned, “I didn’t consider the system to be an automatic evaluator since I was still checking its evaluation results”—illustrating a human-AI collaborative workflow where the system evaluates and user verifies. Some participants also mentioned how their trust was moderated by verifying evaluations as the explanations could “give a reason that [they] did not consider to be important” (P12) or were “not aligned” (P4) with their thoughts.

Despite being skeptical of the evaluation assistant, multiple participants mentioned that they might trust it more than their own manual evaluations. Participants expressed that they were “not completely confident” (P2) in their evaluations as they “couldn’t look at the whole text” (P6) and mostly evaluated “based on feeling” (P12). Additionally, participants felt that they be biased towards their own prompt as they were “the person creating the prompt and also evaluating it” (P11) and considered that the automatic evaluations could be more correct as the “AI might have more background knowledge than [they] do” (P10). Further, P5 and P8 mentioned how they attempted to “not compromise or bias” (P8) the evaluations by using different phrasing in their prompts and criteria. Although these findings portray the usefulness of the automatic evaluations, they also suggest that there is a potential danger in designers becoming over reliant on these evaluations.

2.5. Perceived Workload

Overall, participants felt significant lower mental burden (Assist=3.92 ± 1.78, Manual=5.58 ± 1.31, z=2.50, p=0.01) and significant lower effort (Assist=3.50 ± 1.68, Manual=5.25 ± 1.48, z=5.50, p=0.04) in the Assist condition (Fig. 10). The overall subjective mental workload was also lower in the Assist condition, although with marginal significance (Assist=3.45 ± 1.19, Manual=4.48 ± 0.99, z=13.00, pp=0.08). As described in the findings above, this can be attributed to how the Assist condition facilitated every step of the evaluation-refinement process: ideating on how to revise prompts, verifying the effect of revisions on performance, thinking of what to evaluate, and describing criteria.

Discussion

In this paper, we present EvalLM, an interactive system that supports designers in testing and comparing prompts by evaluating them on user-defined criteria with the aid of an LLM-based evaluator. We believe that EvalLM can narrow the gap between the development and deployment of LLM-based applications by helping designers iterate on their prompts when they cannot recruit external evaluators or testers to provide feedback. While EvalLM focuses on facilitating prompt evaluation, we believe that it can be expanded to facilitate prompt refinement, and enhance the evaluation and refinement of LLMs’ themselves. In this section, we discuss the further potential of EvalLM and opportunities for future work.

In our study, EvalLM helped participants to evaluate their prompts in greater breadth (i.e., samples) and depth (i.e., criteria). Through this, participants were able to identify the limitations of their prompts and prioritized these in their subsequent revisions. Besides supporting participants in identifying needed revisions, the explanations from the evaluation assistant acted as feedback, which advised participants on how to make these revisions. According to participants, these explanations simulated diverse opinions and allowed them to break away from their own biases—suggesting that the evaluations could simulate potential user feedback (Argyle et al., 2023; Park et al., 2022). Overall, the study indicated that EvalLM allows designers to collaborate with an LLM to efficiently iterate on and verify the progress of their application, without requiring a significant commitment of resources to recruit human evaluators or deploy their application to testers.

While our work aims to support designers in reaching these final stages of development, EvalLM is not intended to replace human evaluation or tests. As revealed by various studies, current LLMs are only able to represent a limited set of human perspectives (Kleinberg and Raghavan, 2021; Bommasani et al., 2021) and exhibit higher homogeneity of opinions compared to humans (Argyle et al., 2023; Santurkar et al., 2023). Thus, LLM-based evaluations cannot fully represent the opinions of users or predict how the application will actually be used—meaning that solely relying on these evaluations can leave designers open to potential issues in the future. However, we posit that EvalLM can still help designers prepare for these final human evaluations or tests. As EvalLM helps designers to iterate on their criteria with the review tool and allows them to test them through the evaluation assistant, the criteria sets created through the system could be useful when instructing human evaluators.

2. Refining on User-Defined Criteria

While EvalLM can support the ideation of prompt revisions, the designer is still responsible for implementing these revisions. In fact, several study participants mentioned how they desired for the system to automatically revise their prompts based on its evaluations. Inspired by the success of reinforcement learning from human feedback (RLHF) in guiding models to produce higher-quality outputs (Ouyang et al., 2022; Fernandes et al., 2023), various approaches have investigated how to use LLMs themselves to provide feedback to other LLMs—i.e., reinforcement learning from AI feedback (RLAIF) (Bai et al., 2022; Madaan et al., 2023; Akyürek et al., 2023; Dubois et al., 2023; Sun et al., 2023; Kim et al., 2023a). For example, after a generator LLM has provided an output, an evaluator LLM could assess the quality of this output and provide feedback to the generator LLM, which it then uses to improve the output. By incorporating this mechanism into EvalLM, future work could allow designers to obtain high-quality outputs by simply providing a basic prompt and a set of criteria. The system would use these to automatically generate, evaluate, and revise outputs that satisfy the criteria—without the designer needing to “herd” the LLM themselves (Zamfirescu-Pereira et al., 2023a).

3. Beyond Prompts to Models

EvalLM is also capable of evaluating and comparing models. Instead of only uploading input samples, the system allows developers to upload accompanying output pairs. With the proliferation of high-performing but smaller-scale LLMs (e.g., LLaMA (Touvron et al., 2023a, b)) and the introduction of parameter-efficient fine-tuning (PEFT) methods (Mangrulkar et al., 2022; Hu et al., 2021; Zhang et al., 2023), developers and researchers have started to fine-tune their own LLMs to overcome the restrictions of prompting (Diao et al., 2023; Cui et al., 2023; Wang et al., 2023b). As the efficiency of LLM training increases and its cost decreases, we may see the proliferation of LLMs for a wider array of use cases and applications. In this environment, practitioners could use EvalLM as a first check on the performance of their trained models or, more promisingly, as a validation stage while the model is training to verify progress and identify weaknesses. Furthermore, as suggested in the previous subsection, practitioners can design an effective set of evaluation criteria through EvalLM and then employ this to align the models with these criteria through RLAIF. Thus, beyond prompt design, we believe that our work can also support practitioners in the development of more context- and task-specific models.

4. Evaluation Landscape for Natural Language Generation

Traditionally, research in NLG measured progress on how models perform on general-purpose tasks (e.g., summarization (Nallapati et al., 2016), topical conversations (Gopalakrishnan et al., 2023)), where performance was measured through more general criteria (e.g., “coherency”, “relevance” (Zhong et al., 2022; Fabbri et al., 2021)). As models become more capable of performing specific and long-tail tasks, on the other hand, developers and researchers may evaluate models on more task-specific criteria. While this diversification of criteria could lead to a more comprehensive understanding of LLM performance (Gehrmann et al., 2023), it can also become more challenging to synthesize the results from different evaluations and compare model performance. However, as shown by our formative interviews and user study, most of these task-specific criteria are frequently subordinate to more general criteria—meaning that results on specific criteria can present insights about performance on general criteria. Future work could collect and organize criteria into hierarchies that can represent model performance at both fine-grained and coarse-grained levels to enable practitioners to make more informed model choices.

Conclusion

This paper presents EvalLM, a novel interactive system that supports prompt designers in refining LLM prompts by evaluating generated outputs on user-defined criteria. In the interface, the user composes pairs of prompts and a set of evaluation criteria, and then employs LLMs to both generate outputs with the prompts and evaluate the outputs on the criteria. Through the explanations provided by the LLM on its evaluation, the user can iteratively refine their prompts, by identifying where they lack, and their criteria, by identifying where they are unclear. In a comparative user study (N=12), we compared EvalLM to a condition where participants manually evaluated outputs. We observed that EvalLM allowed participants to easily verify the effects of their prompt revisions, identify possible directions for improvement, and to define criteria that were more adapted to specific contexts.

References

Appendix A EvalLM: Experiment Screen

Figure 11 shows the Experiment screen in EvalLM, which allows users to test and evaluate their prompts on a larger nubmer of samples and test the reliability of the evaluations.

Appendix B Prompts

Below, the full prompts used in EvalLM, where blue text represents content that is programmatically filled in.

System Prompt You are a helpful and precise assistant that can check the quality of responses by other AI assistants for a given user instruction. You can objectively evaluate the holistic quality of responses by assessing how well the responses satisfy a set of quality criteria. You should provide comprehensive feedback on the responses according to each of these criteria and provide detailed justification for your feedback. If you refer to specific fragments of the responses in your feedback, you should also return these fragments as evidence. You should return your final answer as a valid JSON object.

User Prompt We would like to request your feedback on the performance of two AI assistants responding to the user instruction displayed below. Each assistant performed the instruction on the same input displayed below. In the feedback, please rate the quality of the responses on the following criteria. [The Start of Criteria] Each criteria in the form "Name: Description", separated by a new line [The End of Criteria] [The Start of Instructions] Instructions [The End of Instructions] [The Start of Input] Input [The End of Input] [The Start of Assistant 1’s Response] Output from the first prompt [The End of Assistant 1’s Response] [The Start of Assistant 2’s Response] Output from the second prompt [The End of Assistant 2’s Response] [System] Please give feedback on the responses for each criteria. First, provide a comprehensive explanation comparing the two assistants in their ability to satisfy the criterion. You should justify your judgement by providing substantial detail about your reasoning. Ensure that you only write comments about one criterion at a time. Avoid giving feedback on other aspects of the responses that are not described in the criteria. Then, for each assistant, list a maximum of five words or short phrases from their response that illustrate what you described in your explanation. Avoid listing whole sentences or long phrases as evidence. If the whole response is needed as evidence, add the token "$WHOLE$" to the list. Finally, write your scores for each assistant on the criterion. The score should be on a scale of 1 to 10, where a higher score indicates that the assistant’s response was better at satisfying the criterion. Avoid any potential bias and ensure that the order in which the responses were presented does not affect your judgement. Lastly, return a JSON object of the following format: {"": {"explanation": , "assistant_1": {"evidence": [], "score": }, "assistant_2": {}}, ...}

B.2. Criteria Review: Refine

System Prompt You are a helpful and precise assistant that can review the quality of scoring criteria that are used to measure the quality of responses. You can identify whether criteria are vague or confusing. You can also revise the criteria to improve their quality. You return your final answer as a valid JSON object.

User Prompt We would like to request you to examine a set of criteria that AI assistants should satisfy when responding to the user instruction below. Human judges will refer to these criteria to rate the assistants’ responses on how well they satisfy each criteria. [The Start of Instructions] Instructions [The End of Instructions] [The Start of Criteria] Each criteria in the form "Name: Description", separated by a new line [The End of Criteria] Please review the provided list of criteria carefully. Identify criteria that are vague, meaning that they describe general characteristics that are not specifically relevant to the user instruction. Also, identify criteria that have unclear or confusing descriptions. First, provide a comprehensive explanation about how certain criteria are vague, unclear, or both. Then, paraphrase the criteria names and descriptions so that they are more specific to the instruction and their descriptions are clearer. Ensure that these revised criteria have names that are concise and descriptions that are clear so that judges can precisely understand their meaning. You should only rephrase criteria or add more details. Avoid removing details from the criteria. Avoid replacing any criteria or creating new criteria. Finally, ONLY return the revised criteria as a JSON object: {"results": [{"name": , "description": , "original_criteria": }, ...]}. Avoid including the criteria that were not revised in this object. You may be unable to identify any unclear or imprecise criteria. If so, simply return an empty list: {"results": []}."

B.3. Criteria Review: Merge

System Prompt You are a helpful and precise assistant that can review the quality of scoring criteria that are used to measure the quality of responses. You can identify whether criteria are redundant or if they have overlapping areas. You can also revise the criteria to improve their quality. You return your final answer as a valid JSON object.

User Prompt We would like to request you to examine a set of criteria that AI assistants should satisfy when responding to the user instruction below. Human judges will refer to these criteria to rate the assistants’ responses on how well they satisfy each criteria. [The Start of Instructions] Instructions [The End of Instructions] [The Start of Criteria] Each criteria in the form "Name: Description", separated by a new line [The End of Criteria] Please review the provided list of criteria carefully. Identify criteria that are not mutually exclusive, meaning that the criteria have areas of overlap between them. Focus on identifying criteria that have portions that are redundant with portions of other criteria as they measure the same feature of assistants’ responses. For the criteria pairs or groups that may overlap, provide a comprehensive explanation about what parts of the criteria are redundant. Then, combine only these overlapping portions into a new criteria. Ensure that these revised criteria have names that are concise and descriptions that are clear so that judges can precisely understand their meaning. You should only merge the redundant portions and avoid creating new criteria that are excessively broad. Finally, ONLY return the new criteria as a JSON object: {"results": [{"name": , "description": , "original_criteria": []}, ...]}. Avoid including the criteria that were not overlapping in this object. You may be unable to identify any overlapping criteria. If so, simply return an empty list: {"results": []}."

B.4. Criteria Review: Decompose

System Prompt You are a helpful and precise assistant that can review the quality of scoring criteria that are used to measure the quality of responses. You can identify whether criteria are excessively broad or consider multiple unrelated aspects. You can also revise the criteria to improve their quality. You return your final answer as a valid JSON object.

User Prompt We would like to request you to examine a set of criteria that AI assistants should satisfy when responding to the user instruction below. Human judges will refer to these criteria to rate the assistants’ responses on how well they satisfy each criteria. [The Start of Instructions] Instructions [The End of Instructions] [The Start of Criteria] Each criteria in the form "Name: Description", separated by a new line [The End of Criteria] Please review the provided list of criteria carefully. Identify criteria that are excessively broad. You should identify criteria that consider multiple, distinct aspects in the assistants’ responses. Focus on identifying criteria that measure dimensions that are independent and possibly unrelated. For the identified criteria, provide a comprehensive explanation about how these criteria may be excessively broad. Then, divide each identified criterion into a new set of criteria that are specific and mutually exclusive, meaning that they do not overlap. Ensure that these revised criteria have names that are concise and descriptions that are clear so that judges can precisely understand their meaning. Finally, ONLY return the new criteria as a JSON object: {"results": [{"name": , "description": , "original_criteria": }, ...]}. Avoid including the criteria that were not excessively broad in this object. You may be unable to identify any broad criteria. If so, simply return an empty list: {"results": []}."

Appendix C Technical Evaluation: Details

Table 3 lists the original questions taken from the MT Bench dataset for our technical evaluation, and how these were split into a general prompt and a specific input for our evaluation.

Appendix D User Study: Survey Questions

For the post-task surveys in the user study, participants were asked to rate their agreement with the following statements on a seven-point Likert scale (1=Strongly Disagree, 7=Strongly Agree).

Self-Confidence in Designing Prompts: “I know how to design prompts to instruct an LLM to perform a desired task.”

Self-Confidence in Improving Prompts: “I can improve prompts to guide an LLM to produce better outputs for a given task.”

Self-Confidence in Planning Evaluations: “I know what criteria an LLM should satisfy when performing a given task.”

Self-Confidence in Performing Evaluations: “I can measure the performance of a prompt for a given task by examining the outputs it generates.”

Match Goal: “I’m satisfied with the results from my final prompts, they met the task goal.”

Think-Through: “The system helped me think what kinds of outputs I would want to complete the task goal, and how to complete the task.”

Collaborative: “I felt I was collaborating with the system to come up with the outputs.”

Comprehensive Criteria: “My defined criteria, together, consider all of the aspects that are important for measuring the quality of outputs for the given task.”

Mutually Exclusive Criteria: “My final criteria have none or minimal overlaps, and each criterion measures different aspects of quality in the outputs for the task.”

Clear Criteria: “My defined criteria are clearly described without causing possible confusion or misunderstandings.”

The questions for self-confidence in designing and improving prompts were combined into one measure for self-confidence in prompting, and the questions for self-confidence in planning and perfoming evaluations were combined into one measure for self-confidence in evaluating. The question for match goal, think-through, and collaborative were adapted from Wu et al. (Wu et al., 2022).