Source-linked AI summary
EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteria
Tae Soo Kim, Yoonjoo Lee, Jamin Shin, Young-Ho Kim, Juho Kim
TL;DR
Prompt designers need to refine LLM prompts by evaluating outputs against subjective, application-specific criteria, but manual evaluation is effortful and incomplete. EvalLM uses LLMs to evaluate outputs and review criteria interactively, and a comparative study found that it reduced revisions needed to reach satisfactory prompts. Its evaluations cannot fully represent users’ perspectives, so human evaluation remains relevant.
Problem
Manual evaluation of subjective, application-specific criteria makes prompt refinement effortful and prevents designers from fully verifying revisions.
Method
EvalLM interactively evaluates multiple prompt outputs on user-defined criteria while an LLM reviews criteria and explains evaluation results.
Results
59% fewer changes were tested before participants reached satisfactory prompts with EvalLM than with manual evaluation.
Takeaways & Limitations
EvalLM can help designers iterate on prompts before committing resources to more robust human evaluations.
Takeaways & Limitations
LLM-based evaluations cannot fully represent users’ opinions or predict how applications will actually be used.
Abstract
from arXiv · showhide
By simply composing prompts, developers can prototype novel generative applications with Large Language Models (LLMs). To refine prototypes into products, however, developers must iteratively revise prompts by evaluating outputs to diagnose weaknesses. Formative interviews (N=8) revealed that developers invest significant effort in manually evaluating outputs as they assess context-specific and subjective criteria. We present EvalLM, an interactive system for iteratively refining prompts by evaluating multiple outputs on user-defined criteria. By describing criteria in natural language, users can employ the system's LLM-based evaluator to get an overview of where prompts excel or fail, and improve these based on the evaluator's feedback. A comparative study (N=12) showed that EvalLM, when compared to manual evaluation, helped participants compose more diverse criteria, examine twice as many outputs, and reach satisfactory prompts with 59% fewer revisions. Beyond prompts, our work can be extended to augment model evaluation and alignment in specific application contexts.
1 INTRODUCTION
Prompt designers must repeatedly evaluate and revise LLM prompts, but subjective, application-specific criteria make manual evaluation costly and incomplete. EvalLM supports this process with LLM-based evaluation and criteria refinement, helping designers reach satisfactory prompts with fewer revisions.
- EvalLM evaluates outputs on user-defined, application-specific criteria using an LLM-based evaluation assistant.
- EvalLM provides explanations that help designers identify prompt shortcomings and possible criteria misalignment.
- 59% fewer changes were tested before participants reached satisfactory prompts with EvalLM than without evaluation assistance.
- Manual evaluation of multiple, task-specific criteria imposes cognitive load and can inhibit informed prompt-iteration decisions.
- A criteria reviewer refines users’ criteria toward more specific and detailed output dimensions.
- The comparative user study included 12 participants and found that EvalLM aided prompt debugging and revision strategy ideation.
2 RELATED WORK
Related work frames prompt design as an open-ended search problem and natural-language generation as difficult to evaluate because many outputs can be valid. Human evaluation remains costly, motivating interactive and automated evaluation support.
- Prompt design challenges: Prompt performance can change substantially with instruction format, phrasing, content, examples, or example order.
- Prompt design challenges: Because the space of natural-language instructions is near infinite, designers must test many possibilities to identify high-performing prompts.
- Natural language generation and evaluation: Natural language generation seeks text that satisfies a communicative goal while possessing qualities such as fluency and coherence.
- Natural language generation and evaluation: Open-ended generation is difficult to evaluate because multiple dissimilar outputs can be equally valid and comprehensive reference sets are impractical.
- Natural language generation and evaluation: Human evaluation is a standard for generation tasks, but recruiting annotators can be prohibitive during early development.
- Interactive evaluation: Interactive systems such as Zeno, the What-If Tool, and Errudite support practitioners in evaluating model outputs and behaviors.
3 FORMATIVE INTERVIEWS
Formative interviews show that prompt designers iteratively evaluate novel tasks using subjective, multi-faceted criteria, while continually discovering and concretizing what success means. This manual process makes revision and verification difficult.
- 3 FORMATIVE INTERVIEWS: The interviews examined how prompt designers evaluate performance and use evaluations to refine prompts during early development.
- 3.1 Participants and Procedure: The eight interviewees included HCI and ML/NLP students, research scientists, a data scientist, and a startup CEO.
- 3.1 Participants and Procedure: Interviews were semi-structured, recorded, manually transcribed, and coded through thematic analysis.
- 3.2 Findings: All designers developed novel generation tasks and iteratively evaluated and refined prompts using sample inputs.
- 3.2.1 Evaluation is Manual.: Designers manually evaluated outputs because their criteria were subjective and task-specific, with no existing automated metrics and prohibitively expensive annotator recruitment.
- 3.2.1 Evaluation is Manual.: Multiple simultaneous criteria created substantial cognitive load as designers examined outputs for subtle qualities.
- 3.2.2 Evaluation is Multi-Faceted.: Designers expanded criteria after observing unexpected flaws or newly desired qualities in generated outputs.
- 3.2.3 Evaluation is Dynamic.: Designers struggled to define success for subjective criteria and were unsure how to revise prompts after identifying failures.
4 DESIGN GOALS
EvalLM’s design goals center on efficient evaluation against user-defined criteria and support for inspecting, developing, and revising those criteria. These goals connect automated assessment with designers’ own expectations.
- The system should automate evaluation of generated outputs according to user-defined criteria.
- Explanations should help designers inspect automatic evaluations and identify incorrect or unclear assessments.
- Criteria should be definable from both output data and prior literature.
- The system should review user-defined criteria and identify potential revisions that may improve subsequent evaluations.
5 EVALLM
EvalLM supports iterative prompt refinement by comparing outputs from prompt pairs against user-defined criteria, while an LLM evaluates outputs and reviews criteria. Its interface also provides explanations, uncertainty handling, history, and repeated trials to help users inspect and improve evaluations.
- System overview: EvalLM turns prompt refinement into an interactive process where users compare prompt outputs using multiple user-defined criteria.The system uses an LLM as both an evaluation assistant and criteria reviewer.
- Composing prompts: Users compose two prompt templates, sample inputs, and task instructions before generating side-by-side outputs for comparison.Prompt templates can be revised or compared across iterations using sampled dataset inputs.
- Defining and revising criteria: Users define criteria by name and description, optionally adapt criteria from a dictionary, and receive suggestions to clarify, merge, or split them.The criteria review tool checks clarity and relevance, redundancy, and granularity.
- Automatic evaluation: The evaluation assistant scores each output pair on every criterion, identifies winners or ties, explains decisions, and highlights relevant output fragments.The interface also summarizes the proportion of samples won by each prompt and the proportion of ties.
- Automatic evaluation: Multiple evaluation trials reduce uncertainty by selecting criterion winners through majority vote across runs.Users can inspect individual trials when results differ across evaluations.
6 TECHNICAL EVALUATION
The technical evaluation compares LLM judging under overall-quality, general-criteria, and specific-criteria conditions against human evaluations. Specific criteria achieved the strongest agreement, while explanation analysis found high but imperfect reliability across several dimensions.
- Evaluation design: The study evaluates whether task-specific criteria affect LLM evaluation performance relative to human judgments.The evaluation addresses a gap left by prior work focused on overall quality or predefined criteria.
- Evaluation design: The comparison uses Overall-Quality, General-Criteria, and Specific-Criteria conditions on paired responses from selected MT-Bench requests.Specific-Criteria automatically refines and splits general criteria for each request or prompt.
- Results: Specific-Criteria produced the highest agreement and correlation with human annotators, with Fleiss’ kappa similar to sampled human-human agreement.The authors also observed more balanced evaluations in qualitative review.
- Limitations: Agreement between LLM and human evaluations remained imperfect, and equal weighting across criteria may not reflect the precedence of safety-critical criteria.The paper cautions against relying only on LLM evaluations.
- Explanation quality: Explanations were logical for 91.4% of cases, faithful for 99.1%, independent for 84.2%, evidential for 100%, and score-aligned for 98.6%.The analysis covered 194 criterion-wise evaluations from 38 output pairs.
- Limitations: The explanations still showed limitations involving independence, creativity, output-length assessment, and occasional mismatch between explanations and scores.The authors characterize GPT-4 explanations as relatively sensible but limited by detail bias and logical capabilities.
7 USER STUDY
The within-subjects study compared EvalLM with manual evaluation, examining prompt revision, criteria development, trust, and workload. EvalLM broadened evaluation and supported faster, more informed revisions, while raising concerns about control, feedback limitations, and overreliance.
- The within-subjects study compared EvalLM with manual output evaluation and examined prompt revision, criteria definition, and trust.
- Prompt evaluation: Assist participants evaluated 20.42 unique outputs on average versus 10.08 manually and reported higher evaluation confidence, 6.71 versus 4.96.The confidence difference was significant (p<0.001), while the output-count difference was also significant (p=0.03).
- Prompt evaluation: Assist participants evaluated more criteria and used evaluation explanations to identify prompt weaknesses, verify revisions, and guide subsequent changes.Participants described the assistant as a debugger that highlighted satisfied and deficient aspects of prompts.
- Prompt revision: Assist participants reported higher ratings for completing tasks better, 6.83 versus 5.67, and collaborating with the system, 6.58 versus 4.58.The explanations supplied feedback, direction, and diverse opinions for prompt improvement.
- Criteria development: Assist criteria were rated clearer, 6.42 versus 4.92, and underwent more changes, 22.67 versus 13.33, while criteria counts were similar.Comprehensiveness was also higher numerically in Assist, 6.00 versus 4.75, but only marginally significant (p=0.06).
- Criteria development: Participants adopted at least one suggested improvement from 31.3% of automatic criteria reviews, and 78.6% of final criteria were revised through suggestions.The criteria review tool helped participants identify evaluation dimensions they had not previously considered.
- Trust and workload: Assist reduced reported mental burden to 3.92 versus 5.58 and effort to 3.50 versus 5.25, but explanations could reduce control and encourage overreliance.Participants also reported ties or equally high ratings that did not indicate what to improve, and mean trust was 4.91 out of 7.
8 DISCUSSION
The discussion positions EvalLM as support for iterative prompt development and possible model evaluation, while emphasizing that LLM evaluations cannot replace human perspectives or testing. It also identifies automatic prompt revision and broader task-specific evaluation as future directions.
- EvalLM helps designers iterate when external evaluators or testers are unavailable, narrowing the development-to-deployment gap without replacing human evaluation.
- Evaluation breadth and depth helped participants identify prompt limitations, prioritize revisions, and use explanations as feedback for implementing improvements.
- Limitations: LLM evaluations cannot fully represent users’ opinions or predict actual application use because current LLMs represent limited, more homogeneous perspectives.The authors therefore frame EvalLM as preparation for final human evaluations or tests.
- Future work: EvalLM supports ideating prompt revisions, but designers remain responsible for implementing them; participants also requested automatic prompt revision.
- Beyond prompts: EvalLM can evaluate and compare models, potentially supporting context- and task-specific criteria during model training, validation, and human evaluation.
9 CONCLUSION
EvalLM supports prompt refinement by using LLMs to generate and evaluate outputs against user-defined criteria. In a comparative study, it helped participants verify revisions, find improvement directions, and define context-adapted criteria.
- EvalLM evaluates generated outputs against user-defined criteria while users iteratively refine prompts and criteria using evaluation explanations.
- In a comparative user study with 12 participants, EvalLM supported revision verification, improvement-direction discovery, and more context-adapted criteria than manual evaluation.
A EVALLM: EXPERIMENT SCREEN
The Experiment screen lets users configure and revisit prompt evaluations across multiple samples, trials, criteria, and evaluator settings. It also surfaces evaluator reliability and alternative-evaluator comparisons.
- The Experiment screen lets users test prompts on a larger number of samples and assess evaluation reliability.
- Interface layout: The left panel contains instructions, prompts, and criteria, while the right panel configures sample and trial counts and an optional alternative evaluator.
- Reliability and comparison: The screen displays test-retest reliability, inter-rater reliability, and a red dot marking the alternative evaluator’s selected winner.
- Experiment management: Users can run experiments with different prompts, criteria, and settings and revisit them through the experiment log.
B PROMPTS
The appendix specifies EvalLM’s automatic evaluation and criteria-review prompts, which compare outputs and refine user-defined criteria in structured JSON.
- B PROMPTS: The appendix notes that EvalLM prompts are approximately 300–500 tokens and participant-created criteria average approximately 30 tokens.These lengths are discussed alongside the LLM context-length constraint.
- B.1 Automatic Evaluation: The automatic evaluator compares two assistants’ responses against user-defined criteria and returns explanations, evidence fragments, and criterion scores.Its prompt requires comprehensive, criterion-specific feedback and valid JSON output.
- B.1 Automatic Evaluation: The evaluator receives a shared instruction, input, and outputs from two alternative prompts before producing criterion-level comparisons.The prompt separates criteria, instructions, input, and each assistant’s response into labeled sections.
- B.1 Automatic Evaluation: The automatic-evaluation prompt instructs the evaluator to justify judgments with detailed reasoning while restricting comments to the criterion under review.Evidence is limited to at most five words or short phrases from each response.
- B.2 Criteria Review: Refine: Criteria refinement identifies vague or unclear criteria and rewrites only those criteria with more specific names and descriptions.The output preserves each revised criterion’s original name and can be empty when no revision is needed.
- B.2 Criteria Review: Refine: The criteria-review prompts require valid JSON outputs containing only revised, merged, or decomposed criteria rather than unchanged criteria.Each review operation specifies a distinct result structure for the returned criteria.
- B.3 Criteria Review: Merge: Criteria merging identifies overlapping criteria and combines only their redundant portions into concise, clearly described new criteria.Unchanged criteria are excluded, and the output can be an empty list when no overlap is found.
- B.4 Criteria Review: Decompose: Criteria decomposition identifies excessively broad criteria that measure multiple distinct aspects and divides them into more specific criteria.The procedure focuses on independent or possibly unrelated response dimensions.
C TECHNICAL EVALUATION: DETAILS
The technical evaluation adapts MT Bench questions by separating each original question into a general prompt and a specific input.
- C TECHNICAL EVALUATION: DETAILS: MT Bench questions are used as the source material for the technical evaluation.Table 3 lists the original questions selected from the dataset.
- C TECHNICAL EVALUATION: DETAILS: Each original question is split into a general prompt and a specific input for evaluation.The adaptation changes how the dataset questions are presented during technical evaluation.
- C TECHNICAL EVALUATION: DETAILS: Table 3 documents the original questions and their corresponding prompt-input adaptations.The table includes examples for the evaluation setup.
D USER STUDY: SURVEY QUESTIONS
The user-study appendix describes seven-point post-task surveys measuring participants’ confidence in designing, improving, planning, and performing prompt evaluations.
- D USER STUDY: SURVEY QUESTIONS: Post-task survey responses use a seven-point Likert scale ranging from 1, Strongly Disagree, to 7, Strongly Agree.The scale measures agreement with each survey statement.
- D USER STUDY: SURVEY QUESTIONS: The survey measures self-confidence in designing prompts to instruct an LLM to perform a desired task.It also asks whether participants can improve prompts to produce better outputs for a given task.
- D USER STUDY: SURVEY QUESTIONS: The survey measures self-confidence in planning evaluation criteria and performing evaluations by examining generated outputs.These items address knowing which criteria an LLM should satisfy and measuring prompt performance.