PaperlayerView exact revision on arXiv

Source-linked AI summary

Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences

Shreya Shankar, J. D. Zamfirescu-Pereira, Björn Hartmann, Aditya G. Parameswaran, Ian Arawjo

arXiv:2404.12272v1cs.HCcs.AI

TL;DR

LLM-generated evaluators can inherit the reliability problems of the models they assess, yet existing systems often lack ways to verify alignment with human preferences. EvalGen combines LLM-generated criteria and candidate assertions with human grading to select better-aligned evaluations, and a qualitative study found overall support while revealing criteria drift and output-dependent criteria.

  • Problem

    Existing LLM-assisted evaluation systems often ask users to trust evaluators whose sensitivity and alignment with human preferences remain insufficiently verified.

  • Method

    EvalGen generates criteria and candidate code or grader assertions, then ranks them by alignment with users’ grades.

  • Results

    A qualitative study found overall support for EvalGen but identified criteria drift, with users refining evaluation standards as they graded more outputs.

  • Takeaways & Limitations

    Evaluation assistants should support rapid iteration over criteria and implementations because some evaluation criteria depend on the observed LLM outputs.

  • Takeaways & Limitations

    The study evaluated only two pipelines with a small sample of experienced developers, limited iteration, and no deployment-phase coverage.

Abstract

from arXiv · show

Due to the cumbersome nature of human evaluation and limitations of code-based evaluation, Large Language Models (LLMs) are increasingly being used to assist humans in evaluating LLM outputs. Yet LLM-generated evaluators simply inherit all the problems of the LLMs they evaluate, requiring further human validation. We present a mixed-initiative approach to ``validate the validators'' -- aligning LLM-generated evaluation functions (be it prompts or code) with human requirements. Our interface, EvalGen, provides automated assistance to users in generating evaluation criteria and implementing assertions. While generating candidate implementations (Python functions, LLM grader prompts), EvalGen asks humans to grade a subset of LLM outputs; this feedback is used to select implementations that better align with user grades. A qualitative study finds overall support for EvalGen but underscores the subjectivity and iterative process of alignment. In particular, we identify a phenomenon we dub \emph{criteria drift}: users need criteria to grade outputs, but grading outputs helps users define criteria. What is more, some criteria appears \emph{dependent} on the specific LLM outputs observed (rather than independent criteria that can be defined \emph{a priori}), raising serious questions for approaches that assume the independence of evaluation from observation of model outputs. We present our interface and implementation details, a comparison of our algorithm with a baseline approach, and implications for the design of future LLM evaluation assistants.

1 INTRODUCTION

LLM-assisted evaluation can improve the efficiency of validating LLM outputs, but evaluator LLMs are themselves unreliable and often remain unverified. EvalGen addresses this alignment problem by combining generated evaluation criteria and assertions with human output grades, while exposing criteria drift as a fundamental challenge.

  • LLM outputs require validation because models hallucinate, ignore instructions, and produce other mistakes.
  • Evaluator LLMs are also unreliable, with grader prompts sensitive to seemingly minor wording or structural changes and often lacking quality verification.
  • EvalGen generates editable natural-language criteria and candidate assertions, then uses users’ good/bad grades to select assertions aligned with their preferences under effort and latency constraints.
  • EvalGen was embedded in ChainForge and evaluated through an offline comparison with SPADE plus a qualitative study of nine industry practitioners.
  • Criteria drift describes the catch-22 in which people need criteria to grade outputs, while grading outputs helps define those criteria, making complete pre-specification impossible.

2 MOTIVATION AND RELATED WORK

As LLM use expands, practitioners increasingly automate prompt and output evaluation with code or LLM-based evaluators, but existing tools provide limited support for validating evaluator quality. Prior work shows that humans remain essential yet are vulnerable to over-reliance and over-generalization, motivating better support for prototyping and validating evaluations.

  • Automating Evaluations of Prompts: Users increasingly automate evaluation of hundreds or thousands of model queries with code or LLM-based evaluators.Public prompt-engineering tools let users define evaluation metrics, while other systems use LLMs to compare prompts.
  • Automating Evaluations of Prompts: Existing prompt-engineering tools provide little assistance for validating evaluator quality, often leaving users to inspect generated scores manually or hiding individual scores.EvalLM includes an alignment feature, but the supplied passage notes that it was described only in the design section and absent from its user study.
  • Over-trust and Over-generalization of LLM Behavior: People tend to over-rely on and over-trust AI systems, creating risks when LLM-generated evaluations are accepted without sufficient validation.The passage frames this concern through research on human reliance and a disputed claim about GPT-4’s performance on the MIT EECS exam.
  • Over-trust and Over-generalization of LLM Behavior: Users also over-generalize from individual failures, discarding potentially good prompts instead of assessing holistic performance even when systematic testing is available.This pattern was reported among users unfamiliar with prompt engineering and was also observed among developers and ML academics struggling to scale evaluation.
  • Approaches to Aligning LLMs: The work identifies a need for support in both prototyping evaluations and validating evaluators, building on interactive alignment methods and the Python assertion generation approach of SPADE.Interactive machine-learning interfaces connect example selection, labeling, and performance evaluation, while SPADE generates assertion functions from prompt history.
  • Approaches to Aligning LLMs: Prior alignment approaches rely heavily on humans to identify desirable and undesirable outputs, including annotated examples used to calibrate LLM-based evaluators.The literature includes custom model training, finetuning, and few-shot calibration based on criteria such as consistency and relevance.

3 EVALGEN DESIGN

EvalGen is an LLM-powered assistant that gives developers control over evaluation criteria, evaluator types, and generated implementations without requiring them to write criteria, code, or grader prompts. It combines automated generation with selective human grading to align evaluators with developer preferences while reporting alignment and exposing the tradeoff between effort and verification.

  • Design goals: EvalGen lets developers control metric criteria, evaluator type, and implementation generation and selection without writing criteria, code, or grader prompts themselves.Criteria can be code- or LLM-based, and the system supports generated or user-added criteria.
  • Interface workflow: The workflow offers Infer, Manual, and Grade First modes, then lets users edit, add, delete, or deselect criteria and choose code- or LLM-based evaluators.A second LLM generates candidate implementations from the selected criteria.
  • Human validation: Users grade sampled LLM outputs as candidate implementations run, and EvalGen uses those grades to select implementations aligned with their preferences.The grading interface presents the response, prompt context, and input variables, with Good and Bad controls.
  • Feedback: A final Report Card presents per-criterion and aggregate alignment measures, including confusion matrices, coverage, and false failure rate.Per-criterion metrics show how each criterion aligns with human grades.
  • Design trade-offs: The design balances developer effort against verification robustness because fully aligning an LLM evaluator would require labeling every output, defeating the effort-saving purpose.Selective grading uses time developers might otherwise spend waiting for generations and assertions to finish.

4 IMPLEMENTATION

EvalGen decomposes evaluations into criteria and assertions, generates candidate implementations with LLMs, and progressively ranks them using user feedback. Its online architecture maintains dynamic selectivity, confidence, and alignment estimates while sampling outputs for grading.

  • Architecture: EvalGen decomposes evaluations into criteria and boolean assertions, generates candidate implementations, and ranks assertions by alignment with user grades.Candidates may be code-based functions or LLM grader prompts.
  • Architecture: Unlike SPADE’s offline integer-programming approach, EvalGen uses an online streaming architecture to progressively optimize the most aligned assertion set.The system generates and evaluates candidate implementations as user feedback arrives.
  • Candidate generation: GPT-4 proposes binary criteria and asynchronously synthesizes multiple candidate assertions as executable code or grader prompts.Developers can select suggested criteria or add their own, then choose code-based or LLM-based evaluation.
  • Feedback and estimation: User feedback updates each candidate’s alignment estimate, while the grading sampler selects subsequent LLM outputs for binary thumbs-up/down judgments.Alignment is recalculated whenever the user grades an LLM pipeline output.
  • Assertion selection: After grading ends, EvalGen selects the highest-alignment assertion for each criterion, optionally excluding assertions above a user-specified false failure rate threshold.Alignment is defined as the harmonic mean of coverage and false failure rate.
  • Feedback and estimation: EvalGen dynamically estimates assertion selectivity and output-quality confidence; selectivity is the probability that an assertion classifies an output as passing.Confidence scores depend on assertion selectivity and are revised after assertion execution or direct user grading.

5 ALGORITHM EVALUATION

EvalGen’s human input during criteria selection produced assertion sets with equal or better alignment than SPADE while using fewer assertions across medical and product pipelines. The evaluation also exposed coverage–false-failure trade-offs and differences in the criteria each system selected.

  • Method comparison: EvalGen solicited user edits to criteria before generating candidate assertions, whereas SPADE generated criteria and assertions without user input.The systems were compared on assertion-set size and alignment, with SPADE selecting a minimal set subject to coverage and false failure constraints.
  • Datasets and setup: The medical pipeline used 84 doctor–patient transcripts, while the product pipeline used 100 Amazon products and reviews, with outputs generated by GPT-3.5-Turbo.Manual grading found 68% of medical outputs and 51% of product outputs good; issues included PII, bad reviews, and excessive length.
  • Baseline evaluation: SPADE met a 10% false failure rate threshold for the medical pipeline, but the product pipeline required a 40% false failure rate to yield a viable assertion set.The result illustrates the difficulty of balancing coverage with false failures in fully automated assertion selection.
  • Results: EvalGen achieved equal or better alignment than SPADE with fewer assertions in both pipelines by incorporating human judgment during criteria selection.In the product pipeline, EvalGen’s assertion set was less than half the size of SPADE’s and increased coverage from 49%.
  • Qualitative differences: SPADE generated unnecessary or unrealistic assertions, including neutral-tone checks, over-split key checks, and phrase-based negative-review detectors.EvalGen instead used criteria confirmed by the user, avoiding assertions for criteria users did not consider important.

6 USER STUDY

The user study examined how nine industry practitioners used EvalGen to build evaluators for an LLM pipeline, using think-aloud sessions and qualitative coding to identify common themes.

  • Participants: Nine industry practitioners with prior LLM experience used EvalGen and thought aloud while building evaluators.Participants included software engineers, ML scientists, startup executives, and independent consultants.
  • Procedure: Participants evaluated a GPT-3.5-Turbo named entity recognition pipeline on tweets and could inspect outputs, generate or write assertions, run assertion sets, and view results.Each participant had up to 40 minutes to develop an assertion set.
  • Procedure and Analysis: Researchers followed sessions with open-ended questions about assertion generation and alignment, then analyzed think-aloud notes and transcripts using open and axial coding.Researchers recorded participants’ thoughts and visible emotions while using the tool.

7 USER STUDY FINDINGS

Participants generally viewed EvalGen as a useful, controllable starting point for generating assertions, but alignment remained subjective and iterative. Grading both helped validate assertions and reshaped participants’ criteria, producing criteria drift as they encountered new output behaviors.

  • Overall perceptions: Participants generally liked EvalGen as a starting point for generating assertions and wanted control because the system sometimes made mistakes.Eight of nine participants were pleasantly surprised that suggested criteria reflected their preferences.
  • Grading process: Grading was useful while EvalGen worked, but participants struggled to enumerate all criteria and grade outputs consistently.All but one participant considered grading during candidate generation a good use of time, while three wanted criterion-by-criterion prompts.
  • Inspecting alignment: Participants valued alignment scores and assertion-result tables for inspecting implementations and outputs, which helped them identify unaligned assertions rather than blindly trust them.They also wanted visualizations showing how prompt changes affected coverage and false failure rate.
  • Confidence and control: Confidence in alignment was polarized because participants remained uncertain about their own criteria while grading and wanted more flexible, criterion-specific control.Participants especially wanted to prioritize criteria needing alignment and set different false failure rates for different criteria.
  • Criteria drift: Criteria drift occurred as grading outputs led participants to add new criteria and reinterpret existing ones to fit observed LLM behavior.Participants encountered new types of bad outputs and revised interpretations such as what counted as a proper noun or acceptable hashtag usage.

8 DISCUSSION

The discussion finds that criteria and grading should co-evolve because users refine criteria while engaging with observed LLM outputs, challenging fixed ground-truth assumptions. It also outlines design requirements for deployment, collaboration, richer monitoring, prompt improvement, and acknowledges study limitations.

  • Criteria drift: Developers rapidly iterate on criteria, and engaging with LLM outputs helps refine them, so interactive systems should support criteria refinement and grading in tandem.This challenges approaches that assume settled criteria established before observing outputs.
  • Future evaluation assistants: Evaluation assistants could dynamically adjust criteria, elicit per-criteria grades, and include examples of both good and bad outputs to improve implementation alignment.Per-criteria feedback may help adjust both evaluation criteria and their implementations more precisely.
  • Criteria drift: Criteria may adapt to the evaluated LLM outputs rather than settle independently, raising questions about whether alignment is an actualizable goal or an ongoing process.The discussion questions assumptions that evaluation can recover a fixed ground truth of labels.
  • Operational deployment: Code-based and LLM-based assertions are both necessary but require distinct treatment in implementation selection and alignment, with code assertions often suited to critical-path sanity checks.Participants also wanted assertions deployed either inline to block bad outputs or passively through periodic reports.
  • Collaboration: Collaborative grading requires evaluation assistants to address inter-rater reliability and disagreements, potentially using crowdsourcing methods and review workflows.Participants imagined collaborator grading, assertion review, and CI/CD-like production deployment.
  • Beyond binary assertions: Future assistants could monitor output distributions and refine prompts from assertion failures, but prompt improvement risks cyclical interactions between evaluation and generation.The study was limited to two pipelines, a small experienced developer sample, limited iteration time, and no deployment phase.

9 CONCLUSION

The paper presents EvalGen as a mixed-initiative approach for aligning LLM-generated evaluation functions with human preferences. A qualitative study with nine expert users identifies criteria drift as users refine evaluation standards while grading outputs.

  • EvalGen: EvalGen helps users develop acceptable evaluation criteria for LLM outputs and functions that check those standards.The approach is intended to ensure evaluations reflect users’ own grading standards.
  • EvalGen: EvalGen is a mixed-initiative approach to aligning LLM-generated evaluation functions with human preferences.
  • Criteria drift: 9 expert users participated in a qualitative study that revealed criteria drift as users refined evaluation standards while grading more LLM outputs.The passage names this pattern criteria drift.

A ALGORITHMS FOR SELECTING ASSERTIONS & ELICITING GRADES … A.3 Choosing Aligned Assertions

EvalGen combines assertion selectivity, strategic grade sampling, and alignment scoring to identify evaluation assertions that better reflect human judgments of LLM output quality. The approach emphasizes representative feedback and balances coverage of bad outputs against false failures.

  • A.1 Assertion Selectivity and Impact on LLM Output Quality Confidence: Assertions with low selectivity provide limited evidence about output quality because they fail frequently, including assertions that fail every output.A trivially failing assertion has selectivity 0 and offers no discernment.
  • A.1 Assertion Selectivity and Impact on LLM Output Quality Confidence: EvalGen uses assertion selectivity estimates to assign each LLM output a confidence score reflecting the likelihood of poor quality.The score is non-negative; score 0 means no assertions failed, while failures by non-selective assertions indicate uncertainty or potential issues.
  • A.2 Sampling Grades: Because users may grade only a limited number of outputs, EvalGen must select samples that represent the dataset and avoid misalignment from unrepresentative random selections.Prior work also emphasizes soliciting representative graded samples of LLM outputs.
  • A.2 Sampling Grades: EvalGen considers random, highest-score, lowest-score, and alternating sampling strategies for selecting outputs to grade.Highest-score sampling targets potentially problematic content, lowest-score sampling prioritizes outputs passing assertions or failing low-selectivity assertions, and alternating sampling seeks diversity.
  • A.2 Sampling Grades: The study compares these sampling strategies with a random baseline on two LLM pipelines, while user studies use alternating sampling to seek balanced good and bad grades.The authors explicitly do not claim alternating is the best policy.
  • A.2 Sampling Grades: Middle-score sampling is omitted because scores estimate poor-quality likelihood rather than goodness or badness, and low scores can still encompass widely varying quality.Thus, low scores reflect system uncertainty rather than a uniform class of good outputs.
  • A.3 Choosing Aligned Assertions: After grading, EvalGen selects the candidate assertion with the highest alignment score for each criterion.Alignment is defined over LLM pipeline input-output pairs and user labels indicating whether each output is bad.
  • A.3 Choosing Aligned Assertions: An aligned assertion set should achieve high coverage and low false failure rate, with alignment computed as their harmonic mean using the inverse false failure rate.Coverage corresponds to the true negative rate, while false failure rate corresponds to the false negative rate; the formulation resembles F1 but evaluates failures over assertion sets.

A.4 Evaluation of Sampling Policy

EvalGen’s alternating sampling policy was compared with random, highest, and lowest sampling across repeated trials. Random sampling showed substantial alignment variance, while selectivity-weighted alternatives performed more consistently; deterministic policies may still evolve during real-world use as user feedback accumulates.

  • Experimental setup: EvalGen’s alternating policy was evaluated against random, highest, and lowest sampling using 10 trials of 16 graded outputs each.The study used the same LLM pipelines described in Section 5.1.
  • Sampling-policy results: Random sampling introduced large variance in alignment, making assertion relevance unpredictable and potentially frustrating users.The concern is that grading effort may yield assertion sets with inconsistent relevance to specified criteria.
  • Sampling-policy results: Selectivity-weighted alternative policies consistently yielded higher alignment than random sampling.Their grading probabilities were weighted by the selectivity, or pass rate, of assertions that failed each output.
  • Real-world behavior: Non-random policies produced no outcome variation offline because they were deterministic, but EvalGen can change predictions over time as new feedback arrives.The first one or two outputs may effectively be sampled randomly while σscores update and stabilize.

B TASK PROMPTS

The study used two task prompts and datasets: medical-record insight extraction and SEO-oriented product-description writing. The prompts specified structured outputs, privacy constraints, content requirements, and approximate length limits.

  • Task coverage: Two task pipelines presented to users covered medical-record processing and product-description writing, adapted from prior work.Each pipeline had a corresponding prompt and dataset.
  • Medical-record prompt: The medical prompt required extracting specified clinical fields, including complaints, history, examination, symptoms, medication changes with dosages, and follow-up instructions.Missing values had to be reported as N/A.
  • Medical-record prompt: The medical prompt prohibited personally identifiable information, required referring to “the patient,” and prescribed a bullet-list format of around 150 words.The prompt also instructed users to summarize extracted values when necessary to meet the word limit.
  • Product-description prompt: The product prompt requested an SEO-optimized, active-voice description covering features, buyer benefits, unique selling points, and a call to action without overpromising.It also prohibited invented review text, links, and heavy review citation, while requiring Markdown subheadings and approximately 200 words, capped at 300.
Loading 2404.12272v1…