Source-linked AI summary
Towards a Reliable and Practical Eval Pipeline
Emma Thuong Nguyen, Abhishek Ghose
TL;DR
LLM-based software features need LLM-assisted quality tests, while existing research does not cover the full practical requirements of reliable evals. This paper assembles an end-to-end pipeline using checklists and learned aggregation, improving judge agreement and providing self-consistency, explanations, and uncertainty.
Problem
LLM-based software features require LLM-assisted quality tests, but existing eval research addresses individual reliability aspects without covering the full range of practical use cases.
Method
The framework decomposes evaluation prompts into atomic YES/NO checklist questions and uses a learned tabular aggregator to produce final judgments and explanations.
Results
0.84 →0.96 overall average agreement when the aggregator is trained on data from all LLMs, while held-out-LLM evaluations also show high agreement across pairs.
Takeaways & Limitations
The pipeline combines practical eval components that can augment standard quality assurance and support explanations through influential checklist questions.
Takeaways & Limitations
Agreement scores are not chance-adjusted and should be interpreted jointly with accuracy.
Abstract
from arXiv · showhide
LLM-based software systems increasingly require effective "evals" as quality gates in the development lifecycle. However, existing work typically addresses individual aspects of eval reliability rather than the full set of practical requirements. We present an end-to-end eval pipeline that combines eval checklist creation, with learned aggregation for checklist responses, to improve agreement across LLM judges and accuracy against human judgments. The framework additionally pro- vides self-consistency, explanations, and prediction uncertainty, and we empirically demonstrate its effectiveness.
1 Introduction
LLM-based software features require LLM-based quality tests, but those tests inherit the nondeterminism they must measure. The paper proposes an end-to-end eval pipeline to address practical deployment gaps beyond individual alignment aspects.
- LLM-driven software features require quality tests that assess properties such as summary coherence or fluency.
- Because LLM-based tests share behaviors such as nondeterminism with the systems they evaluate, reliable measurement remains difficult.
- Prior eval research commonly focuses on specific alignment aspects rather than the full range of practical use cases.
- The paper presents practical eval desiderata and an end-to-end framework that augments a standard quality assurance pipeline.
- The pipeline sends checklist questions and eval inputs to an LLM, then uses a tabular model to produce judgment scores, explanations, and confidence.
2 A Practical Eval Pipeline
The proposed practical eval framework requires agreement across LLMs, accuracy against human judgments, self-consistency, explainability, and confidence scores. These requirements connect eval reliability to operational risks such as failures after judge changes, development cost, and uncertain decisions.
- P1. Inter-LLM Agreement: Inter-LLM agreement requires minimal variance across LLMs because changing to a harsher judge can increase test failures.
- P2. Accuracy: Accuracy requires eval outcomes to closely match human judgments, especially when product quality gates use hard thresholds.
- P3. Self-consistency: Self-consistency requires minimal variation across repeated executions, affecting the number of runs, development cost, and time-to-ship.
- P4. Explainability: Explainability should let users trace an eval judgment to contributing questions, analogous to tracing traditional test failures through stack traces or logs.
- P5. Confidence scores: Confidence scores should represent probabilistic outcomes with soft estimates such as prediction intervals rather than hard predictions.
3 Methodology
The methodology decomposes eval prompts into atomic YES/NO checklist questions, aggregates their LLM responses with a learned tabular model, and adds explanations and confidence estimates. Human examples support authoring the checklist and aggregator.
- Each eval is represented as a checklist of d atomic YES/NO questions whose responses form x ∈ {0, 1}^d.
- A learned model f aggregates checklist responses into the final judgment y = f(x).
- SHAP explanations trace judgments to specific checklist questions, while calibrated probabilities or conformal prediction provide confidence representations.
- A prior eval-authoring step uses a seed prompt and sample inputs and outputs to create the checklist and model f.
- Prompt decomposition targets ambiguity reduction and uses multiple related questions as an error buffer to improve agreement and self-consistency.
- The aggregator is trained with Gradient Boosted Decision Trees because they support varied output types and exact TreeSHAP attributions.
4 Experiment Setup
The experiments evaluate agreement, self-consistency, accuracy, explanations, and confidence using SummEval summaries across four quality axes. Four LLMs and five repeated trials provide the configurations for comparing these properties.
- The study measures agreement, self-consistency, and accuracy, while evaluating confidence scoring and explanations with standard techniques.
- Dataset and LLMs: SummEval provides documents, summaries, and human scores for coherence, consistency, fluency, and relevance, with summaries as measurement units.
- LLMs: Four LLMs—Opus, Sonnet, Sol, and Grok—are used to measure agreement.
- Number of trials: Each eval execution is repeated T = 5 times to measure statistical significance and self-consistency.
- For each eval, predicted scores are indexed by LLM and trial, with scores denoted as ŷ_ilt for summary s_i and an eval axis.
- One GBDT model per eval is trained on noisy checklist-response realizations paired with ground-truth labels, using a 50:10:40 train-conformal-test split.
- Accuracy is measured with RMSE, where lower scores are better.
- Agreement and self-consistency average absolute score differences over LLM or trial combinations and report 1 − D̄_i, but these scores are not chance-adjusted.
5 Results
The framework improves agreement across LLM judges, maintains or slightly improves self-consistency, and achieves the lowest RMSE across evaluated axes.
- Agreement: 0.84 →0.96 overall average agreement when the aggregator trains on data from all LLMs.The setup substantially outperforms the baseline on held-out comparisons.
- Agreement: High agreement persists when each row’s indexed LLM is held out from aggregator training.Agreements are computed with the other LLMs on held-out data.
- Self-consistency: Sol and Grok benefit most from the framework, while overall self-consistency improves slightly.Scores are averaged over summaries and evaluation axes.
- Accuracy: CheckEval reduces variability but is insufficient for good accuracy, and is worse than Baseline for coherence.The full CheckEval+ML pipeline produces the lowest RMSE score in every evaluated case.
6 Explanations
The checklist representation makes individual evaluation questions traceable, while SHAP identifies which questions most influence a particular judgment.
- Traceability: Checklist-based evals provide specific contributing questions to trace back from an outcome.This addresses the difficulty of locating causes in long prompts and inputs.
- SHAP explanations: TreeSHAP gives exact attribution values for GBDT aggregators, identifying influential checklist questions.Figure 4 presents these influences in a waterfall plot for a consistency evaluation on opus.
- Example: The example predicts a score of 4.85 for claude-opus-4-8, with an attribution of +0.11 from a question about correctly attributing statements.The displayed example ranks question influences from top to bottom.
7 Prediction Confidence
The framework uses conformal prediction to quantify uncertainty in LLM evaluations and produce prediction intervals with stated empirical coverage.
- Sources of uncertainty: Uncertainty reflects variability across LLMs, nondeterminism, and inputs poorly represented in training data.These factors motivate reporting prediction confidence.
- Uncertainty method: Conformal prediction reports intervals likely to contain the true outcome 90% of the time.The residual-normalized method is model-agnostic and remains invariant to future model changes.
- Validation: Higher bit-flip fractions b produce wider intervals on synthetic noisy data for both consistency and coherence evaluations.The inlier test-data distribution is expected to have relatively small widths, with empirical coverage shown.
8 Limitations and Conclusion
The paper presents a reliable eval pipeline intended to satisfy multiple practical requirements and support further industry-oriented development.
- Limitations: The pipeline is validated only on one summarization dataset and against proprietary LLMs.The authors identify expanding beyond this setting as an obvious limitation.
- Conclusion: The authors argue that the architecture and individual components may be built upon even when the exact pipeline is not adopted.The stated goal is to connect academic research with industry practice.
A.1 Baseline Prompts
The baseline prompts evaluate summaries one metric at a time using four dimensions: consistency, fluency, relevance, and coherence. Each dimension is defined by criteria for factuality, sentence quality, content selection, or organization.
- The evaluation form presents a summary and asks the evaluator to rate it on one metric.
- Coherence measures the collective structure and organization of sentences into a coherent body of information about a topic.
- Consistency measures factual alignment between a summary and its source, penalizing statements that are not entailed or that contain hallucinated facts.
- Fluency measures individual sentence quality, including formatting, capitalization, and grammatical correctness.
- Relevance measures whether the summary selects important source content while avoiding redundancies and excess information.
A.2 Generated Checklists
The generated checklists specify checklist sizes for SummEval dimensions, using CheckEval-derived sub-dimensions and seed questions with final questions generated by Opus 4.8. The section also reports GBDT search settings and interval-width results for synthetic noisy data.
- Checklist sizes are reported separately for each SummEval dimension, with sub-dimensions and seed questions taken from CheckEval and final questions generated by Opus 4.8.
- The GBDT hyperparameter search varies n_estimators, learning_rate, max_depth, min_child_samples, and reg_lambda across specified candidate values.
- Figure 6 compares interval-width distributions for inliers and synthetic noisy data created by flipping fractions b ∈{0.2, 0.4, 0.6} of bits.
- The inlier empirical coverage is close to the intended coverage of 90%.