Source-linked AI summary
Beyond Aggregate Scores: Behavioral Correctness Assumptions for Assessing Reference-Based Automatic Evaluation Methods
Maria Mahbub, Ashley Rice, Michael R. Munroe, Amidu Kamara, Amir Sadovnik
TL;DR
Existing meta-evaluation mainly measures aggregate agreement or benchmark performance, leaving evaluator behavior under controlled response variations insufficiently characterized. The paper introduces behavioral correctness assumptions operationalized as controlled transformations and evaluates diverse reference-based evaluators. Results show distinct stability–sensitivity trade-offs, no evaluator satisfies all proposed assumptions, and similar aggregate performance can conceal different behavioral profiles.
Problem
Existing aggregate measures provide limited insight into whether evaluators respond appropriately to correctness-preserving and correctness-altering changes in generated responses.
Method
The paper defines behavioral correctness assumptions and tests them with controlled response transformations across lexical, character-level, semantic, LLM-based, and hybrid evaluators.
Results
Evaluators exhibit distinct stability–sensitivity profiles, with no evaluator satisfying all proposed correctness assumptions and similar aggregate performance masking different behavioral responses.
Takeaways & Limitations
Evaluator selection should consider application-relevant correctness properties rather than evaluation paradigm or aggregate performance alone.
Takeaways & Limitations
The test suite relies on LLM-generated baselines and transformations that cannot capture the full diversity of naturally occurring response variations, and the assumptions are not exhaustive.
Abstract
from arXiv · showhide
Automated reference-based evaluation methods play a critical role in assessing natural language generation systems. Existing meta-evaluation primarily measures agreement with human judgments or benchmark labels, providing limited insight into evaluator behavior under controlled conditions. We introduce behavioral correctness assumptions, a complementary framework for evaluating reference-based automatic evaluation methods. We define a taxonomy of correctness-preserving and correctness-altering assumptions and operationalize them through controlled response transformations that specify expected scoring behaviors. We evaluate diverse lexical, character-level, semantic, LLM-based, and hybrid evaluators and analyze their assumption-level behavior, stability, sensitivity, repeat-run variability, configuration sensitivity, and reproducibility. Our experiments reveal distinct behavioral trade-offs across evaluation paradigms: no evaluator satisfies all proposed correctness assumptions, and evaluators with similar aggregate performance can exhibit substantially different behavioral profiles. These findings demonstrate that behavioral correctness assumptions provide diagnostic information obscured by conventional aggregate meta-evaluation.
1 Introduction
Reference-based evaluators are widely used for assessing generated responses, but aggregate evaluation provides limited insight into whether they behave correctly under controlled response changes. This paper introduces a diagnostic framework using behavioral correctness assumptions and transformation-based tests to characterize evaluator behavior.
- Reference-based evaluation supports benchmarking, system optimization, training objectives, and deployment criteria for tasks including question answering, summarization, RAG, and conversational AI.
- Evaluation methods can assign conflicting scores to the same response because lexical, semantic, and LLM-based evaluators respond differently to surface form, prompting, models, and answer formulation.
- Aggregate agreement and benchmark measures provide limited evidence about responses to paraphrasing, added information, omissions, hallucinations, and logical inconsistencies.
- The framework defines behavioral correctness assumptions and operationalizes each through controlled response transformations that isolate response-quality dimensions.
- The study evaluates lexical, character-level, semantic, LLM-based, and hybrid methods, showing that assumption-level analysis reveals differences obscured by aggregate benchmark evaluation.
2 Related Work
Prior work spans overlap-based, semantic, learned, and LLM-based evaluators, but conventional meta-evaluation offers limited behavioral diagnosis. This paper extends controlled behavioral testing to reference-based automatic evaluation through explicit correctness assumptions.
- Lexical and character-level metrics rely on overlap, while semantic and learned metrics use contextual representations or learned models beyond surface matching.
- LLM-based evaluators score, rank, classify, or critique responses, but their judgments are sensitive to prompting, evaluator models, presentation biases, and inference configuration.
- Behavioral testing uses controlled changes to probe expected system behaviors beyond aggregate held-out accuracy and has been applied to perturbation sensitivity in semantic-similarity judges.
- This work extends behavioral testing to reference-based evaluation by providing an assumption-level diagnostic characterization complementary to benchmark-based meta-evaluation.
3 Methodology
The methodology evaluates reference-based evaluators through explicit correctness assumptions and controlled response transformations, complementing aggregate performance measures with fine-grained behavioral analysis. It aggregates behavior across transformations and assumptions, then summarizes evaluator stability and sensitivity across diverse evaluation methods.
- Aggregate benchmarks and human-agreement measures do not reveal whether evaluators respond appropriately to specific changes such as paraphrasing, omissions, or hallucinations.
- Correctness assumptions specify desirable evaluator behavior for correctness-preserving and correctness-altering response transformations.Preserving transformations should leave scores similar, whereas altering transformations modify response quality through changes such as factual errors, omissions, or irrelevant content.
- Each assumption is operationalized through controlled transformations that alter one targeted property while keeping other response characteristics as unchanged as possible.This design supports attributing score changes to the targeted transformation rather than unrelated response variation.
- The framework applies across lexical, character-level, semantic, hybrid, and LLM-based evaluators, including deterministic and potentially stochastic methods.
- Transformation-level analysis: Transformation-level analysis computes score differences between transformed and baseline responses, expecting approximately zero for correctness-preserving changes and negative differences for correctness-altering changes.
- Assumption-level analysis: Assumption-level analysis averages associated transformation results to produce one score difference per correctness assumption and evaluator.This aggregation summarizes behavior with respect to assumptions rather than individual transformation implementations.
- Summary-level analysis: Summary-level analysis measures stability under correctness-preserving transformations and sensitivity under correctness-altering transformations.The test suite is constructed from reference answers, generated baseline responses, and controlled transformations.
4 Results
Results reveal distinct behavioral trade-offs across evaluator paradigms: methods differ in how they handle correctness-preserving and correctness-altering transformations, and no evaluator is uniformly reliable. Stability, sensitivity, repeat-run variability, configuration sensitivity, and test-suite reproducibility further distinguish evaluators with otherwise similar aggregate behavior.
- Transformation-Level Analysis: Alternative valid answers are especially challenging: nearly every evaluator assigns them lower scores, with Semantic F1 (-0.25), Factual Correctness (-0.21), and Truthfulness (-0.20) most affected.This failure spans lexical, semantic, hybrid, and LLM-based evaluators despite the transformation preserving correctness.
- Transformation-Level Analysis: Correctness-altering transformations produce heterogeneous responses, with Logical Contradiction reducing Truthfulness (-0.41) and Factual Correctness (-0.44) more than Completeness (-0.25) or Answer Relevance (-0.16).Some evaluators detect violations but still assign transformed responses moderately high absolute scores.
- Transformation-Level Analysis: Evaluators exhibit two complementary failure modes: penalizing correctness-preserving changes or insufficiently penalizing genuine correctness violations.Each evaluator therefore has distinct strengths and weaknesses across correctness assumptions rather than universal reliability.
- Assumption-Level Analysis: Similarity-based evaluators are relatively uniform across assumptions, whereas LLM-based and hybrid evaluators show more differentiated profiles without consistently better behavioral correctness.Differentiated profiles encode richer response-quality distinctions, but some preserve unnecessary penalties and weak violation penalties.
- Summary-Level Analysis: Jaro combines the highest stability (0.937) with the lowest sensitivity (0.028), showing that consistency under preserving changes does not ensure detection of violations.Factual Correctness and Truthfulness instead provide the strongest stability–sensitivity combinations at (0.889,0.238) and (0.895,0.195).
- Implications: Evaluator selection should target the correctness properties important for an application because evaluators within the same family are not behaviorally interchangeable.Behavioral profiles can differ substantially even among LLM-based evaluators often grouped under “LLM-as-a-Judge.”
- Robustness and Reproducibility: Repeat-run profiles are highly consistent, with standard deviations below 0.0025 for every assumption and below 0.001 for Answer Relevance.Backbone choice shifts profiles more than decoding configurations, while regenerated test suites yield profile MAE values below 0.075.
- Robustness and Reproducibility: METEOR and chrF have the largest regenerated-profile MAE values, whereas Embedding Cosine Similarity and Jaro Similarity have the smallest.The differences suggest greater sensitivity of METEOR and chrF profiles to generated baselines and transformations, while principal behavioral patterns remain broadly preserved.
5 Discussion
The framework complements aggregate evaluation by revealing behavioral differences among evaluators and informing objective-specific selection. Its scope is bounded by generated transformations, incomplete assumptions, task-specific experiments, and single-reference evaluation.
- Practical Implications: Evaluators with similar aggregate performance may respond very differently to distinct aspects of response quality.
- Practical Implications: Evaluator selection should reflect the intended objective, balancing stability under valid reformulations with sensitivity to genuine correctness violations.
- Practical Implications: Behavioral profiles vary substantially within evaluator families, making lexical, semantic, hybrid, and LLM-based labels only coarse characterizations.
- Practical Implications: The framework complements conventional benchmarks by identifying correctness properties under which existing evaluators succeed or fail.
- Limitations: The test suite relies on LLM-generated baselines and transformations, which may not capture the full diversity of naturally occurring response variations.
- Limitations: The proposed assumptions are not exhaustive, and experiments on question answering with fixed documents and single references leave broader task and multi-reference generality unresolved.
6 Conclusion
The study introduces behavioral correctness assumptions as a diagnostic framework for reference-based evaluators. Results show distinct profiles and trade-offs across methods, with no evaluator satisfying all proposed assumptions.
- Conclusion: The framework evaluates lexical, semantic, hybrid, and LLM-based evaluators through controlled transformations at transformation, assumption, and summary levels.
- Conclusion: No evaluator satisfies all proposed correctness assumptions, and methods trade off stability under preserving transformations against sensitivity to correctness violations.
- Conclusion: Behavioral profiles are highly consistent across repeated runs and decoding configurations, while backbone choice produces larger shifts.
- Conclusion: Independent test-suite regeneration changes some effect magnitudes while broadly preserving the main behavioral characterization.
Funding Information
The research was funded under an interagency agreement with the U.S. Department of Homeland Security Science and Technology Directorate.
- Funding Information: Funding was provided under Interagency Agreement 70RSAT23KPM000049 by the U.S. Department of Homeland Security Science and Technology Directorate.
A Additional Evaluation Method Details
The study covers diverse reference-based evaluation paradigms, including deterministic metrics, LLM-based evaluators, and hybrid methods. Its evaluation setup uses controlled LLM-generated transformations and examines stochasticity across evaluator configurations.
- Evaluation Method Details: The study groups evaluation methods into lexical-overlap, character-level, semantic-similarity, LLM-based, and hybrid categories.
- Evaluation Method Details: Truthfulness evaluates factual accuracy against the reference, while Completeness assesses coverage of explicit and implicit key information.
- Evaluation Method Details: Hybrid methods combine LLM assessment with deterministic components such as embedding similarity or structured score computation.
- Evaluation Method Details: Lexical, character-level, and semantic methods are deterministic, whereas LLM-based and hybrid methods may produce non-deterministic scores.
- Evaluation Method Details: Prometheus-2 (8×7B) is the default LLM evaluator, with Llama-3.1-8B and Qwen-3-8B used to test backbone dependence.
- Evaluation Method Details: Each transformation is LLM-generated with instructions targeting one response modification while minimizing unintended changes.
C Data Statistics
The evaluation test suite compares reference answers with baseline and transformed responses, revealing expected length changes across transformation types.
- Reference answers average 55.09 words, whereas baseline responses average 34.31 words.
- Verbose transformations substantially increase response length.
- Concise and keydetail-omitted transformations produce shorter responses.
D Additional Results
Absolute transformation-level scores complement baseline-relative differences and provide context for judging whether evaluator behavior is desirable.
- Absolute scores complement baseline-relative differences when interpreting controlled transformation results.
- Small relative differences may still reflect undesirable behavior when evaluators score both baseline and transformed responses relatively low.
D.2 Stability–Sensitivity Analysis after Test-Suite Regeneration
Regenerating the evaluation test suite preserves the broad stability–sensitivity organization of evaluators while shifting individual measured values. Factual Correctness and Truthfulness remain sensitive and stable, whereas Jaro and Embedding Cosine Similarity remain stable but insensitive.
- Stability–Sensitivity Analysis after Test-Suite Regeneration: Test-suite regeneration preserves the broad organization of the evaluator space despite shifts in individual stability and sensitivity values.
- Stability–Sensitivity Analysis after Test-Suite Regeneration: Factual Correctness remains the most sensitive evaluator while maintaining high stability.
- Stability–Sensitivity Analysis after Test-Suite Regeneration: Truthfulness retains relatively high stability and sensitivity after test-suite regeneration.
- Stability–Sensitivity Analysis after Test-Suite Regeneration: Jaro and Embedding Cosine Similarity remain highly stable but comparatively insensitive to correctness-altering transformations.
- Stability–Sensitivity Analysis after Test-Suite Regeneration: Answer Relevance maintains high stability, although its sensitivity decreases on the regenerated test suite.
- Stability–Sensitivity Analysis after Test-Suite Regeneration: Regeneration affects measured-effect magnitudes more than the qualitative characterization of evaluator behavior.