PaperlayerView exact revision on arXiv

Source-linked AI summary

SANE: Automated Scorer-Specific Metamorphic Testing for LLM Judges

Ethan Ludwick, Yu-Chung Hsiao

arXiv:2610.28886v2cs.CLcs.AIcs.SE

TL;DR

LLM-judge reliability testing often relies on fixed, human-authored recipes that do not adapt to scorer-specific criteria. SANE derives and validates scorer-specific tests; across five domains it produces more validated families than JRH, and its faithfulness scores align with human judge rankings.

  • Problem

    Fixed, human-authored perturbation recipes can limit reliability tests to transformations and expected score relations that do not fit every scorer.

    Source: §1 · p. 2

  • Method

    SANE uses an agent to derive scorer-specific capabilities, perturbation recipes, and expected relations, then validates recipe alignment, case realization, and relation applicability.

    Source: §1 · p. 3

  • Results

    Across five domains, SANE produces 3.7 times as many validated test families as JRH and detects scorer deficiencies in all five, including professionalism and toxicity where JRH detects none.

    Source: §1 · p. 3

  • Takeaways & Limitations

    SANE's faithfulness test-suite scores rank 11 judge models consistently with human-labeled FaithBench performance, supporting automated scorer-specific testing as a practical method for LLM-judge reliability.

    Source: §1 · p. 3, §7 · p. 11

    Judge-ranking alignment is evaluated only for faithfulness, because comparable human-labeled benchmarks for other scorer criteria are not available in this evaluation.

    Source: §6 · p. 11

Abstract

from arXiv · show

LLM judges increasingly mediate benchmark evaluation, model selection, data curation, and training. Their decisions are useful only when each judge responds reliably to changes affecting its scoring criterion. Existing methods for reliability testing perturb inputs with known expected effects on judge scores. Prominent LLM-judge testing harnesses combine human-authored perturbation recipes with synthetic test generation, leaving scorer-specific recipe ideation outside the automated pipeline. This limits their use to fixed perturbations and pre-assumed scorer types. We introduce SANE, an agent-based framework that encodes behavioral requirements for stochastic evaluators as reusable agent instructions. Given a scorer specification, implementation, and representative inputs, the pipeline first identifies scorer-specific capabilities, then automatically derives perturbation recipes and expected score relations for each. SANE also defines a scorer-independent validation framework for generated judge tests. It determines whether each recipe agrees with scorer intent and whether each instantiated case realizes the recipe and supports its expected relation. Because SANE generates a recipe inventory tailored to each scorer, the number of aligned recipes grows as scorers are added. Across five domains, SANE produces 3.7 times as many validated test families as the Judge Reliability Harness (JRH) at comparable end-to-end validity and detects scorer deficiencies in every domain, including professionalism and toxicity, where JRH detects none. On faithfulness, SANE achieves strong human alignment in judge ranking: its test-suite scores rank 11 judge models consistently with human-labeled FaithBench performance (Spearman rho = 0.852), while JRH does not (rho = 0.100).

1 Introduction

LLM judges are useful only when they reliably measure their stated criteria, but aggregate agreement does not show how they respond to controlled input changes. SANE automates scorer-specific test design and validation, producing more validated test families than JRH across five domains.

1 Introduction

  • Aggregate agreement with human annotations does not characterize whether a judge responds reliably to input variations that should preserve or alter its criterion.Behavioral testing uses controlled changes to examine these responses before comparable variation occurs in deployment.Source: §1 · p. 1
  • Fixed perturbation recipes can misstate score direction across criteria: adding relevant detail may lower concision scores but raise coverage scores.The appropriate perturbation and expected relation depend on scorer specification, output contract, and evaluation data.Source: §1 · p. 2
  • SANE automates scorer-specific capability identification, recipe ideation, test-case generation, and validation in a reference-free agent-based framework.Its six fixed behavioral requirements are monotonicity, subadditivity, repeatability, robustness, dynamic range, and calibration; capabilities, perturbations, and expected relations vary by scorer.Source: §1 · pp. 2–3
  • SANE separates validation of recipe alignment, perturbation realization, and whether the expected relation applies to the realized case.Its behavioral requirements and relation types draw on established testing foundations; the contribution is automating scorer-specific recipe design.Source: §1 · p. 3
  • Across five domains, SANE produces 3.7 times as many validated test families as JRH and detects deficiencies in all five, including professionalism and toxicity where JRH detects none.SANE accumulates 108 unique aligned recipes, compared with seven for JRH.Source: §1 · p. 3

2 Related work

Earlier behavioral and metamorphic testing systems automate parts of test construction but often retain human-defined capabilities, topics, templates, or relation inventories. SANE instead derives scorer-specific capabilities, recipes, perturbations, and expected score relations.

2 Related work

  • Behavioral-testing systems reduce test-instantiation work, but many retain human selection of capabilities, topics, or templates; SANE automates those scorer-specific decisions.CheckList uses a fixed capability-and-test matrix, while later systems automate suggestions or case construction within human-guided structures.Source: §2 · p. 3
  • Metamorphic-testing research automates relation recovery or follow-up generation, yet examples retain developer-written tests, curated relation inventories, or domain context.SANE's entry point is an arbitrary scorer, from which it derives capabilities, recipes, perturbations, and expected score relations.Source: §2 · p. 4
  • Judge agreement with human preferences does not establish consistent measurement; reliability questions include repeated execution, prompt variation, scale choice, and criterion validity.Prior work studies these properties separately, including self-consistency, intra-rater reliability, and sensitivity to prompt complexity.Source: §2 · p. 4
  • Judge-test systems demonstrate targeted testing and automatic perturbation generation, but their inventories remain fixed after authors define the capabilities, categories, or recipes.JRH is the closest engineering comparison and uses a fixed suite of judge-reliability tests.Source: §2 · p. 4
  • SANE separates three validity questions: whether recipes match scorer intent, cases realize recipes, and expected relations apply to those cases.This provides explicit validity accounting from candidate recipe to validated test family.Source: §2 · p. 4

3 SANE

SANE derives scorer-specific capabilities, perturbation recipes, expected score relations, and test cases from general behavioral requirements. It validates both recipe intent and whether generated cases realize the recipe and support its expected relation.

3.2 Validity accounting

  • Validation separately checks whether each recipe and expected relation match scorer intent, whether each case realizes the recipe, and whether the relation applies to that case.A case is joint-valid only when all three conditions hold; otherwise it cannot support the intended reliability claim.Source: §3 · p. 5, §3.2 · pp. 5–6
  • A family is validated when a strict majority of its sampled cases are joint-valid; validated families together form the scorer’s test suite.The validation sample may be partial or include all generated cases, depending on budget and feasibility.Source: §3 · p. 5, §3.2 · p. 6, Figure 3 · p. 7

3.1 Scorer-specific test-suite generation

  • SANE identifies scorer-dependent capabilities, derives perturbation recipes with expected score relations, and instantiates cases from representative examples.The inputs include a scorer specification, implementation and output contract, representative examples, and fixed behavioral requirements.Source: §3 · pp. 4–5, §3.1 · p. 5
  • Each generated recipe is paired with its instantiated cases, preserving their provenance in a candidate suite that has not yet been validated.The pipeline distinguishes recipe design from test-case instantiation.Source: §3.1 · p. 5
  • For toxicity, SANE tests implicit toxicity by adding profanity-free belittling language and requiring the score not to decrease beyond a noise tolerance.The example passes the oracle panel unanimously for recipe alignment, realization, and relation applicability.Source: §3.1 · p. 5, Figure 2 · p. 6

4 Experimental design

The experiments compare fully automated SANE and JRH pipelines across five scorer domains, then test recipe validation, fault detection, and judge-ranking alignment with human evidence.

Experimental phases

  • The three-phase evaluation prepares and validates test suites, executes them against target scorers to identify violated expected relations, then measures whether harness pass rates rank judges consistently with a human-labeled benchmark.A fault-detecting family contains at least one joint-valid case that detects a scorer fault.Source: §4 · p. 7

Scorers and source data

  • The study covers faithfulness, fluency, coherence, professionalism, and toxicity using 1,200 examples sampled evenly from four summarization sources.The sources contribute 300 examples each; source genre varies while the summarization task remains common.Source: §4 · p. 7

Generation conditions

  • Both methods run without human intervention, and JRH is regenerated with the same model used for SANE to control for generator capability.The study preserves JRH’s released recipes, prompts, and expected relations.Source: §4 · p. 7

Oracle validation

  • Three independent GPT-5.2 predictions are majority-voted for each oracle decision, and unresolved oracle predictions count as invalid.Each aligned scorer-recipe pair receives n = 10 cases, with a strict-majority family threshold of τ = 6.Source: §4 · pp. 7–8

Human alignment

  • Human alignment is assessed by comparing harness pass-rate rankings with FaithBench AUROC rankings across 11 judge models, excluding Opus 5.The metrics measure different quantities, so the analysis compares their induced rankings rather than numerical values.Source: §4 · p. 8

Aligned-recipe analysis

  • Figure 5 enumerates scorer subsets, reporting individual-subset recipe counts, subset means, and the full minimum-to-maximum ranges.Enumerating subsets avoids imposing an arbitrary scorer-addition order.Source: Figure 5 · p. 9

5 Results

Across five scorers, SANE produces more validated test families than JRH and detects faults in every domain. On faithfulness, its test-suite rankings align with human-labeled judge rankings, whereas JRH’s do not.

5.1 Aligned recipes accumulate with scorers

  • Across five scorers, SANE produces 108 unique aligned recipes versus JRH’s seven, with aligned recipes increasing as scorers are added.The mean rises from 21 to 108 for SANE and from five to seven for JRH, from one scorer to all five.Source: §5.1 · p. 9

5.2 Validated and fault-detecting test families

  • 3.7 times as many validated families: SANE yields 77 of 133 candidates versus JRH’s 21 of 35, at similar end-to-end rates of 57.9% and 60.0%.Figure 6(a) tracks the scorer-recipe pairs through validation.Source: §5.2 · p. 9, Figure 6 · p. 10
  • 31 SANE families and 10 JRH families detect faults; SANE detects faults in all five domains, while JRH detects none in professionalism or toxicity.For coherence, JRH detects faults in three of four validated families and SANE in four of eleven.Source: §5.2 · p. 9, Figure 6 · p. 10
  • Across all measured per-scorer thresholds, Sane retains fault-detecting families in all five domains; its family-count advantage is independent of threshold, though its magnitude decreases at stricter settings.Source: §5.2 · p. 10

5.3 Human-aligned judge differentiation

  • Across 11 judge models, SANE’s faithfulness pass-rate ranking aligns with FaithBench (Spearman ρ = 0.852), while JRH’s does not (ρ = 0.100).SANE’s pass-rate span is 2.87 times wider, separating models in line with the FaithBench ranking.Source: §5.3 · p. 10, Figure 7 · p. 11

6 Limitations

The evaluation leaves open whether generation and realization rates transfer to weaker agent models, and human annotation could independently estimate oracle error.

6 Limitations

  • Whether the absolute generation and realization rates transfer to weaker agent models remains open.The comparison controls for generator capability by using the same model for both harnesses.Source: §6 · p. 11
  • Human annotation could provide an independent estimate of oracle error, which may affect reported scorer-deficiency counts.Judge-ranking alignment is evaluated only for faithfulness; extending it requires comparable benchmarks for other scorer criteria.Source: §6 · p. 11
  • Future work should test additional generator models and human validation panels.A retrieval library of validated scorer problems, capabilities, and recipes is also proposed for reuse across related domains while preserving scorer-specific reasoning.Source: §6 · p. 11

7 Conclusion

SANE automates scorer-specific metamorphic recipe ideation and validates recipes and generated cases before execution; across five scorers, it yields more validated and fault-detecting families than JRH and supports human-aligned judge differentiation on faithfulness.

7 Conclusion

  • SANE automates scorer-specific recipe ideation from general behavioral requirements and validates recipes and generated cases before execution.Across five scorers, it produces more aligned, validated, and fault-detecting test families than JRH; on faithfulness, its test-suite scores differentiate judge models in alignment with human evidence.Source: §7 · p. 11

A Scorer evaluation against human labels

The scorers are evaluated against domain-matched human-label benchmarks, with Stanford Politeness serving as evidence for one component of professionalism.

A Scorer evaluation against human labels

  • Faithfulness uses FaithBench, fluency and coherence use SummEval, professionalism uses Stanford Politeness as evidence, and toxicity uses Civil Comments.The benchmark targets and metrics differ, and results are not averaged across criteria.Source: A Scorer evaluation against human labels · p. 14, Table 1 · p. 14
  • Stanford Politeness evaluates one component of professionalism, rather than the full criterion.Source: Table 1 · p. 14

B Validity decomposition

The validity accounting separates recipe alignment, case realization, and relation applicability; SANE’s larger validated-family yield primarily reflects more applicable recipes.

B Validity decomposition

  • 133 SANE recipes and 35 JRH recipes were candidates, with 108 and 25 aligned recipes taken forward for case generation.Each candidate recipe contributes 10 potential cases; unresolved oracle predictions count as invalid.Source: Table 2 · p. 14
  • SANE and JRH yield candidate-to-joint-valid rates of 0.568 and 0.606, respectively, despite SANE having higher recipe alignment.JRH has higher conditional case realization, while relation applicability is nearly equal after realization.Source: B Validity decomposition · p. 14
  • The primary difference in validated-family counts comes from the number of applicable recipes, not a higher per-candidate validation rate.Source: B Validity decomposition · p. 14

C Aggregate threshold sensitivity

SANE retains more validated and fault-detecting families than JRH across the tested thresholds, though the gap narrows at stricter thresholds.

C Aggregate threshold sensitivity

  • The family threshold requires a strict majority of 10 sampled cases at τ = 6, and the sensitivity analysis recomputes counts through τ = 10.Source: C Aggregate threshold sensitivity · p. 14
  • At τ = 6, SANE has 77 versus 21 validated families and 31 versus 10 fault-detecting families; at τ = 10, the counts are 22 versus 14 and 8 versus 7.The comparison favors SANE at every threshold from τ = 5 through τ = 10, with a narrower gap at stricter settings.Source: C Aggregate threshold sensitivity · p. 14, Figure 8 · p. 15

D Threshold sensitivity by scorer

Across the three reported thresholds, SANE retains fault-detecting families for every scorer, while JRH has none for professionalism or toxicity.

D Threshold sensitivity by scorer

Related papers

Loading 2610.28886v2…