PaperlayerView exact revision on arXiv

Source-linked AI summary

Mitigating Rubric Interference in LLM Judges via On-Policy Self-Distillation

Dingyao Yu, Tong Zhang, Yutao Mou, Yunxiao Zhang, Wei Ye, Shikun Zhang

arXiv:2608.14684v1cs.LGcs.AI

TL;DR

Joint multi-rubric LLM evaluation suffers from rubric interference, with verdicts changing as rubric sets vary. SARA anchors joint reasoning to the model’s isolation judgments through self-distillation, improving consistency across datasets, model families, and scoring formats while transferring across domains without retraining.

  • Problem

    Joint evaluation of multiple rubrics can produce verdicts that shift with rubric-set composition, unlike separate isolation evaluation.

  • Method

    SARA uses the model’s stable single-rubric judgments as anchors for aligning joint-mode reasoning through on-policy self-distillation.

  • Results

    SARA improves evaluation consistency across datasets, model families, and scoring formats, with consistency transferring across domains without retraining.

  • Takeaways & Limitations

    The results support self-distillation from simpler evaluation settings as a way to improve consistency without external supervision.

  • Takeaways & Limitations

    SARA assumes isolation judgments are interference-free anchors and does not distinguish harmful interference from useful cross-rubric context.

Abstract

from arXiv · show

LLM judges increasingly evaluate responses against fine-grained rubric checklists. When a sample requires multiple rubrics, current methods typically assess each in a separate inference call. Evaluating all rubrics in a single pass is a natural alternative with greater efficiency, but we find that it introduces rubric interference: the verdict on one rubric shifts depending on which other rubrics are co-present. In a preliminary study, only one-third of samples receive fully consistent verdicts when evaluated under rubric sets of varying composition. We develop a measurement framework that probes interference through four controlled operations: rubric set expansion, subsetting, reordering, and noise injection. To mitigate interference without external supervision, we propose Self-Anchored Rubric Alignment (SARA). SARA uses a model's own single-rubric judgments as stable anchors and aligns multi-rubric reasoning with these anchors through on-policy self-distillation. We validate SARA on three datasets (HealthBench, FLASK, ResearchQA) and two model families (Qwen3, Llama-3.1). SARA consistently improves evaluation consistency while maintaining agreement with both base models and GPT-4.1 as a reference judge. Furthermore, the learned consistency transfers across datasets, confirming that SARA teaches a general capability rather than fitting dataset-specific patterns.

1 Introduction

Rubric-based LLM evaluation becomes unreliable when multiple rubrics are assessed jointly: verdicts can shift with rubric composition or ordering, and Qwen3-32B achieves only 36% sample-level exact match against isolation judgments on HealthBench. SARA addresses this interference by aligning joint-mode reasoning with the model’s isolation judgments through on-policy self-distillation, improving consistency across datasets and model families without external supervision.

  • 1 Introduction: 36% sample-level exact match is achieved by Qwen3-32B when comparing isolation and joint verdicts on HealthBench.Reordering rubrics can also flip a verdict.
  • 1 Introduction: Rubric interference is a systematic failure mode in which a rubric’s verdict changes with the composition or ordering of co-present rubrics.The measurement framework quantifies interference through rubric set expansion, subsetting, reordering, and noise injection across binary and scored formats.
  • 1 Introduction: Isolation judgments provide stable, annotation-free anchors, while differences between isolation and joint verdicts expose where interference distorts reasoning.These differences supply a training signal without external supervision.
  • 1 Introduction: SARA uses on-policy self-distillation to align joint-mode reasoning with the model’s own isolation judgments.It parses multi-rubric evaluations into per-rubric segments, uses a slowly updated isolation-context teacher, applies divergence loss to distorted analysis, and preserves verdict structure.
  • 1 Introduction: SARA improves evaluation consistency across three datasets and two model families while preserving agreement with base models and GPT-4.1.The method requires no external supervision.

2 Related Work

Prior work established LLM judges as scalable, rubric-based evaluation infrastructure while documenting systematic biases and broad prompt sensitivity. This paper focuses specifically on how context changes the labels judges assign to other models’ outputs and draws on on-policy and self-distillation methods.

  • LLM-as-a-Judge: LLM judges became standard evaluation infrastructure, with GPT-4 matching inter-annotator agreement on MT-Bench and chain-of-thought improving correlation with human ratings.Task-specific checklists have also shifted evaluation toward structured, per-sample criteria.
  • Biases and reliability of LLM judges: LLM judges exhibit position, verbosity, and self-enhancement biases, while prompt complexity sensitivity and general leniency further compromise reliability.Position bias persisted across 12 judge models and over 100K evaluation instances, indicating a stable, model-specific pattern rather than a sampling artifact.
  • Prompt sensitivity and evaluation consistency: Formatting changes alone can cause up to 76-point accuracy swings, motivating multi-prompt evaluation and broader consistency studies.Paraphrase-based frameworks measure whether equivalently phrased queries elicit the same factual answer.
  • Prompt sensitivity and evaluation consistency: This work studies LLM sensitivity as judges: how context changes the label assigned to another model’s output.The focus differs from prior work examining prompt sensitivity in LLMs as task-solvers.
  • On-policy distillation: On-policy distillation trains students on self-sampled trajectories with per-token teacher guidance to mitigate distribution shift, while self-distillation removes the need for a separate teacher.Prior LLM work also trains models to distinguish their own outputs from references and to serve as their own reward function.

3 Problem Definition

The paper formalizes rubric interference as verdict dependence on co-present rubrics and defines consistency as invariance to rubric-set composition. It evaluates this invariance through four controlled rubric-set operations and reports rubric- and sample-level agreement metrics.

  • Formal setup: The judge evaluates a conversation against rubrics R and assigns each rubric a verdict from a discrete value set, covering binary, scored, and ordinal scales.The framework explicitly includes V = {0, 1}, scored scales such as V = {1, . . . , 5}, and extensions to any ordinal scale.
  • Formal setup: Isolation evaluates each rubric in a separate call, whereas joint evaluation assesses all rubrics in one pass but may introduce rubric interference.Joint evaluation is more efficient, but its verdict for a rubric may depend on the other rubrics in the set.
  • Consistency testing: An ideal judge produces the same verdict for a rubric regardless of which other rubrics accompany it.The consistency tests change rubric sets while keeping the conversation and target rubric fixed, checking whether the shared-rubric verdict remains stable.
  • Consistency testing: The framework probes interference through expansion, subsetting, reordering, and noise injection, using agreement, Cohen’s κ, and sample-level exact match.Expansion compares isolation with joint verdicts and tracks overall consistency and consistency-at-K; subsetting compares nested subsets of arbitrary size.

4 Self-Anchored Rubric Alignment

SARA reduces rubric interference by distilling multi-rubric reasoning toward stable single-rubric teacher judgments while preserving verdict structure. On-policy generation, rubric-context augmentation, and isolation-anchored losses expose and correct diverse interference patterns without external annotations.

  • 4.1 Overview: SARA uses the model’s reliable single-rubric judgments as stable, annotation-free anchors for training joint evaluations.The same model that struggles with joint evaluation can judge each rubric reliably in isolation.
  • 4.1 Overview: At each step, the student generates a joint evaluation, while a slowly updated teacher evaluates each rubric segment in isolation to produce interference-free logits.A divergence loss aligns analysis segments, and a preservation loss maintains verdicts, formatting, and transitions.
  • 4.2 On-Policy Generation and Segment Extraction: The on-policy student output is parsed into per-rubric analysis and structural segments, so training follows the model’s evolving behavior.Analysis tokens receive an analysis mask, while verdict, formatting, and transition tokens receive the complementary structure mask.
  • 4.2 On-Policy Generation and Segment Extraction: Segment extraction relies on predictable instruct-mode rubric-analysis-verdict outputs, whereas thinking-mode models require model-specific parsing.This assumption is addressed in Appendix I.
  • 4.3 Isolation-Anchored Distillation: The teacher is a slowly updated exponential-moving-average copy whose high decay stabilizes anchors and avoids reinforcing drifting or biased student judgments.EMA smooths the reference temporally and prevents a degenerate self-anchoring loop.
  • 4.3 Isolation-Anchored Distillation: For analysis tokens, SARA aligns student logits under the full rubric set with teacher logits for the target rubric under isolation context.The student may exhibit interference because it sees all rubrics, while the teacher produces interference-free logits from only the target rubric.
  • 4.3 Isolation-Anchored Distillation: Structural tokens instead use full-context teacher logits because verdicts, formatting, and transitions coordinate across rubrics and lack meaningful isolation equivalents.The preservation-loss weighting coefficient is α=0.4 across all experiments.
  • 4.4 Rubric Context Augmentation: Rubric shuffling and noise injection expose diverse interference patterns by disrupting positional memorization and adding unrelated criteria as distractors.Shuffling targets positional sensitivity, while noise injection targets content-based distraction.

5 Experiments

Across three datasets and two model families, SARA consistently improves isolation–joint evaluation consistency, with especially large gains on ResearchQA and under rubric-count, shuffle, and noise stress tests. It reduces interference while preserving base-model behavior and GPT-4.1 agreement, and its effects transfer across domains through reduced cross-rubric attention.

  • Overall consistency: SARA consistently improves consistency across model scales, architectures, and evaluation domains, with the largest gains on ResearchQA.Qwen3-8B’s ResearchQA EM nearly triples from .22 to .59, while κ rises from .665 to .868.
  • Overall consistency: +14.2% Agr is achieved by Llama-3.1-8B on ResearchQA, while Qwen3-32B still improves substantially on scored formats.For Qwen3-8B on ResearchQA, Agr increases by .140 but EM increases by .370, showing disproportionately large exact-match gains.
  • Quality and behavioral preservation: SARA preserves evaluation behavior while reducing interference: SFTiso has higher consistency, but lower agreement with the untrained base in both isolation and joint modes.SFTiso versus SARA achieves Agr .909 vs .900 and EM .54 vs .52, while base agreement is Iso .892 vs .925 and Joint .777 vs .806.
  • Quality and behavioral preservation: SARA improves GPT-4.1 agreement on HealthBench from .795 to .806 for Qwen3-14B joint outputs and from .726 to .755 for Llama-3.1-8B joint outputs.Agreement with the untrained base exceeds .84 in all settings.
  • Robustness to rubric composition: Qwen3-8B’s MAD remains .080, .080, and .081 at rubric counts K=2, 4, and 8 with SARA, whereas Qwen3-32B’s base MAD rises from .203 to .362.Across models, SARA compresses the MAD range from .147–.374 to .080–.176 and eliminates its upward trend with rubric count.
  • Robustness to rubric composition: SARA raises Qwen3-8B shuffle EM from .298 to .594 and noise EM from .255 to .620 on ResearchQA.The base model reaches only .262 EM under shuffling for Qwen3-14B and falls to .200 under noise for Qwen3-32B.

6 Conclusion

The paper identifies rubric interference as a systematic failure mode in multi-rubric LLM evaluation and proposes SARA to align joint-mode reasoning with isolation judgments. SARA improves consistency broadly, transfers without retraining, and supports a broader self-distillation principle for capabilities under complex conditions.

  • Rubric interference is a systematic failure mode in multi-rubric LLM evaluation, addressed by SARA’s alignment of joint-mode reasoning with the model’s isolation judgments.
  • SARA improves consistency across datasets, model families, and scoring formats, transfers across domains without retraining, and selectively suppresses cross-rubric information flow while preserving within-rubric coherence.
  • The results suggest that self-distillation from a simpler setting can close capability gaps under complex conditions without external supervision.

Limitations

SARA has limitations in thinking-mode models and in its assumption that isolated rubric judgments are interference-free anchors. These limitations motivate better format supervision and methods that preserve helpful inter-rubric context while suppressing harmful interference.

  • Thinking-mode models: SARA yields smaller preliminary gains in thinking-mode models because segment extraction is less precise without predictable rubric-analysis-verdict structure.Stronger thinking-mode performance may require format supervision, such as explicit rubric delimiters within the thinking block.
  • Isolation as anchor: ≥0.98 self-agreement across all models supports isolation judgments as anchors, but isolation is not infallible and co-evaluating related rubrics may sometimes improve judgment quality.SARA does not distinguish beneficial context from harmful interference.

Ethics Statement … A.3 Aggregate-Level Metrics

The paper recommends using SARA-trained judges alongside human evaluation in high-stakes settings and defines consistency metrics at rubric, sample, and aggregate levels. These metrics compare verdicts for the same conversation–rubric pair across evaluation conditions using binary- and scored-format measures.

  • Ethics Statement: SARA-trained models should complement rather than replace human evaluation because greater consistency does not guarantee correctness, especially in high-stakes domains.The datasets and models are publicly available under permissive licenses, and no personally identifiable information is collected or processed.
  • A Detailed Metric Definitions: The evaluation compares verdicts for the same conversation–rubric pair across two conditions, such as isolated versus joint evaluation or different rubric permutations.The same core metrics are used for binary and scored formats.
  • A.1 Rubric-Level Metrics: Agreement (Agr) is the fraction of rubrics receiving identical verdicts; for scored verdicts, it is exact score match.For binary verdicts, Agreement is equivalent to classification accuracy between two annotators and, for scored verdicts, to Exact Agreement Rate.
  • A.1 Rubric-Level Metrics: Cohen’s κ corrects agreement for chance, using unweighted κ for binary verdicts and quadratic-weighted κ for scored verdicts.Quadratic weighting penalizes larger deviations more heavily, while both variants are reported under κ.
  • A.1 Rubric-Level Metrics: MAD measures average per-rubric deviation, equaling 1 −Agr for binary verdicts and capturing typical disagreement magnitude for scored verdicts.RMSE is more sensitive to large deviations than MAD and is reported in appendix tables for scored formats.
  • A.1 Rubric-Level Metrics: Pearson Correlation (ρ) measures whether two conditions preserve the relative ordering of scores.It is defined as the linear correlation between v(A) and v(B).
  • A.2 Sample-Level Metric: Exact Match (EM) is the fraction of samples where all rubrics agree, making it the strictest metric because one rubric disagreement causes failure.EM is therefore a sample-level rather than rubric-level measure.
  • A.3 Aggregate-Level Metrics: For scored formats, TotalEM measures identical total scores, Total MAE measures mean absolute error of total scores, and Total Pearson (ρT) measures preservation of sample-level ranking.These aggregate-level metrics operate on total score per sample T (X).

A.4 Metric Selection … E Supplementary Metrics for Main Experiment

The supplementary sections specify metric applicability, evaluation prompts, and expanded robustness and agreement analyses. They also document dataset-specific constraints and the use of isolation versus joint rubric evaluation.

  • A.4 Metric Selection: FLASK’s fixed 3-rubric format does not support Con-K or Subsetting.This constraint determines which metric and operation analyses apply to FLASK.
  • B Evaluation Prompts: Joint-mode prompts evaluate responses against multiple rubrics, while isolation mode uses the same prompt with a single-rubric list.The FLASK prompt follows the ResearchQA format but adapts its scoring scale to 1–5.
  • B Evaluation Prompts: Binary HealthBench evaluation determines whether each rubric criterion is met and outputs the indices of met rubrics.The prompt separately analyzes each rubric and emits Met/Not Met verdicts before listing met rubric indices.
  • C Consistency-at-K Full Results: FLASK is omitted from Consistency-at-K results because its fixed 3-rubric format does not support variable-K evaluation.Tables 9 and 10 provide the full results underlying Figure 3 for the remaining applicable settings.
  • D Shuffle Invariance and Noise Robustness Full Results: Figure 6 summarizes shuffle invariance and noise robustness, with full numerical results reported in Tables 11 and 12.The section therefore supplements the main robustness summary with complete tabulated results.
  • E Supplementary Metrics for Main Experiment: Table 13 reports MAD, RMSE, and Pearson ρ for rubric-level consistency, while Table 14 reports aggregate metrics from per-sample total scores.These supplementary metrics cover scored datasets in the main experiment and compare isolation with joint mode.

F Hyperparameter Sensitivity … I.3 Training Adjustments

The paper finds that SARA depends on structurally decoupled distillation, while remaining robust to moderate hyperparameter changes and increasing output length with rubric count. Adapting SARA to thinking mode requires heuristic segment extraction and a lower preservation weight because reasoning contains substantial reflection and summary content.

  • F Hyperparameter Sensitivity: EMA decay=1 drops EM from .52 to .48, while performance remains robust across decay ∈[.99, .999].The result confirms that the teacher must be decoupled from the student to provide a stable distillation target.
  • F Hyperparameter Sensitivity: EM varies by only .02 across βd ∈[0.1, 0.7], but βd=0.9 causes EM to collapse to .38.The JSD interpolation weight is therefore robust over a moderate range but fails at a high value.
  • F Hyperparameter Sensitivity: α = 0 causes structural outputs to disappear within tens of training steps, while even α = 0.1 makes a difference.The preservation coefficient is described as a binary structural requirement rather than a continuously tunable dial.
  • G Output Token Statistics: SARA-trained models produce longer joint-mode outputs because they provide more detailed per-rubric analyses, with increases proportional to average rubric count.HealthBench, averaging 11.5 rubrics, has the largest absolute increase, while FLASK, with 3 rubrics, has the smallest.
  • H Case Study: Full Outputs: In the case study, the assistant identifies the blank’s answer as business ethics management after reasoning about managing ethical issues through policies, practices, and programs.The complete joint-mode output is reported for the fill-in-the-blank case in Table 7.
  • I.1 Differences from Instruct Mode: Thinking mode moves reasoning into a hidden thinking block and exposes only a final score line, eliminating enforced structured segment boundaries.Rubric analyses may be interleaved with reflection, planning, and self-correction.
  • I.2 Segment Extraction: The thinking-mode extractor assigns each rubric the first paragraph mentioning “Rubric k” and assigns rubric-free paragraphs to the structure mask.This paragraph-level heuristic is less precise because paragraphs may cover multiple rubrics and reasoning may span non-contiguous paragraphs.
  • I.3 Training Adjustments: The preservation coefficient is reduced from α = 0.4 in instruct mode to α = 0.2 in thinking mode.Substantial reflection and summary content makes a high preservation weight overly constraining, which would reduce the distillation signal available to correct interference.

I.4 Preliminary Results · I.5 Discussion · J Declaration of AI Assistant Usage

Thinking-mode SARA improves over the base model but substantially trails instruct-mode SARA, revealing that effectiveness depends on segment extraction quality and reasoning entanglement. The paper also discloses limited AI assistance while retaining human ownership of the scientific work.

  • I.4 Preliminary Results: Thinking-mode SARA improves over the base model but substantially underperforms instruct-mode SARA on ResearchQA.The reported metrics are EM .23→.29 and κ .658→.771 for the base-model comparison, versus instruct-mode SARA at EM .52 and κ .879.
  • I.4 Preliminary Results: Two factors explain the gap: noisy paragraph-level segment boundaries dilute the distillation signal, while entangled thinking interleaves or revisits multiple rubrics.Misaligned segments can align incorrect tokens with isolation anchors, and interleaved reasoning makes per-rubric correction harder.
  • I.5 Discussion: SARA’s isolation-anchor principle applies across output formats, but its effectiveness depends heavily on segment extraction quality.The discussion identifies explicit rubric delimiters in thinking blocks and attention-based attribution as open directions.
  • I.5 Discussion: Improving thinking-mode parsing remains an open direction, including explicit rubric delimiters or attention-based attribution methods.These methods are proposed to make rubric-specific reasoning easier to extract and align.
  • J Declaration of AI Assistant Usage: Claude was used strictly for writing refinement and routine boilerplate or infrastructure coding assistance.The disclosed uses include grammar, style, manuscript refinement, data-processing scripts, and execution modules.
  • J Declaration of AI Assistant Usage: Human authors retained sole ownership of the core algorithms, distillation losses, pipeline code, scientific ideas, formulations, experimental designs, and data interpretations.They also manually created all primary visualizations, plots, and tables without AI generation.
  • J Declaration of AI Assistant Usage: GPT-4.1 served strictly as an automated reference baseline and judge, not as an interactive research or writing assistant.Its role was confined to the evaluation framework described in Section 5.
Loading 2608.14684v1…