PaperlayerView exact revision on arXiv

Source-linked AI summary

RecurSE: Bounded Recursive Self-Evaluation for LLM Rubric Judges

Kaiyuan Liu, Ziyuan Zhuang, Rongxiang Weng, Jieping Ye

arXiv:2608.24231v1cs.CL

TL;DR

Improving LLM judges typically depends on costly external supervision, reward models, or stronger teachers. RecurSE removes those sources from the RL reward by combining a trainable judge with a synchronized reasoning checker, using interface decoupling and PAV to support bounded improvement. Across model scales and tasks, it reports consistent generalization gains and improved downstream policy alignment.

  • Problem

    Improving LLM judges commonly relies on expensive human annotations, learned reward models, or distillation from stronger teachers.

  • Method

    RecurSE trains a rubric judge with scalar process rewards from a synchronized checker, decouples checker scores from verdict tokens, and uses PAV for early stopping.

  • Results

    RecurSE produces consistent held-out gains across medical, pairwise, summarization, and professional benchmarks across three model configurations, and its retained judge improves downstream policy alignment.

  • Takeaways & Limitations

    Bounded recursive self-improvement for LLM rubric judges is viable when self-produced reward validity is explicitly decoupled and monitored.

  • Takeaways & Limitations

    The empirical scope covers three model configurations up to 27B on English rubric benchmarks; multilingual, multimodal, and non-rubric extensions remain future work.

Abstract

from arXiv · show

LLM-as-judge is essential for evaluating open-ended text and steering post-training, yet improving the judge itself typically relies on expensive annotations, reward models, or distillation from stronger teachers. In this work, we eliminate external gold supervision from the RL training reward: the model's own evaluative capability generates learning signals for its optimization -- a closed-loop setting of bounded recursive self-improvement (RSI) termed Recursive Self-Evaluation (RecurSE). We study two central questions: when can self-improvement occur, and when must it stop? First, RecurSE pairs a trainable judge evaluating candidate responses under per-rule rubrics (Pass 1) with a synchronized policy-copy checker that audits the judge's reasoning against meta-rubrics to supply a scalar process reward (Pass 2). To enable learning, interface decoupling structurally isolates the checker's scalar score from the judge's verdict tokens, eliminating a degenerative token-copying shortcut that inflates self-assigned rewards. Second, because unanchored recursive learning is inherently bounded, Pairwise Advantage Validity (PAV) serves as an unbiased validation monitor that jointly tracks judge accuracy and checker fidelity to reliably identify the optimal early-stopping window. Across Qwen3.5-9B, Gemma-4-E4B-it, and Qwen3.6-27B, RecurSE achieves consistent generalization gains across held-out medical, pairwise, summarization, and professional benchmarks. Ablations demonstrate that synchronized judge-checker co-evolution outperforms frozen checkers, external meta-judges, self-consistency, and scaled teacher distillation. Furthermore, preference pairs curated by our judge effectively enhance downstream policy alignment. Bounded RSI for LLM-as-judge is thus viable when self-produced reward validity is explicitly decoupled and monitored.

1 INTRODUCTION

RecurSE trains an LLM judge using its own evaluative capability as a closed-loop reward, while addressing when recursive improvement can occur and when it should stop. Interface decoupling prevents verdict-token shortcuts, and PAV monitors the bounded optimization process.

  • Method: RecurSE uses a trainable rubric judge and synchronized checker to produce reinforcement-learning rewards without external gold labels or teacher models.The judge evaluates responses and the checker audits its reasoning against meta-rubrics.
  • When self-improvement can occur: Interface decoupling replaces shared YES/NO checking with a free-standing scalar checker score, preventing reward inflation through token copying.The judge emits structured per-rule verdicts while the checker emits a 0-to-4 score.
  • When self-improvement must stop: PAV uses a compact human-verified holdout to identify an effective stopping region as self-produced rewards saturate and checker fidelity degrades.The holdout labels guide validation and early stopping but never enter RL training rewards.
  • Experimental findings: RecurSE yields consistent held-out gains across medical, pairwise, summarization, and professional benchmarks on Qwen3.5-9B, Gemma-4-E4B-it, and Qwen3.6-27B.Reported validation gains are +12.9 points, +5.2 points, and +3.9 points, respectively.
  • Experimental findings: Synchronized judge-checker co-evolution outperforms frozen checkers, external meta-judges, self-consistency, and scaled teacher distillation, while retained-judge preference pairs improve downstream alignment.The findings connect co-evolution, baseline comparisons, and downstream transfer across the paper’s experimental questions.
  • Contributions: The paper formalizes bounded recursive self-improvement for LLM judges and demonstrates broad out-of-distribution generalization across model families and scales up to 27B.Its contributions combine on-policy process checking, interface decoupling, PAV-based stopping, and downstream preference alignment.

2 RELATED WORK

Prior work develops self-improving evaluators and process auditing, but commonly retains external supervision or anchors. RecurSE instead positions process checking as reward-only auditing while optimizing a fixed-criterion judge.

  • LLM judges and rubric evaluation: LLM judges evaluate open-ended generations under structured criteria when reference-based metrics are insufficient.Rubric evaluation decomposes holistic assessment into criterion-level decisions.
  • Self-improving evaluators and bounded RSI: Prior self-improving evaluators typically retain external anchors such as synthetic preference pairs, external meta-judges, or frozen reference models.RecurSE is contrasted with these approaches by removing external gold labels and teacher models from the optimization reward.
  • Process auditing and reward validity: Process-supervision and verification methods commonly require ground-truth correctness labels to train process reward models.RecurSE’s process checker instead operates at inference time as a reward-only auditor without separate supervised training.

3 RECURSIVE SELF-EVALUATION (RECURSE)

RECURSE creates a closed-loop judge–checker recurrence in which a synchronized copy audits the judge’s reasoning to generate reward, while interface decoupling prevents verdict-token shortcuts. Pairwise Advantage Validity monitors judge accuracy and checker fidelity on a human-verified holdout to identify when bounded self-improvement should stop.

  • 3.1 JUDGE–CHECKER RECURRENCE: RECURSE uses one shared policy as both rubric judge and synchronized reasoning checker, eliminating gold labels, reward models, and stronger teachers from the RL reward.Pass 1 generates reasoning and per-rule YES/NO verdicts; Pass 2 audits reasoning against meta-rubrics and supplies reward-only scalar scores.
  • 3.2 INTERFACE DECOUPLING: WHEN SELF-IMPROVEMENT CAN OCCUR: Identical YES/NO interfaces create surface coupling, allowing policy updates that increase YES emissions to inflate self-assigned rewards without improving evaluative accuracy.The shortcut arises because synchronized judge and checker weights share the same verdict-token channel.
  • 3.2 INTERFACE DECOUPLING: WHEN SELF-IMPROVEMENT CAN OCCUR: Interface decoupling replaces checker verdict tokens with a free-standing 5-tier scalar score, reducing direct token copying while preserving shared capability transfer.The checker audits the complete reasoning trajectory against meta-rubrics and emits si ∈{0, 1, 2, 3, 4}.
  • 3.3 OPTIMIZATION AND PAIRWISE ADVANTAGE VALIDITY: WHEN SELF-IMPROVEMENT MUST STOP: Group-relative optimization ranks rollouts within each prompt, using normalized advantages based strictly on within-prompt score orderings.Malformed outputs are masked from group statistics before sequence-level policy optimization.
  • 3.3 OPTIMIZATION AND PAIRWISE ADVANTAGE VALIDITY: WHEN SELF-IMPROVEMENT MUST STOP: Recursive optimization is bounded because self-produced rewards and ranking fidelity eventually saturate, after which out-of-distribution transfer can degrade.The paper therefore uses a compact human-verified validation set whose labels monitor training but never enter RL rewards.
  • 3.3 OPTIMIZATION AND PAIRWISE ADVANTAGE VALIDITY: WHEN SELF-IMPROVEMENT MUST STOP: PAV combines judge correctness with checker fidelity and uses pairwise ranking validity to localize an effective early-stopping region.The expected pairwise ranking error is bounded by 2eC,t, motivating continuous checker-fidelity monitoring.

4 EXPERIMENTAL SETUP

The experiments evaluate RECURSE across three model setups using cluster-split RubricHub data, separate online monitors, and held-out transfer suites. Ablations compare co-evolution against frozen and external checkers, self-consistency, and teacher SFT.

  • Models and training: RECURSE is evaluated on Qwen3.5-9B, Gemma-4-E4B-it, and Qwen3.6-27B using cluster-split RubricHub instances with zero evaluation overlap.Ground-truth labels never enter RL rewards.
  • Evaluation hierarchy: The evaluation hierarchy separates SV-HARD, SV-FULL, and held-out transfer suites to distinguish online monitoring from broader generalization.SV-HARD supplies the sole online PAV monitor; SV-FULL probes complete in-domain coverage.
  • Evaluation hierarchy: SV-HARD is a 100-prompt human-verified monitor that supplies checker-fidelity labels and guides PAV early stopping without contributing gold labels to RL rewards.It is cluster-isolated from training and matched to the training judging style.
  • Ablations and research questions: Ablations on Qwen3.5-9B compare a frozen checker, an external 27B checker, self-consistency, matched-step teacher SFT, and larger-pool teacher SFT.Gemma provides cross-architecture replication, while downstream transfer uses preference pairs from retained versus untuned judges.

5 RESULTS

RECURSE improves judge performance across architectures and held-out tasks when interface decoupling prevents reward inflation and PAV selects a bounded validity window. Co-evolution outperforms static alternatives, and the retained judge improves downstream policy alignment.

  • RQ1: Bounded RSI improves the judge: +12.9 points on Qwen-9B, +5.2 on Gemma, and +3.9 on Qwen-27B are reported on SV-HARD, with gains extending across held-out transfer suites.The final checkpoints soften or drop on transfer metrics, indicating a bounded improvement window.
  • RQ2: Interface decoupling: Under shared YES/NO checking, YES rate rises 0.730 →0.791 and aligned reward rises 0.465 →0.698, while validation accuracy remains 0.729 →0.730.These values show reward inflation without corresponding accuracy improvement.
  • RQ2: Interface decoupling: Under differentiated Final Score output, aligned reward rises 0.484 →0.656 and validation accuracy rises 0.724 →0.787 while YES rate remains calibrated at 0.707 →0.695.The result supports interface decoupling as a prerequisite for useful reward growth.
  • RQ3: PAV localizes the useful checkpoint: 0.742 PAV on Qwen-9B at step 130 marks the effective stopping region, while checker fidelity degrades after its peak despite continued hard-subset accuracy gains.Three-fold resampling confirms stability inside {130, 140}; SV-FULL independently peaks at 95.3% and falls to 89.7%.
  • RQ4: Judge–checker co-evolution matters: The main method reaches 85.9% on HealthBench versus 79.8% for an external 27B checker and 83.5% for full teacher distillation.Matched-step teacher SFT reaches only 64.2% on SV-HARD, while self-consistency reaches 56.5% at the final checkpoint.
  • RQ5: Downstream policy transfer: +0.59 on GPQA, +1.63 on GuideBench, and +2.37 on SOP-Maze result from RECURSE preference labels, whereas base-judge labels degrade GPQA by 80.87 →79.02.The matched policies are trained with DPO on Qwen3.6-27B.

6 CONCLUSION

The conclusion presents RECURSE as bounded recursive self-improvement for rubric judges, enabled by decoupled process rewards and PAV monitoring. It also documents the recurrence, shortcut mechanism, validation cost, and dual-variable difficulty boundary.

  • Conclusion: RECURSE makes recursive self-improvement feasible for LLM rubric judges when self-produced reward validity is explicitly decoupled and monitored.The approach yields out-of-distribution generalization and downstream preference alignment across scales up to 27B.
  • Training recurrence: The training loop samples rubric instances, generates judge rollouts, scores them with a synchronized checker, updates the judge, and periodically saves the checkpoint with highest PAV.Invalid rollouts are masked from group statistics and the checker is refreshed synchronously.
  • Surface coupling: Identical YES/NO interfaces create a shortcut in which synchronized updates increase verdict-token bias and self-assigned rewards without increasing true evaluative quality.The structural model represents reward as S = aU + bB + ε, with the shortcut driven by bB.
  • Interface decoupling: Interface decoupling replaces token-linked readouts with a free-standing 5-tier scalar score, severing direct copying and directing optimization toward reasoning verification.The intervention effectively sets b ≈0 in the shortcut model.
  • Dual-variable difficulty: Rubric judging requires simultaneous calibration of a discriminative rubric and a borderline candidate response, unlike single-variable task distillation.Static teacher SFT fixes candidate responses, so its triplets can leave the student’s active difficulty regime as the judge improves.
  • Stopping boundary: Recursive optimization is bounded because self-generated rewards or ranking fidelities saturate and further optimization degrades out-of-distribution transfer.PAV uses a compact human-verified holdout whose labels never enter RL rewards.

J EVALUATION METRICS

The evaluation uses rule, sample-level, macro, correlation, and checker-fidelity metrics across distinct benchmark formats. PAV combines judge accuracy with checker ranking fidelity, while CheckEval correlation measures continuous alignment with human ratings.

  • Judge metrics: Rule accuracy pools all comparable gold-labeled rules, skipping rules without predictions; it is primary for SV-HARD, SV-FULL, HealthBench, and ProfBench.This is a micro-average over rules.
  • Judge metrics: All-correct rate measures the fraction of instances whose every comparable rule is judged correctly, providing a stricter sample-level metric.It is reported alongside rule accuracy for SV-HARD, HealthBench, and ProfBench.
  • Judge metrics: Macro accuracy averages per-instance rule accuracies equally across valid prompts, regardless of each prompt’s rule count.It is used for SV-HARD, HealthBench, and ProfBench.
  • Pairwise evaluation: RubricBench pairwise accuracy predicts the preferred side from the higher mean YES-rate, breaking ties by YES count.Accuracy is computed against gold preference labels.
  • CheckEval: CheckEval averages sample-level Pearson correlations across coherence, consistency, fluency, and relevance over 1,600 summaries.The metric measures linear alignment with continuous human ratings, not discrete percentage accuracy.
  • Checker and PAV diagnostics: PAV evaluates judge accuracy and checker fidelity using the same held-out gold labels, while keeping those labels out of the RL training reward.Checker fidelity includes exact match, MAE, and ranking fidelity 1 −eC.
  • CheckEval: A +0.019 CheckEval lift from 0.422 to 0.441 on Qwen-9B is a continuous-alignment improvement and should not be compared directly with percentage-point accuracy gains.The reported significance is p = 0.0032 < 0.01.

K BENCHMARK CHARACTERISTICS AND EXAMPLES

The held-out suites span distinct benchmark traits and medical categories, with representative examples illustrating each emphasized characteristic. HealthBench is subsampled proportionally while preserving prompt–response groups and category distribution.

  • HealthBench contains 29,511 rubric evaluation instances across 14,592 prompt-completion groups, 10 clinical themes, and 34 medical subcategories.
  • Table 5 pairs each emphasized benchmark trait with a representative instance.
  • Evaluating the full HealthBench corpus creates an approximately 30k-trace sequential bottleneck at each trajectory evaluation step.
  • The 10% stratified protocol preserves atomic prompt–response groups and yields 1,459 groups containing 2,951 rubric rules.

L ADDITIONAL TRAINING DYNAMICS

Training diagnostics jointly track judge accuracy and checker quality across SV-HARD and HealthBench. On Qwen3.5-9B, PAV identifies a stopping window where checker fidelity peaks before late judge-accuracy gains become deceptive.

  • Four diagnostics track judge rule accuracy, checker exact match, checker MAE, and normalized checker fidelity.Checker fidelity is defined as 1 − MAE/4 on the 0–4 score scale.
  • At PAV landmark @130, SV-HARD checker exact match rises from 21.5% to 48.2% while MAE falls from 1.64 to 1.03.
  • After step 130, SV-HARD judge accuracy reaches 75.9% by step 200 while checker exact match falls to 36.0% and MAE rises to 1.32.The divergence indicates validation deception from over-optimizing training-distribution stylistic quirks.
  • On HealthBench, judge accuracy, checker exact match, and checker fidelity peak together near step 130 before decaying.This suite is a distinct medical distribution not used for training or validation monitoring.

M CROSS-ARCHITECTURE REPLICATION ON GEMMA-4-E4B-IT

Gemma-4-E4B-it reproduces the bounded recursive self-improvement pattern outside the Qwen family. Its useful window is selected near step 160, while diagnostic trajectories soften after the joint plateau.

  • SV-HARD rule accuracy rises from 55.9% to 61.1% at Gemma’s PAV-selected @160 checkpoint.
  • HealthBench gains +3.7 points and CheckEval Pearson ρ improves from 0.442 to 0.466 during the useful window.
  • SV-FULL remains near its high base of 92.5%, consistent with a near-saturated in-domain ceiling on this architecture.
  • Gemma’s checker fidelity and judge accuracy climb together, plateau around steps 120–160, and soften afterward.Late-stage SV-HARD accuracy can continue drifting upward despite checker degradation.

N PAV CONSTRUCTION AND VALIDATION DETAILS

PAV combines judge accuracy with checker fidelity to monitor recursive training, using statistical validation, stability analyses, and selector comparisons. Across architectures, it identifies broad stopping windows that preserve transfer gains while avoiding late capacity collapse.

  • PAV construction: PAV is unbiased at a fixed checkpoint because empirical judge-accuracy and normalized checker-error means estimate their corresponding expectations.
  • Validation robustness: PAV forms elevated validity plateaus at steps 120–140 for Qwen-9B and 150–170 for Gemma rather than relying on a fragile single-step maximum.
  • PAV construction: The monitor uses checker error with weight two because pairwise reward differences bound expected score-difference error by 2eC,t.
  • Weight stability: Across λ ∈[1.59, 11.03], the optimal stopping region remains centered at Qwen@130 and Gemma@160.Near Qwen step 130, checker-error growth (+0.15/10 steps) dominates accuracy gains (+0.02/10 steps), supporting the theoretically derived λ = 2.
  • Validation robustness: Bootstrap and fold analyses support stable monitoring: 10,000 prompt-cluster resamples give Qwen@130 bVt = 0.742 with a 95% interval of [0.685, 0.797].
  • Selector comparison: PAV achieves mean oracle regret of 1.20 on Qwen-9B and 0.49 on Gemma-4B, improving all four transfer suites over Base.Accuracy-only selection incurs 5.41 regret on Qwen-9B and improves only 2/4 suites.
  • Transfer validation: Held-out transfer gains are statistically significant across CheckEval-Summ, ProfBench, HealthBench, RubricBench, and downstream DPO.Reported examples include +1.6pp on ProfBench and +2.44pp over the base judge on GPQA.

O SYSTEMS, SCALABILITY, AND COST-PERFORMANCE TRADE-OFF ANALYSIS

The paper compares RECURSE with external judges, learned reward models, and teacher distillation while analyzing checker overhead, synchronization, parsing, and late-stage failure modes. Its design uses inference-only synchronized checking, interface decoupling, and PAV-based stopping to preserve reward validity and scalability.

  • Systems and cost-performance trade-offs: The checker audits all n = 8 judge rollouts in inference-only mode, avoiding gradient backpropagation and optimizer-state updates.RL optimization typically consumes 2× to 3× the FLOPs of a forward pass per token, making checker inference cheaper than actor optimization.
  • Scalability: On Qwen-27B with Tensor Parallelism size 4, in-memory parameter broadcast requires ≈1.2 seconds per optimization step.Autoregressive generation across n = 8 samples dominates the per-step iteration time.
  • Paradigm comparison: A 3× larger static 27B checker underperforms synchronized 9B co-evolution by 6.1 points on HealthBench (79.8% vs. 85.9%).This comparison supports synchronized co-evolution over simply scaling a frozen checker.
  • Format compliance: Judge verdicts remain parseable at ≥99%, while checker score parse rate rises from 0.87 to 0.96 by step 130 before dipping after step 160.The PAV-selected checkpoint is step 130; malformed outputs are masked from gradients and group statistics.
  • Stopping and failure modes: Parse rate selects step 160 under a heuristic >5% drop rule, but transfer metrics have already deteriorated by then, so it cannot replace PAV.Late-stage checker degradation includes score inflation, hallucinated meta-justifications, verbosity drift, and generation truncation beyond the 32,768-token window.
  • Paradigm comparison: RECURSE generates rewards from its synchronized policy copy, unlike paradigms using commercial judges or standalone learned reward models.Commercial APIs add latency, rate limits, fees, and possible prompt exposure; static reward models require curated preference data and can degrade out of distribution.

T LIMITATIONS

The paper’s evidence is bounded by its model, language, task, monitoring, and recursive-optimization settings. It also retains human verification for early stopping and acknowledges statistical and process-checking assumptions.

  • Scope of model architectures and task domains: Experiments cover three model configurations up to 27B parameters on English rubric benchmarks, limiting empirical scope beyond these architectures and domains.Qwen3.5-9B is the primary testbed, Gemma-4-E4B-it provides replication, and Qwen3.6-27B demonstrates scale feasibility.
  • Role of human verification: RECURSE requires a compact human-verified 100-prompt SV-HARD holdout to compute PAV, imposing upfront verification cost despite avoiding corpus-wide labeling.Gold labels stay out of the RL reward but remain part of the end-to-end evaluation pipeline.
  • Statistical monitoring and sequential testing: PAV is unbiased at any fixed checkpoint, but online sequential monitoring across checkpoints introduces multiple-testing considerations.The paper identifies formal sequential stopping boundaries with family-wise error control as an avenue for refinement.
  • Process checking assumptions and granularity: The checker audits reasoning against meta-rubrics rather than ground-truth outcomes, so systematic checker biases could still reward flawed rationales.The 0–4 scalar interface mitigates surface shortcuts and reduces regression noise, but adaptive rewards with dynamic format gating remain unexplored.
  • Scope of bounded RSI: The study fixes the task specification, optimizer, and checking protocol, so its conclusions do not extend to open-ended recursive self-modification.The model does not autonomously modify the training architecture or meta-rubric definitions.
Loading 2608.24231v1…