PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Evaluation Is Already There: Eliciting Latent Judge Calibration in Base LLMs with Minimal Data

XiuYu Zhang, Yi Shan, Junfeng Fang, Zhenkai Liang

arXiv:2606.05122v1cs.CL

TL;DR

Can a base language model predict how an external judge will score its open-ended responses, and how much of this ability exists before targeted training? The paper introduces Self-Evaluation Elicitation, which surfaces this ability with minimal data and improves held-out calibration across three benchmarks while preserving answer quality.

  • Problem

    It remains unclear whether base models can predict external judges’ multi-attribute scores for open-ended responses without verifiable answers or targeted training.

  • Method

    Self-Evaluation Elicitation alternates a brief answer-improving reinforcement-learning phase with masked distillation that trains only self-evaluation tokens.

  • Results

    Using 160 unique examples, roughly 31× fewer than the reinforcement-learning baseline, SEE improves held-out calibration across three benchmarks while preserving answer quality.

  • Takeaways & Limitations

    The findings reframe judge-aligned self-evaluation as elicitation of latent capability rather than acquisition of a new capability.

  • Takeaways & Limitations

    Evidence comes from a single base model and judge family, without human evaluation, so the method’s alignment with human preferences remains untested.

Abstract

from arXiv · show

Large language models are increasingly evaluated by other models, raising a natural question: can a model predict how a judge will score its own output? We find that the ability is largely present before any targeted training: prompted few-shot, a base model already predicts an external judge's multi-attribute quality scores on open-ended responses well above chance across three benchmarks. We introduce Self-Evaluation Elicitation (SEE), a method that surfaces this latent ability through a short cycle comprising a calibration-coupled reinforcement learning phase that improves the answer and predicts the judge, followed by a masked distillation phase that sharpens the prediction while leaving the answer untouched. From 160 unique examples, roughly 31x fewer than a reinforcement learning baseline, SEE improves held-out calibration across three benchmarks while preserving answer quality. The elicited self-evaluation is sharply localized within the model's own token distribution and stable across judges it was never trained against, indicating a transferable notion of quality rather than a single judge's preference. These results reframe judge-aligned self-evaluation as a problem of elicitation rather than acquisition.

1 Introduction

LLMs are increasingly used as judges, motivating models that can anticipate external scores for their own outputs. The paper argues this capability is already latent in base models and can be elicited with minimal-data training that preserves answer quality.

  • Motivation: Self-evaluation could let models rerank samples, defer low-scoring outputs, or escalate difficult prompts without querying the judge at inference time.These uses follow from anticipating how an output will be judged.
  • Motivation and finding: Base LLMs can already approximate an external judge’s multi-attribute scores when prompted few-shot in the paper’s scoring format.This reframes self-evaluation as elicitation rather than acquisition.
  • Self-Evaluation Elicitation: The method alternates a brief RL phase that improves answers with masked distillation on judge scores restricted to self-evaluation tokens.The masked phase leaves the answer unchanged while re-anchoring score predictions to the judge on the model’s current output distribution.
  • Results: After 15 cycles, self-prediction error decreases by 0.25 ∼0.66 mean absolute error (MAE) across all evaluated benchmarks.The answer is improved during RL, while calibration is sharpened without disturbing it during masked distillation.
  • Data efficiency: 160 unique training examples—roughly 31× fewer than the reinforcement learning baseline—improve held-out calibration across three benchmarks while preserving answer quality.The approach replaces a usual large training run with a short alternating cycle.
  • Robustness: The judge’s score falls within the model’s top-5 predicted tokens at high rates, and gains persist with held-out judges rather than the training judge.This supports robust, transferable elicited self-evaluation.

2 Related Work

Related work shows that self-evaluation can serve as an RL signal and that post-training often elicits capabilities already present in base models. This paper extends that line to multi-attribute predictions of external judges on open-ended responses, while drawing on alternating RL/SFT methods and LLM-based evaluation.

  • Self-evaluation as an RL signal: RLCR trains models to emit calibrated correctness probabilities using a Brier-score confidence term, and the construction extends to any bounded proper scoring rule.This work instead predicts an external judge’s scores across several quality attributes on open-ended prompts without verifiable answers.
  • Eliciting latent capability: Prior studies report that post-training can surface base-model abilities in correctness estimation, instruction-following quality, reasoning, and safety rather than installing entirely new capabilities.The cited evidence spans task accuracy, self-knowledge about correctness, and safety.
  • Eliciting latent capability: Multi-attribute self-evaluation of how an external judge rates open-ended responses had not been examined; this paper provides that measurement and builds SEE on it.The gap concerns judge-aligned predictions across multiple attributes, rather than self-knowledge about answer correctness alone.
  • Alternating supervised and reinforcement learning: ReST, ReSTEM, RAFT, and STaR alternate reinforcement learning with supervised fine-tuning on high-scoring samples from the current policy.These methods improve models by generating rollouts, selecting survivors under a reward, and fine-tuning on them.
  • LLM judges and multi-attribute reward: LLM judges commonly evaluate outputs, while datasets such as HelpSteer2 score helpfulness, correctness, coherence, complexity, and verbosity separately rather than using one scalar.This paper adopts the multi-attribute view but turns it inward toward predicting an external judge.

3 Method

Self-Evaluation Elicitation (SEE) surfaces a model’s latent ability to predict external judge scores while improving its open-ended responses. It alternates Calibration-Coupled RL with Masked Judge Distillation in a single model.

  • Method: SEE targets open-ended responses scored by a multi-attribute judge, eliciting predictions of the judge’s scores while improving the responses themselves.The method alternates two phases over a single model.
  • Method: Calibration-Coupled RL rewards both response quality and calibration between the model’s self-scores and the external judge’s scores.The reward combines a quality term with a calibration term when the self-evaluation block is well-formed.
  • Method: The model emits each response followed by a [SELF_EVAL]-delimited block containing integer 0–9 scores for helpfulness, correctness, coherence, complexity, and verbosity.The external judge scores the same response on the same five attributes.
  • Method: A self-evaluation block is well-formed only when it parses to integer scores in [0, 9] for all five attributes, with calibration measured by MAE(s, j).The quality and calibration terms are weighted by wq and wc.
  • Method: SEE proceeds through Phase 1 Calibration-Coupled RL and Phase 2 Masked Judge Distillation.These phases form the method’s two-stage alternation.

I. User request

The request illustrates SEE’s two-phase design: calibration-coupled reinforcement learning improves responses while eliciting judge-aligned self-scores, and masked distillation sharpens those scores without changing answers. Alternating both phases keeps self-evaluation aligned with the model’s evolving response distribution.

  • Calibration-Coupled RL: Calibration-Coupled RL jointly improves the answer and trains an inline five-attribute self-evaluation against an external judge.Its reward combines answer quality over three evaluative attributes with calibration over all five attributes.
  • Masked Judge Distillation: 5×5 = 25 score-attribute cells support stratified rollout selection, improving coverage beyond the mid-range scores that dominate open-ended responses.The grid spans five attributes and five score bins, including the extremes.
  • Masked Judge Distillation: Masked Judge Distillation supervises directly on the judge’s five scores while restricting updates to self-evaluation tokens, preserving the answer distribution.This provides a denser signal than the scalar calibration reward and avoids shifting responses toward the judge’s preferences.
  • SEE cycle: Alternating reinforcement learning with distillation over several cycles re-grounds predictions after each shift in the model’s answer distribution.A single distillation pass would soon describe responses the model no longer produces.

4 Experiments

Experiments show that Qwen3-4B-Base already predicts GPT-5.4’s scores, and SEE elicits stronger calibration and quality with substantially less data than Adapted RLCR. The resulting predictions generalize across judges and are sharply localized in the model’s token distribution.

  • Experimental setup: GPT-5.4 scores five HelpSteer2 attributes, while quality averages helpfulness, correctness, and coherence and calibration compares self-scores with judge scores.Evaluation covers HelpSteer2 validation plus LC AlpacaEval 2.0, Arena-Hard-Auto v2.0, and WildBench v2; only correctly formatted [SELF_EVAL] responses count.
  • Latent self-evaluation: 0.63 calibration on HelpSteer2 validation and 0.50–0.70 across three benchmarks show the untrained base model already predicts judge scores above chance.The base model places the judge’s score within its top five score tokens 77.07% of the time on HelpSteer2 validation.
  • Quality and calibration: 0.7312 calibration from SEE exceeds 0.6752 from Adapted RLCR on HelpSteer2 validation using 160 unique examples versus roughly 31× more unique data.SEE wins the majority of per-sample quality and calibration comparisons against the base model.
  • Quality and calibration: 0.6088 calibration on WildBench v2 for SEE exceeds 0.5040 for the base and 0.5414 for Adapted RLCR, while quality gains remain consistent without trading quality for calibration.SEE is best across response win-rate, quality, and calibration on all three open-ended benchmarks.
  • Sample efficiency: ∼0.8k sample-passes let SEE reach the baseline’s final quality and calibration, roughly 12× fewer than ∼9.6–10k, while unique-example usage is about 31× lower.SEE uses 160 unique examples; Adapted RLCR consumes roughly 5,000 over two epochs.
  • Generalization and localization: SEE > Adapted RLCR > base holds across all four benchmarks for both quality and calibration under Claude Sonnet 4.6 and Gemini 3.1 Flash-Lite.Absolute scores shift by judge, but the ranking and gains persist; SEE’s top-5 accuracy reaches 0.8776 on HelpSteer2 validation and 0.9078 on LC AlpacaEval 2.0.

5 Discussion

A base model already approximates an external judge’s multi-attribute scores, so SEE primarily surfaces and enhances latent judge-aligned quality assessment. Its disjoint updates improve answer quality and self-evaluation together, while the elicited signal may support judge-free decisions that remain untested.

  • Core finding: A base model already approximates an external judge’s multi-attribute scores before targeted training, reframing judge-aligned quality assessment as largely a readout problem.SEE provides a short cycle for surfacing this latent ability rather than installing it.
  • Mechanism: SEE improves answer quality and self-evaluation together because reinforcement learning updates the entire response while distillation updates only the self-evaluation.The disjoint updates let supervised correction sharpen prediction without perturbing the answer produced by reinforcement learning.
  • Calibration: SEE maintains calibration across the judge-score range, whereas the base model degrades at the extremes.This contrast is shown in Figure 4.
  • Potential uses: Elicited self-evaluation could rerank samples, defer on predicted low scores, or escalate hard prompts without a judge in the loop, but these uses remain un demonstrated.The paper identifies evaluating decision-making performance as a natural next step.

6 Conclusion

A base model already largely possesses the ability to predict how an external judge will score its open-ended responses. SEE elicits this ability with minimal data, improving held-out calibration while preserving answer quality and generalizing across unseen judges.

  • 6 Conclusion: SEE elicits latent judge-aligned self-evaluation through Calibration-Coupled RL followed by Masked Judge Distillation.The method uses a short two-phase cycle.
  • 6 Conclusion: Held-out calibration improves across three benchmarks from 160 unique examples while answer quality remains intact.The calibration gains are achieved without degrading the evaluated responses.
  • 6 Conclusion: The elicited self-evaluation is sharply localized in the model’s own distribution and remains stable under judges never seen during training.This stability extends beyond the judges used for elicitation.

Limitations

The evidence is limited to one base model and one family of LLM judges, with no human evaluation. Thus, cross-judge robustness shows independence among LLM judges rather than alignment with human preferences.

  • Scope: The study evaluates only a single base model and a single family of judges.Cross-judge results show gains do not depend on the training judge.
  • External validity: The judges are language models, so cross-judge transfer demonstrates judge-independence among LLM judges rather than human-preference alignment.Claude Sonnet 4.6 and Gemini 3.1 Flash-Lite are both LLM judges.
  • Evaluation basis: No human evaluation was conducted because of resource constraints, and both quality and calibration targets are defined by an LLM judge.The method therefore inherits limitations from its LLM-judge-based targets.

A Training Configuration · B Prompt Templates · C Case Studies

The appendices specify SEE’s two-phase training setup, tightly constrained answer-and-self-evaluation generation, and judge prompts that score five HelpSteer2 attributes on a 0–9 scale. Two LC AlpacaEval 2.0 cases illustrate SEE correcting a confident error and lowering self-assessment for an adequate answer.

  • A Training Configuration: SEE combines a GRPO reinforcement-learning phase with a single supervised distillation epoch over selected rollouts.The distillation phase uses rollouts selected from the buffer.
  • A Training Configuration: SEE and Adapted RLCR share rewards, prompts, judges, rollout counts, and GRPO optimizer settings, differing only in distillation and batch size.Batch sizes are 16 for SEE and 48 for Adapted RLCR; the batch-size effect is not isolated.
  • A Training Configuration: The approximately 31× unique-data gap is far larger than batch size alone could account for.This comparison concerns SEE versus Adapted RLCR.
  • B.1 Self-Evaluation Generation Template: The policy template requires a helpful, safe answer followed by exactly one [SELF_EVAL] block containing integer 0–9 scores for five attributes.The required fields are helpfulness, correctness, coherence, complexity, and verbosity, with strict placement and formatting constraints.
  • B.2 Judge System Prompt: The judge system prompt scores helpfulness, correctness, coherence, complexity, and verbosity using integer scores from 0 to 9.It emphasizes that complexity and verbosity are descriptive properties rather than targets to maximize.
  • B.3 Judge User Prompt: The judge user prompt supplies the conversation, optional human-scored reference responses as calibration anchors, and the target answer without its self-evaluation block.The judge outputs only a JSON object containing the five integer scores.
  • C Case Studies: Two LC AlpacaEval 2.0 cases show SEE correcting a confident error and lowering self-assessment on a merely adequate answer.The tables report the model’s self-evaluation and the judge’s scores, with SEE rows shaded.

C.1 Correcting a Confident Error · C.2 Lowering Confidence on an Adequate Answer

SEE improves self-evaluation in two contrasting cases: it corrects a confident factual error and lowers confidence on an adequate but mediocre answer. In both cases, SEE brings self-assessed quality closer to the judge’s assessment.

  • C.1 Correcting a Confident Error: The base model incorrectly describes the AK-47 as a bullpup rifle while assigning itself high helpfulness and correctness scores.The answer is factually wrong and the model is unaware of its error.
  • C.1 Correcting a Confident Error: SEE answers the rifle question correctly and aligns its self-evaluation closely with the judge on quality attributes.The case illustrates SEE reducing confident hallucination and aligning self-assessment with answer quality.
  • C.1 Correcting a Confident Error: 8 versus 2 and 1: the base model rates its helpfulness and correctness at 8, while the judge assigns scores of 2 and 1.This pattern constitutes a confident error.
  • C.2 Lowering Confidence on an Adequate Answer: The Anki response offers generic troubleshooting advice, including clearing browser cookies and disabling extensions, despite Anki being a desktop application.The judge rates the answer as moderate because the advice is poorly matched to the application.
  • C.2 Lowering Confidence on an Adequate Answer: 8 versus near 5: the base model self-rates helpfulness and correctness at 8, whereas SEE’s self-evaluation is near 5 and closer to the judge.The judge rates both answers as only moderate.
  • C.2 Lowering Confidence on an Adequate Answer: SEE’s Anki answer remains imperfect, but its self-evaluation is close to the judge’s assessment rather than assigning high confidence to a mediocre answer.This case demonstrates the reliability improvement quantified by the aggregate calibration results.

D Training Algorithm

SEE alternates calibration-coupled reinforcement learning with masked judge distillation for C cycles. It trains only on format-valid rollouts and restricts distillation to five self-evaluation score tokens using stratified round-robin targets.

  • D Training Algorithm: SEE alternates Calibration-Coupled RL and Masked Judge Distillation for C cycles.Calibration-Coupled RL optimizes the whole response under Equation 1's reward, while Masked Judge Distillation fine-tunes judge-score predictions.
  • D Training Algorithm: Only format-valid rollouts enter the training buffer.
  • D Training Algorithm: Distillation restricts its loss to five self-evaluation score tokens and uses stratified round-robin target selection.

E Responsible Research Details

The study uses existing benchmark-derived data without collecting user data or recruiting annotators, while acknowledging inherited sensitive content and limited statistical reporting. It provides implementation artifacts and code, but withholds raw datasets and generated experiment records.

  • Data content and privacy: No new user data were collected and no new annotators were recruited; training uses HelpSteer2-derived data and evaluation uses existing public benchmarks.The released package excludes raw datasets, generated outputs, rollout logs, API-key files, and experiment logs.
  • Data content and privacy: Benchmark prompts may contain sensitive, offensive, or identifying content inherited from source artifacts, which the authors do not use to identify users or infer protected attributes.Qualitative examples were manually inspected for obvious identifying information.
  • Data and split statistics: 160 unique HelpSteer2-derived training prompts are reused across 15 cycles for 2,400 total sample-passes, versus roughly 5,000 unique examples and about 10,000 sample-passes for Adapted RLCR.Evaluation covers HelpSteer2 validation, LC AlpacaEval 2.0, Arena-Hard-Auto v2.0, and WildBench v2.
  • Compute and implementation details: Approximately 300 GPUhours cover all reported training and evaluation experiments, using four RTX PRO 6000 GPUs with 96 GB memory each, bf16 precision, VeRL, and vLLM.The base model has 4.0B parameters.
  • Result aggregation: Aggregate results are means over relevant evaluation examples for a single training run, without multi-seed error bars or confidence intervals.The authors identify repeated RL training and repeated LLM-judge evaluation costs as the reason for this limitation.
  • Code availability: The GitHub repository provides the core SEE implementation, including data preparation, Calibration-Coupled RL, rollout collection, score-token SFT construction, and Masked Judge Distillation scripts.AI assistants supported writing, editing, and code/documentation, while authors reviewed and verified the scientific work and final text.
Loading 2606.05122v1…