Source-linked AI summary
Competence, Not Accuracy: A Diagnostic for Reference-Free Judge Gates in Skill Optimization
Chenle Chen, Yangbo Wei, Chao Yao, Shaoqiang Lu, Junhong Qian, Chen Wu, Lei He
TL;DR
Reference-free LLM judges could broaden text-space skill optimization beyond tasks with automatic verifiers, but their usable signal is untested. This paper models judge evaluation as latent solving and probes judges without changing decisions, finding discriminability near chance at the competence floor and usable above it, while benchmark accuracy overstates relevant competence. The result is a pre-deployment screen for judge gates.
Problem
Whether reference-free LLM-judge gates carry usable signal for text-space skill optimization remains untested, limiting evidence for replacing verifiable validation signals.
Method
The paper models a reference-free judge as a latent solver, derives competence-based discriminability predictions, and uses a non-intervening probe during genuine optimization runs.
Results
Discriminability varies by task, collapsing near the competence floor; benchmark accuracy overstates relevant competence, and the screen predicts closed-loop gating error types.
Takeaways & Limitations
The screen is a cheap pre-deployment prerequisite: estimate uncontaminated competence on the target task and decline to gate with judges below the chance floor.
Takeaways & Limitations
The conclusions are preliminary, based on two to three seeds, and the model is scoped to single-answer tasks judged without rubrics or external evidence.
Abstract
from arXiv · showhide
Text-space skill optimization adapts a frozen agent by evolving a natural-language skill document, accepting each candidate through a validation gate. Existing gates rely on verifiable rewards, confining these methods to tasks with an automatic verifier. Replacing the verifier with an LLM-judge gate would lift that restriction, but whether such a gate carries usable signal is untested. We ask a prior question: can we tell, before placing a judge in the loop, whether its scores separate correct from incorrect answers at all? We formalize a reference-free judge as a latent solver -- its verdict rests on agreement with whatever it would itself conclude, so its capacity to evaluate is bounded by its capacity to solve. The model yields a closed-form bound on discriminability (ROC-AUC) in the judge's competence $c$ and answer-space size $k$, a necessary condition $c > 1/k$, and the result that the marginal AUC is confounded by item difficulty while a within-question estimator is not. A non-intervening probe records judge scores on genuine optimization runs without altering any decision. We find discriminability at chance where competence sits near the floor and usable above it; that a judge's benchmark accuracy overstates the competence that matters; and, in a closed-loop study, that the screen predicts which kind of gating error occurs. The result is a cheap pre-deployment diagnostic for judge gates.
1 Introduction
The paper proposes testing whether a reference-free LLM judge can distinguish correct from incorrect answers before deploying it as a validation gate. It models the judge as a latent solver, measures competence and discriminability with a non-intervening probe, and finds that the screen predicts downstream gating errors.
- Motivation: Text-space skill optimization evolves a natural-language skill document and retains candidates through validation gates based on verifiable signals such as exact-match or executable checks.The optimized object is a persistent document while model weights remain frozen.
- Motivation: An LLM-judge gate could extend optimization beyond verifiable rewards, but its reliability must be tested before replacing a verifier gate.The proposed diagnostic scores every candidate without affecting accept/reject decisions, preserving the optimizer’s original dynamics.
- Findings: Judge discriminability is at chance on research mathematics but usable on factual QA and graduate science, collapsing when competence nears the chance floor.The closed-form bound holds where it can be tested without difficulty confounding.
- Findings: A judge’s headline benchmark accuracy overstates relevant competence, while the pre-deployment screen predicts gating-error type: failed judges import regressions, whereas a passing judge only over-rejects.The overstatement reflects censoring convention, benchmark exposure, and the solve/grade context gap.
- Theory: The theory models a reference-free judge as a latent solver and derives a closed-form ROC-AUC bound in competence c and answer-space size k, requiring c > 1/k for discrimination.A reference-free judge effectively re-solves each item, so scores approach randomness as competence nears the chance floor 1/k.
2 Related Work
Prior work develops self-evolving natural-language skills and uses judges mainly with reference-based evaluation or as assumed learning signals. This paper shifts attention to reference-free judge reliability, arguing that evaluation ability is bounded by problem-solving ability rather than generic bias alone.
- Self-evolving agent skills: Self-evolving agent research equips frozen agents with reusable natural-language skills and optimizes them through offline distillation, online reflection, validation gates, and related mechanisms.The cited approaches also include adversarial co-evolution, shared asset layers, and credit signals.
- Judges already inside the evaluation loop: Existing judges inside skill-evolution evaluation are primarily reference-based, receiving the question, gold answer, and agent response before issuing a verdict.Supplying the gold answer turns judging into comparison and avoids the reference-free evaluation question studied here.
- Reward-free and judge-based optimization: Reward-free optimization uses LLM judges for rollout ranking and combined preference, judge, and programmatic signals, but assumes rather than characterizes when judge signal is usable.This paper transposes that judge-signal idea from weight-space posttraining to frozen-weight, text-space skill optimization.
- LLM-as-judge reliability: LLM judges show systematic biases and can be gamed, while this work adds that evaluation ability is bounded by the judge’s ability to solve the task.The paper presents this task-dependent bound, rather than generic bias, as explaining where judge gating fails.
3 Method
The method models reference-free judging as implicit re-solving, deriving competence- and answer-space-based bounds on discriminability and a chance-floor condition. It then uses a non-intervening probe and within-question estimation to diagnose judge quality without changing optimization decisions.
- Latent-solver model: A reference-free judge scores candidate answers by agreement with an internally re-solved answer, whose competence c bounds evaluation when no independent ground truth exists.The model assumes conditional independence, uniform dispersion across wrong answers, and consistency scoring.
- Discriminability bounds: AUC is above chance only when c > 1/k under the base model, while noise and correlated judge–candidate errors can reduce observed discriminability toward or below chance.The closed-form relation is an upper bound under collusion, and c > 1/k is necessary but not sufficient in the relaxed family.
- Diagnostic instrumentation: The non-intervening probe records judge scores and verifier labels during genuine optimization runs without entering acceptance, yielding an episode-level dataset for offline diagnostics.Because acceptance remains a deterministic function of the verifier score, probed and unprobed runs are path-wise identical in distribution.
- Difficulty control: Marginal AUC can be inflated by item difficulty alone, whereas within-question AUC removes additive item-only score effects and preserves the threshold at pair-weighted competence.The method reports marginal and within-question AUC side by side, with their gap estimating the difficulty-confound share.
- Competence measurement: The method distinguishes grading-context competence cgrade from closed-book competence csolve, treating csolve as a lower-bound proxy when candidate answers scaffold re-solving.The expected relationship is csolve ≤ cgrade in recall-friendly domains.
4 Experiments
Experiments test whether reference-free judge discriminability tracks competence across three tasks, competence estimators, providers, and a closed-loop optimization study. Discriminability is near-random near the chance floor, usable above it, and screening predicts whether judge gating causes harmful acceptance or conservative over-rejection.
- Discriminability: Across mathematics, factual QA, and GPQA-Diamond, marginal AUC spans 0.46, 0.86, and 0.74, respectively, but difficulty confounds the marginal estimate.Within-question AUC pairs positive and negative episodes from the same question; GPQA has no within-question strata.
- Bound validation: The closed-form competence bound holds on every task with within-question stratification; the sole violation is GPQA×Haiku, where neither clean-test condition holds.The excess is consistent with marginal difficulty inflation and with self-solve competence being a lower bound on grading competence.
- Chance-floor diagnostic: Mathematics has self-solve competence 0.267 versus the chance floor 1/k = 0.2, yet observed discriminability is 0.489, marginally below chance.Competence near the floor cannot support a usable gate, while competence above it does not guarantee one.
- Competence measurement: On GPQA-Diamond, Sonnet’s answered-only competence is 0.840 versus approximately 0.55 genuine competence, with censoring accounting for 0.840 → 0.626 and exposure for 0.626 → 0.547.The answered-only figure excludes non-convergent episodes, creating survivorship bias; the residual exposure component is materially smaller.
- Cross-provider replication: Across four independently trained provider families, mathematics competence at or below 8–21% yields near-random AUC 0.31–0.57, while factual-QA competence of 33–80% yields usable AUC 0.67–0.80.Answered-only scoring can inflate competence: V4-flash falls from 94.8% on answered GPQA items to 73.0% when non-convergence is included.
- Closed-loop gating: In the closed-loop pilot, failing the screen produced false acceptance of harmful candidates at 0.33 on factual QA and 0.50 on mathematics, while passing produced a 0.00 false-accept rate.Passing judges instead made one-sided over-rejection errors; the screen is a prerequisite for safe optimization, not a guarantee of competitive performance.
5 Conclusion
Before using an LLM judge to gate skill optimization, its scores must first be shown to carry signal. Modeling a reference-free judge as a latent solver yields a competence-based necessary condition and motivates a non-intervening probe, while sustained optimization under drift and Goodhart pressure remains open.
- Conclusion: Reference-free judge discriminability is bounded by judge competence, requiring competence above the chance floor as a necessary but insufficient condition.The latent-solver model makes this limitation precise.
- Conclusion: A non-intervening probe provides support for assessing discriminability on the tested tasks before deployment.The probe evaluates whether judge scores carry signal without altering optimization decisions.
- Conclusion: Whether a judge that clears the screen remains effective during sustained optimization under drift and Goodhart pressure is left as the natural next question.The necessary condition does not resolve performance in a changing closed loop.