PaperlayerView exact revision on arXiv

Source-linked AI summary

J-Zero: Unified Challenger--Solver--Judge Co-Evolution from Zero Data

Gyouk Chu, Myeongho Jeon, Eunho Yang

arXiv:2608.26582v1cs.LGcs.AIcs.CL

TL;DR

Human-curated task design and labeling are costly bottlenecks in language-model development. J-Zero co-evolves a Challenger, Solver, and Judge from zero data. It achieves the best scores across all benchmark groups at both tested scales, including a 4.2-point accuracy gain over the baseline on verifiable tasks.

  • Problem

    Human-curated task design and labeling are costly bottlenecks in developing language models that may eventually surpass human intelligence.

    Source: §1 · p. 1

  • Method

    J-Zero co-evolves a Challenger, Solver, and Judge from zero data, using role asymmetry and subtask amplification to supervise the Judge.

    Source: §6 · p. 9

  • Results

    J-Zero achieves the best scores across all benchmark groups at both scales, including a 4.2-point accuracy gain over the baseline on verifiable tasks.

    Source: §1 · p. 2, §4.2 · p. 6

  • Takeaways & Limitations

    Judge co-evolution is a key component for sustaining self-improvement across iterative rounds.

    Source: §1 · p. 2

    Evaluation covers only base models with Challenger and Solver policies up to 8B parameters and an 8B Judge; larger scales and post-trained reasoning models remain untested.

    Source: §6 · p. 9

Abstract

from arXiv · show

Self-evolving language models have recently emerged as a promising path toward superintelligence, with the advantage of reducing the cost of human supervision. While considerable progress has been made in verifiable domains, self-evolution in unverifiable domains remains substantially less explored. We propose Judge co-adaptation from Zero data (J-Zero), a unified Challenger--Solver--Judge co-evolution framework that supports self-improvement across both domains. The Challenger and Solver co-evolve through an adversarial interaction: the Challenger generates increasingly difficult tasks, while the Solver learns to produce higher-quality responses to them. In parallel, the Judge co-adapts using preference pairs whose ordering is known in advance from how each response was produced, i.e., the Solver's answer over the Challenger's, and its decomposed-and-recombined answer over its one-shot answer, rather than from the Judge's own scores. J-Zero outperforms the baselines by an average of 4.2 points on verifiable and 8.0 points on unverifiable domains, and continues to improve through at least ten iterations, whereas the baselines degrade after two.

1 INTRODUCTION

J-ZERO addresses the frozen-Judge ceiling on self-improvement by co-adapting the Challenger, Solver, and Judge using structurally derived preferences. It reports improvements in both verifiable and unverifiable domains, with Judge co-evolution sustaining improvement across iterative rounds.

1 INTRODUCTION

  • Human-designed tasks and labels are costly bottlenecks for developing AI systems that may surpass human intelligence (Tao et al., 2024; Jeon et al., 2025).Self-evolving LLMs offer an approach to overcoming dependence on human-curated supervision.Source: §1 · p. 1
  • A frozen Judge bounds self-improvement because training loses its signal once the Solver saturates the Judge’s evaluative distinctions (Huang et al., 2026a).Unlike verifiable tasks with objective ground truth, unverifiable tasks have no single correct answer; a Judge supplies preference-based scores instead of a verifier (Kuba et al., 2025).Source: §1 · p. 1
  • J-ZERO co-adapts the Judge alongside an adversarial Challenger–Solver pair to enable self-evolution in both verifiable and unverifiable domains.Using GRPO (Shao et al., 2024), the Challenger minimizes Judge-assigned reward by generating difficult tasks, while the Solver maximizes it by producing high-quality responses.Source: §1 · pp. 1–2
  • Structurally derived preferences train the Judge with Bradley–Terry objectives, favoring Solver over Challenger responses and divide-and-conquer over one-shot Solver responses.Role-asymmetry pairs exploit the Solver’s optimization for answer quality versus the Challenger’s optimization for task difficulty. Subtask-amplification pairs exploit easier subtasks and aggregation to produce more comprehensive responses (Christiano et al., 2018).Source: §1 · p. 2
  • 4.2-point accuracy improvement over the baseline on verifiable tasks and 8.0-point performance improvement across three unverifiable benchmarks are reported for J-ZERO.Further analysis identifies Judge co-evolution as the key component sustaining improvement across iterative rounds.Source: §1 · p. 2

2 RELATED WORK

J-Zero targets data-free self-evolution across verifiable and unverifiable domains by co-evolving the Judge with the Challenger and Solver, allowing the evaluation signal itself to improve.

Self-evolution with external tasks and supervision

Self-evolution with external seed resources

Data-free self-evolution

  • Data-free methods largely rely on rewards suited to verifiable domains, while a static Judge may limit further improvement in unverifiable domains.Absolute Zero uses execution-based verification, whereas R-Zero (Huang et al., 2026b) uses majority voting over sampled responses. Kuba et al. (2025) extends data-free self-evolution to unverifiable domains, but its static Judge may impose an improvement ceiling (Huang et al., 2026a).Source: §2 · p. 2

2 RELATED WORK

  • J-Zero directly co-evolves its Judge with the Challenger and Solver, distinguishing it from concurrent approaches that train the Solver using constructed preferences.G-Zero (Huang et al., 2026a) uses Challenger-generated hints to construct preference pairs between Solver responses and trains the Solver through direct preference optimization. J-Zero instead improves the evaluation signal over successive rounds to support more stable, sustained self-improvement.Source: §2 · p. 2

3 METHODOLOGY

J-ZERO iteratively co-adapts a Challenger, Solver, and Judge through adversarial task generation, response learning, and Judge updates. Judge training uses internally constructed preference pairs whose labels do not depend on the Judge’s own scores.

3 METHODOLOGY

  • J-ZERO alternates Challenger, Solver, and Judge updates rather than keeping the Judge fixed throughout self-evolution.The Challenger minimizes the Judge’s reward for Solver responses, the Solver maximizes it, and the Judge learns from internally constructed preference pairs.Source: §3 · p. 3

3.1 ADVERSARIAL EVOLUTION OF THE CHALLENGER AND SOLVER

  • The Challenger uses GRPO to generate difficult tasks while preserving diversity and valid formatting, making its interaction with the Solver adversarial but not strictly zero-sum.Tasks form one GRPO group. Reward combines 1 minus the mean Judge score across Solver responses, BLEU-cluster repetition penalties, and a format check requiring <question> tags.Source: §3.1 · p. 4
  • The Solver trains with GRPO on top-K tasks with greatest Judge-score dispersion, targeting its capability frontier while the Challenger and Judge remain fixed.This continuous-score filter generalizes the intermediate-accuracy selection used by R-Zero (Huang et al., 2026b) with a binary verifier. Alternating updates expand the task frontier while the Solver learns higher-quality responses.Source: §3.1 · pp. 4–5

3.2 JUDGE ADAPTATION

  • Judge adaptation constructs preferences without external supervision or Judge-derived labels, ranking Solver responses above Challenger answers because only the Solver is trained to answer tasks.Both policies answer held-out tasks under the same prompt, providing role-based supervision even when the Judge is uncertain or miscalibrated.Source: §3.2 · p. 5
  • Subtask amplification counters role-only saturation by preferring decomposed-and-recombined responses over one-shot Solver answers, exposing the Judge to quality beyond the current one-shot frontier.The Challenger decomposes held-out tasks and recombines the Solver’s subtask answers. These amplified responses tend to be more complete and accurate than direct one-shot responses.Source: §3.2 · pp. 5–6
  • The Judge minimizes Bradley–Terry loss over both preference datasets, using the latest Challenger–Solver outputs to correct misrankings at the advancing self-evolution frontier.Training starts from the previous iteration’s Judge parameters. The two pair types provide complementary supervision because their reliability differs across training stages.Source: §3.2 · p. 6

4 EXPERIMENTS

J-ZERO achieves the best score in every benchmark group at both model scales, with larger gains over base models in unverifiable than verifiable domains. Experiments cover 11 verifiable and three unverifiable benchmarks.

4.1 EXPERIMENTAL SETUP

  • 4B and 8B Qwen3 base models (Yang et al., 2025) are evaluated against their original base models and two zero-data self-play frameworks.The frameworks are R-Zero (Huang et al., 2026b) and G-Zero (Huang et al., 2026a). Skywork-Reward-V2-Llama-3.1-8B (Liu et al., 2026a) serves as the Judge.Source: §4.1 · p. 6
  • 11 verifiable and three unverifiable benchmarks are evaluated, selecting each method's last checkpoint before either domain's average score declines.Verifiable evaluation covers seven math reasoning benchmarks, three general-domain reasoning benchmarks, and IFEval (Zhou et al., 2023b). Unverifiable evaluation uses AlpacaEval 2.0, Arena-Hard-v2.0 (Li et al., 2025), and EQ-Bench Creative Writing v3.Source: §4.1 · p. 6

4.2 RESULTS

  • 9.47 and 7.88 points are J-ZERO's verifiable-domain gains over Qwen3-4B-Base and Qwen3-8B-Base, respectively, exceeding R-Zero by 4.74 and 3.56 points.J-ZERO surpasses R-Zero despite the latter being specifically designed for self-evolution in verifiable domains.Source: §4.2 · p. 6
  • J-ZERO achieves the best score in every benchmark group at both model scales across verifiable and unverifiable domains.Verifiable Overall scores average three domain averages; unverifiable Overall scores average three benchmarks after first averaging the two Arena-Hard subsets.Source: §4.2 · p. 6, Table 1 · p. 7, Table 2 · p. 7
  • 11.23 and 10.18 points are J-ZERO's average unverifiable-domain gains over the corresponding base models, with its largest gains on AlpacaEval 2.0.AlpacaEval 2.0 scores rise from 6.22 to 28.56 and from 12.93 to 33.53.Source: §4.2 · p. 7
  • 3.08 and 2.31 points are R-Zero's unverifiable-domain gains, while G-Zero gains only 1.31 and 2.08 points.R-Zero's majority-vote reward does not extend to unverifiable open-ended tasks, while G-Zero does not use a Judge; both improve less here than in verifiable domains.Source: §4.2 · p. 7

5 ANALYSIS

J-ZERO improves monotonically through ten iterations, while R-Zero and G-Zero peak at iteration two and then decline. Its two preference sources provide complementary training signals, and Judge co-evolution sustains improvement beyond the frozen-Judge variant.

5.1 RELIABILITY OF SELF-GENERATED PREFERENCE LABELS

  • More than 60% of comparisons favor the Solver over the Challenger at every iteration, supporting role-asymmetry labels throughout training.In Qwen3-4B-Base experiments, Claude Opus 4.8 evaluates both presentation orders, with ties excluded. The decline from 87.9% to approximately 66% is attributed to increasingly difficult tasks narrowing the Solver–Challenger quality gap.Source: §5.1 · p. 8
  • 21.1% at iteration 1 rises to roughly 70 to 80% later for divide-and-conquer responses, whose win rate exceeds 50% from iteration 4.Amplification responses win fewer than half of comparisons during the first three iterations because decomposition requires reliable subtask solving. The reliability curves cross: role-asymmetry pairs supply the early signal, and amplification pairs take over as the Solver matures.Source: §5.1 · p. 8, Figure 2 · p. 8
  • Judge co-evolution also improves performance on RM-Bench (Liu et al., 2025b), an independent reward-model benchmark unrelated to the loop’s constructed preference pairs.This improvement accompanies the primary objective of adapting the Judge to the current Solver, rather than improving standalone evaluation capability.Source: §5.1 · p. 8

5.2 ABLATION STUDY ON PREFERENCE DATA FOR JUDGE ADAPTATION

  • −1.64 is the performance change from removing subtask-amplification pairs, versus −0.97 from removing role-asymmetry pairs; combining both sources performs best.Both sources contribute gains over the fixed-Judge baseline. Role-asymmetry pairs teach distinctions below the Solver’s current capability, while amplification exposes the Judge to responses beyond the Solver’s one-shot frontier.Source: §5.2 · p. 8, Table 3 · p. 8

5.3 SUSTAINED IMPROVEMENT OVER ITERATIONS

  • 9.47 and 11.23 points over the base model are J-ZERO’s gains on verifiable and unverifiable benchmarks, respectively, with monotonic improvement through iteration 10.R-Zero and G-Zero peak at iteration 2 and decline thereafter; J-ZERO does not plateau within the training budget.Source: Figure 3 · p. 9, §5.3 · p. 9
  • 1.66 and 4.44 points separate the frozen-Judge variant from full J-ZERO on verifiable and unverifiable domains, respectively, after the variant plateaus.The frozen-Judge variant tracks J-ZERO for the first three iterations. The authors attribute its subsequent plateau to the Solver reaching the fixed Judge’s evaluation ceiling, whereas co-evolution keeps rewards discriminative.Source: §5.3 · p. 9, Figure 3 · p. 9

6 CONCLUDING REMARKS

J-ZERO co-evolves a Challenger, Solver, and Judge for zero-data self-improvement in verifiable and unverifiable domains. It outperforms prior zero-data methods and improves through ten iterations, but larger scales and post-trained reasoning models remain untested.

Conclusion

  • Two model scales show J-ZERO outperforming prior zero-data methods, with continued improvement through ten iterations versus existing approaches saturating within two.The authors identify the Judge as a critical trainable component whose evaluation capability constrains self-improvement.Source: §6 · p. 9
  • J-ZERO co-evolves Challenger, Solver, and Judge across verifiable and unverifiable domains, using role asymmetry and subtask amplification to keep the evaluator improving.Role asymmetry supplies reliable preferences early in training; subtask amplification supplies supervision beyond the Solver’s one-shot capability. Together, they let the Judge overcome a fixed evaluation ceiling without external data or human feedback.Source: §6 · p. 9

Limitations

  • 8B parameters is the tested upper limit for Challenger and Solver policies, with an 8B Judge; larger scales and post-trained reasoning models remain untested.Compute constraints restricted experiments to base models, excluding post-trained reasoning models that emit long chains of thought.Source: §6 · p. 9
  • The Judge is a discriminative classifier initialized from an off-the-shelf checkpoint and trained with BT loss, unlike the Challenger and Solver’s shared generative initialization.A generative Judge could let one base model instantiate all three roles and provide critiques as richer in-loop supervision. Making it co-adapt within self-play remains future work.Source: §6 · p. 9

APPENDIX · A FULL EXPERIMENTAL DETAILS · A.1 BENCHMARKS AND EVALUATION

Evaluation covers math reasoning, general-domain capabilities, instruction following, and unverifiable tasks using benchmark-specific metrics and decoding settings. Unverifiable evaluations assign different judges to general assistance and creative writing.

A.1 BENCHMARKS AND EVALUATION

  • Seven math benchmarks use avg@32 for AMC and AIME, and greedy-decoding accuracy for the remaining benchmarks.Benchmarks are GSM8K (Cobbe et al., 2021), MATH500 (Hendrycks et al., 2021), Minerva (Lewkowycz et al., 2022), OlympiadBench (He et al., 2024), AMC23, AIME24, and AIME25. Following R-Zero (Huang et al., 2026b), responses allow up to 4096 tokens, with a system prompt requesting step-by-step reasoning and boxed final answers.Source: §A.1 · p. 13
  • General-domain evaluation reports greedy-decoding accuracy with an 8192-token response limit and strengthens evaluation code to prevent incorrect answers from being marked correct.Benchmarks are MMLU-Pro, SuperGPQA, and Big-Bench Hard (BBH; Suzgun et al., 2023). Evaluation otherwise mostly follows R-Zero.Source: §A.1 · p. 13
  • IFEval (Zhou et al., 2023b) reports prompt-level and instruction-level accuracy under both strict and loose criteria, following the official evaluation code.The four metrics are prompt-level strict accuracy, instruction-level strict accuracy, prompt-level loose accuracy, and instruction-level loose accuracy.Source: §A.1 · p. 13
  • AlpacaEval 2.0 reports length-controlled win rate against GPT-4-Turbo.Qwen3.6-27B serves as the judge.Source: §A.1 · p. 13
  • Arena-Hard-v2.0 (Li et al., 2025) reports style-controlled Hard Prompt win rates against o3-mini and Creative Writing win rates against gemini-2.0-flash.Qwen3.6-27B judges Hard Prompts; gemma-4-31B-it (Team et al., 2026) judges Creative Writing.Source: §A.1 · p. 13
  • EQ-Bench Creative Writing v3 reports rubric score rather than Elo rating to avoid model-pool dependence and align metric scales.gemma-4-31B-it (Team et al., 2026) serves as the judge.Source: §A.1 · p. 13
  • gemma-4-31B-it (Team et al., 2026) is selected for creative-writing judging because its EQ-Bench Judgemark v44 score exceeds Qwen3.6-27B’s.The higher score indicates stronger discriminative performance in creative-writing evaluation; except for judge models, all benchmarks follow official protocols and configurations.Source: §A.1 · p. 13

A.2 IMPLEMENTATION DETAILS

J-ZERO largely follows R-Zero (Huang et al., 2026b) training hyperparameters, uses equal proportions of two Judge preference-pair types, and retains baseline-specific configurations.

Training setup

  • Experiments use verl (Sheng et al., 2024), four NVIDIA B200 GPUs, and four NVIDIA H200 GPUs, largely following R-Zero (Huang et al., 2026b) hyperparameters.J-ZERO uses a Challenger training batch size of 16 and disables weight decay. Table 4 provides the full hyperparameter settings.Source: §A.2 · p. 14, Table 4 · p. 14
  • Judge training uses equal proportions of role-asymmetry and subtask-amplification preference pairs.Challenger and Solver prompts are provided in Sections C.1 and C.2, respectively.Source: §A.2 · p. 14

Baseline configurations

  • 128 is G-Zero’s (Huang et al., 2026a) retained Challenger training batch size, alongside its original LoRA-based training setup.Replacing LoRA with full-parameter fine-tuning or reducing the Challenger training batch size to 16 lowered performance in preliminary experiments. R-Zero retains the same Challenger training batch size of 16 as J-ZERO.Source: §A.2 · p. 14

B JUDGE IMPROVEMENTS · C PROMPTS

Judge co-adaptation maintains supervision appropriate for the latest Solver while improving RM-Bench accuracy, particularly on hard preference pairs. The prompts specify task generation, decomposition, subtask solving, and answer recombination.

B JUDGE IMPROVEMENTS

  • 4.77 points gained in Hard-pair accuracy (85.08 to 89.85) exceed Normal’s +0.01, while Easy declines 0.74 points (97.40 to 96.66).Hard pairs present better answers more plainly than worse answers, testing exploitable surface-polish bias that matters for strong policies’ continued improvement. The authors consider Easy’s decline less informative because strong Solvers rarely produce obviously bad responses.Source: B JUDGE IMPROVEMENTS · pp. 14–15, Figure 4 · p. 15
  • 93.95 average accuracy on RM-Bench (Liu et al., 2025b), rising monotonically from 92.61 (+1.34), accompanies improvements across all four domains.Chat improves +3.70, Math +1.24, Code +0.34, and Safety +0.10, with Safety already at 98.69 initially. Co-adaptation primarily keeps supervision appropriate for the latest Solver rather than targeting static reward-model benchmarks.Source: B JUDGE IMPROVEMENTS · p. 14, Figure 4 · p. 15

C.1 CHALLENGER PROMPTS IN J-ZERO

  • The Challenger generates diverse, self-contained, nontrivial requests using only a user prompt modified from G-Zero (Huang et al., 2026a).Reasoning, math, or logic problems are limited to roughly one in six requests. Outputs contain only the request wrapped in <question> and </question> tags.Source: §C.1 · p. 15
  • 3-5 essential, simpler subtasks are ordered so earlier subtasks support later ones when constructing subtask-amplification pairs for Judge training.Subtasks must be specific to the original task and genuinely necessary, avoiding vague steps such as understanding the problem.Source: §C.1 · pp. 15–16
  • The Challenger recombines the Solver’s subtask answers into one coherent final answer, resolving contradictions and correcting obvious subtask errors.The composition prompt requires only the final answer, without restating subtasks, their solutions, or the procedure; closed-form answers are boxed.Source: §C.1 · pp. 15–16

C.2 SOLVER PROMPTS IN J-ZERO

  • The Solver answers Challenger-generated questions with step-by-step reasoning and boxed final answers, while solving amplification subtasks concisely using the larger task only as context.Each subtask response must be complete, correct, self-contained, and contain only that subtask’s answer.Source: §C.2 · pp. 16–17

C.3 LLM-AS-A-JUDGE PROMPT FOR JUDGE TRAINING DATA ANALYSIS · D SUBTASK AMPLIFICATION EXAMPLES

The judge-analysis prompt specifies forced-choice, domain-specific comparisons with degeneration checks and structured outputs. Subtask-amplification examples illustrate five-part decompositions across planning, creative writing, and coding.

C.3 LLM-AS-A-JUDGE PROMPT FOR JUDGE TRAINING DATA ANALYSIS

  • The analysis prompt requires an impartial judge to choose exactly one winner between two answers to the same question, never allowing ties.The candidate answers are produced by different models.Source: §C.3 · p. 17
  • Judging first classifies questions into math, code, explain, advice, writing, or other, then applies domain-specific priorities.Math prioritizes final-answer correctness before reasoning; code prioritizes satisfying specifications, completeness, and edge cases before readability; explanations prioritize factual accuracy before completeness and clarity. Advice favors targeted, actionable guidance and sound judgment; writing emphasizes constraints, coherence, craft, and originality; other tasks emphasize instruction-following, accuracy, and helpfulness.Source: §C.3 · p. 17
  • Degeneration receives heavy penalties, and each answer receives an independent degeneration flag regardless of which answer wins.Degeneration includes repetition loops, unanswered prompt echoes, empty outputs, leaked scaffolding or role markers, and self-cut-off sentences.Source: §C.3 · p. 17
  • Judges assess substance rather than length or inferred model identity, with candidate positions already randomized.A terminal ...[TRUNCATED] marker indicates the display cutoff, not model degeneration; judges assess only the visible text without penalizing that marker.Source: §C.3 · p. 17
  • The required output is one JSONL object per item containing its identifier, domain, winner, two degeneration flags, and a one-sentence reason.The prompt forbids surrounding prose and Markdown fences.Source: §C.3 · pp. 17–18

D SUBTASK AMPLIFICATION EXAMPLES

  • Five-subtask examples cover project planning, travel itineraries, coffee-shop marketing, character sketches, and factorial coding.Project planning separates requirements, responsibilities, resources, development, and GDPR compliance; travel and marketing separate logistics and strategy components. Character writing separates appearance, age/status, motivations, backstory, and style, while coding separates function definition, calculation, negative-input handling, and explanations.Source: D SUBTASK AMPLIFICATION EXAMPLES · pp. 18–19

Related papers

Loading 2608.26582v1…