PaperlayerView exact revision on arXiv

Source-linked AI summary

J-Zero: Unified Challenger--Solver--Judge Co-Evolution from Zero Data

Gyouk Chu, Myeongho Jeon, Eunho Yang

arXiv:2608.26582v1cs.LGcs.AIcs.CL

TL;DR

Self-evolution has advanced in verifiable domains, but remains less explored in unverifiable domains. J-Zero co-evolves a Challenger, Solver, and Judge using zero-data preference signals, outperforming baselines by 4.2 points on verifiable and 8.0 points on unverifiable domains while improving through at least ten iterations.

  • Problem

    Self-evolution in unverifiable domains remains substantially less explored than in verifiable domains, limiting evidence about self-improvement without human supervision.

  • Method

    J-Zero co-evolves Challenger, Solver, and Judge models through adversarial self-play and structurally ordered preference pairs derived without external data or human feedback.

  • Results

    J-Zero improves performance by 4.2 points on verifiable tasks and 8.0 points across unverifiable benchmarks over baselines.

  • Takeaways & Limitations

    J-Zero continues improving through ten iterations, whereas existing approaches saturate within two, identifying Judge co-evolution as critical for sustained self-evolution.

  • Takeaways & Limitations

    Experiments are limited to base models with policies and a Judge of up to 8B parameters; larger scales and post-trained reasoning models remain untested.

Abstract

from arXiv · show

Self-evolving language models have recently emerged as a promising path toward superintelligence, with the advantage of reducing the cost of human supervision. While considerable progress has been made in verifiable domains, self-evolution in unverifiable domains remains substantially less explored. We propose Judge co-adaptation from Zero data (J-Zero), a unified Challenger--Solver--Judge co-evolution framework that supports self-improvement across both domains. The Challenger and Solver co-evolve through an adversarial interaction: the Challenger generates increasingly difficult tasks, while the Solver learns to produce higher-quality responses to them. In parallel, the Judge co-adapts using preference pairs whose ordering is known in advance from how each response was produced, i.e., the Solver's answer over the Challenger's, and its decomposed-and-recombined answer over its one-shot answer, rather than from the Judge's own scores. J-Zero outperforms the baselines by an average of 4.2 points on verifiable and 8.0 points on unverifiable domains, and continues to improve through at least ten iterations, whereas the baselines degrade after two.

1 INTRODUCTION

J-ZERO addresses the limits of human-curated supervision and frozen judging by jointly co-evolving a Challenger, Solver, and Judge from zero data. It improves performance in both verifiable and unverifiable domains while sustaining progress across iterative rounds.

  • Motivation: Self-evolving LLMs reduce dependence on costly human task design and labeling, which otherwise bottlenecks progress toward systems surpassing human intelligence.The motivation is framed by the expense of human-curated supervision (Tao et al., 2024; Jeon et al., 2025).
  • Method: J-ZERO jointly co-evolves Challenger and Solver through a GRPO-based minimax game, with the Challenger generating harder tasks and the Solver producing higher-quality answers.The Judge co-adapts alongside them, enabling self-evolution in both verifiable and unverifiable domains.
  • Motivation and Method: The Judge learns preferences from structural asymmetries in how responses are produced rather than from external labels, resolving the apparent circularity of Judge training.This replaces the frozen-Judge ceiling that limits learning in unverifiable domains, where quality depends on human preference rather than checkable correctness.
  • Results: 4.2 points on verifiable tasks and 8.0 points across three unverifiable benchmarks are achieved over the baseline by J-ZERO.These results are reported in Table 1 and Table 2, respectively.
  • Results: Judge co-evolution sustains improvement across iterative rounds and is identified as the key component enabling continued progress.The analysis uses Table 3 and Figures 3–4.

2 RELATED WORK

Prior self-evolution methods progressed from ground-truth supervision to external seed resources and then data-free reward mechanisms, but adaptive evaluation across verifiable and unverifiable domains remained unaddressed. J-Zero targets this gap with jointly updated Challenger, Solver, and Judge components trained through iterative self-play.

  • Self-evolution with external tasks and supervision: Prior methods improved Solvers using ground-truth labels in verifiable domains, remaining bounded by the availability and scope of human-labeled data.These approaches adapted training to the model’s current capabilities (Zelikman et al., 2022; Yuan et al., 2023; Singh et al., 2024; Zhang et al., 2024; Pang et al., 2024).
  • Self-evolution with external seed resources: Later approaches reduced reliance on labels but still depended on external resources, including SFT references or externally seeded prompts.Self-play fine-tuning uses reference responses from an SFT corpus (Chen et al., 2024), while self-rewarding uses externally seeded prompts and the model as judge (Yuan et al., 2024; Prasad et al., 2025; Wang et al., 2025; Wu et al.).
  • Data-free self-evolution: Data-free self-play removes external-resource dependence by deriving rewards from execution-based verification or majority voting over sampled responses.Absolute Zero verifies self-proposed coding tasks with an executor (Zhao et al., 2025), related work extends execution feedback to software engineering (Wei et al., 2026), and R-Zero uses majority voting instead of an external oracle (Huang et al., 2026b).
  • Data-free self-evolution: The remaining gap is data-free self-evolution with an evaluation signal that learns and continually improves across both verifiable and unverifiable domains.G-Zero (Huang et al., 2026a) concurrently addressed the limitation that a fixed Judge can cap further improvement, while J-Zero jointly updates the Challenger, Solver, and Judge through iterative self-play.

3 METHODOLOGY

J-ZERO co-evolves the Challenger, Solver, and Judge in an iterative self-play loop rather than keeping the Judge fixed. The framework combines adversarial task generation, capability-frontier task selection, and role-grounded preference pairs to sustain self-improvement.

  • Judge co-evolution: Judge co-evolution avoids the ceiling of a fixed reward model, which may upper-bound self-evolution (Huang et al., 2026a), by updating from preference pairs whose ordering is known independently of Judge scores.This addresses the bias-reinforcement risk of Judge-generated labels described for self-rewarding methods (Yuan et al., 2024).
  • 3 METHODOLOGY: J-ZERO alternates Challenger, Solver, and Judge updates so increasingly difficult tasks and increasingly capable responses expand the task frontier together.The Challenger seeks difficult, diverse, valid tasks, while the Solver learns to obtain high Judge scores on them.
  • Adversarial Challenger–Solver game: The Challenger maximizes difficulty while penalizing repetitive or malformed tasks, using repetition distances, cluster penalties, and a <question>-tag format check.Difficulty is defined from the Solver’s mean Judge score, with higher reward for tasks the Solver handles poorly; updates use GRPO.
  • Task selection for Solver evolution: The Solver trains on top-K candidate tasks with the largest response-score dispersion, targeting tasks near its capability frontier where reward variance indicates room for learning.This continuous criterion generalizes R-Zero’s (Huang et al., 2026b) informative-band selection from binary accuracy to Judge-score dispersion; training uses GRPO.
  • Judge co-evolution: Role-asymmetry pairs label Solver responses above Challenger responses, while subtask-amplification pairs label decomposed-and-recombined Solver answers above one-shot answers.The two orderings are yS ≻ yC and yamp ≻ yS, and their union trains the Judge with a Bradley–Terry loss.

4 EXPERIMENTS

J-ZERO achieves the best score on every benchmark group across verifiable and unverifiable domains at both Qwen3 model scales. Its advantage is especially pronounced in unverifiable tasks, where existing zero-data baselines make substantially smaller gains.

  • Experimental setup: The experiments use Qwen3-4B-Base and Qwen3-8B-Base, comparing the base models with R-Zero and G-Zero across 11 verifiable and 3 unverifiable benchmarks.Verifiable evaluation covers math reasoning, general-domain reasoning, and instruction following; unverifiable evaluation covers AlpacaEval 2.0, Arena-Hard-v2.0, and EQ-Bench Creative Writing v3.
  • Overall results: J-ZERO attains the best score on every benchmark group in both verifiable and unverifiable domains at both model scales.The evaluations are reported in Table 1 for verifiable domains and Table 2 for unverifiable domains.
  • Verifiable domain: 9.47 and 7.88 points: J-ZERO improves over the corresponding Qwen3-4B-Base and Qwen3-8B-Base models in verifiable-domain average performance.It also exceeds R-Zero by 4.74 and 3.56 points, despite R-Zero being designed for verifiable-domain self-evolution.
  • Unverifiable domain: 3.08 and 2.31 points: R-Zero’s unverifiable-domain gains are roughly half its verifiable-domain gains, while G-Zero improves by only 1.31 and 2.08 points.R-Zero relies on majority-vote reward, which does not extend to unverifiable open-ended tasks; G-Zero remains behind R-Zero and barely above the base model.

5 ANALYSIS

J-ZERO’s co-evolving Judge sustains Solver improvement by adapting evaluation standards as adversarial difficulty increases. Its preference pairs remain reliable, with role-asymmetry supervision effective throughout and subtask-amplification supervision becoming effective after Solver maturation.

  • Preference reliability: Over 60% of role-asymmetry comparisons favor the Solver at every iteration, although its win rate declines from 87.9% to approximately 66% as adversarial difficulty increases.The persistent majority supports the reliability of Drole labels, while the decline is attributed to the increasingly difficult curriculum rather than mislabeled preferences.
  • Preference reliability: 21.1% of subtask-amplification comparisons favor decomposition at iteration 1, but the win rate exceeds 50% from iteration 4 and later reaches roughly 70 to 80%.Decomposition becomes beneficial once the Solver can reliably solve subtasks, providing supervision beyond its one-shot frontier.
  • External evaluation: Judge co-evolution also improves performance on RM-Bench (Liu et al., 2025b), an independent reward-model benchmark unrelated to the loop’s constructed preference pairs.The primary purpose remains adapting the Judge to the current Solver rather than maximizing standalone evaluation capability.
  • Judge-data ablation: −1.64 versus −0.97 points are lost when removing subtask-amplification versus role-asymmetry pairs, showing that both components improve over the fixed-Judge baseline but subtask amplification contributes more.This ablation is consistent with role-asymmetry tracking Solver-versus-Challenger quality and subtask amplification extending beyond the Solver’s one-shot capability.
  • Training dynamics: 9.47 and 11.23 points are gained over the base model on verifiable and unverifiable benchmarks, respectively, with J-ZERO improving monotonically through iteration 10.R-Zero and G-Zero peak at iteration 2 and then decline, whereas J-ZERO does not plateau within the training budget.
  • Training dynamics: 1.66 and 4.44 points separate the full run from the frozen-Judge variant on verifiable and unverifiable domains, respectively, after the latter plateaus.The frozen Judge tracks J-ZERO for three iterations, then reaches its evaluation ceiling; co-evolution keeps the evaluation standard above that ceiling.

6 CONCLUDING REMARKS

J-ZERO is a zero-data framework in which Challenger, Solver, and Judge co-evolve for self-improvement across verifiable and unverifiable domains. Its limitations include restricted model scale and architecture choices, with larger models and post-trained reasoning models remaining untested.

  • Conclusion: J-ZERO co-evolves the Challenger, Solver, and Judge to support zero-data self-improvement in verifiable and unverifiable domains.Role asymmetry provides early preference signals, while subtask amplification supplies supervision beyond the Solver’s one-shot capability.
  • Conclusion: Role asymmetry and subtask amplification help the Judge overcome a fixed evaluation ceiling and keep pace with evolving policies without external data.
  • Limitations: Compute constraints limit the Challenger, Solver, and Judge to policies of up to 8B parameters with an 8B Judge, and larger scales and post-trained reasoning models remain untested.The Judge is a classifier-based discriminative reward model initialized from an off-the-shelf checkpoint and trained with the BT loss, while the Challenger and Solver share a single generative initialization.

APPENDIX · A FULL EXPERIMENTAL DETAILS · A.1 BENCHMARKS AND EVALUATION

The evaluation covers verifiable reasoning, general-domain, instruction-following, and unverifiable-domain benchmarks, using task-specific decoding, scoring, and comparison protocols. It spans seven math, three general-domain, one instruction-following, and three unverifiable benchmarks.

  • A.1 BENCHMARKS AND EVALUATION: Seven math benchmarks—GSM8K, MATH500, Minerva, OlympiadBench, AMC23, AIME24, and AIME25—evaluate reasoning performance.Responses use up to 4096 tokens with the specified step-by-step system prompt; AMC and AIME report avg@32, while the remaining benchmarks use greedy decoding accuracy.
  • A.1 BENCHMARKS AND EVALUATION: Math evaluation combines greedy accuracy with avg@32 sampling for AMC23, AIME24, and AIME25 under a step-by-step reasoning prompt.All math responses may contain up to 4096 tokens, following R-Zero (Huang et al., 2026b).
  • A.1 BENCHMARKS AND EVALUATION: Three general-domain benchmarks—MMLU-Pro, SuperGPQA, and Big-Bench Hard (BBH)—are scored by greedy-decoding accuracy.Responses use up to 8192 tokens, and the evaluation code is strengthened to prevent incorrect answers from being randomly marked correct.
  • A.1 BENCHMARKS AND EVALUATION: Instruction following is evaluated on IFEval using prompt- and instruction-level strict and loose accuracy.The evaluation follows the official source code and reports all four metrics: prompt-level strict, instruction-level strict, prompt-level loose, and instruction-level loose accuracy.
  • A.1 BENCHMARKS AND EVALUATION: Three unverifiable-domain benchmarks—AlpacaEval 2.0, Arena-Hard-v2.0, and EQ-Bench Creative Writing v3—measure generation quality with task-specific metrics.AlpacaEval reports length-controlled win rate against GPT-4-Turbo; Arena-Hard reports win rates against o3-mini and gemini-2.0-flash; EQ-Bench reports rubric score instead of Elo rating.
  • A.1 BENCHMARKS AND EVALUATION: Unverifiable evaluation uses benchmark-specific opponents and scoring: GPT-4-Turbo for AlpacaEval, o3-mini and gemini-2.0-flash for Arena-Hard, and rubric scoring for EQ-Bench.Arena-Hard separates Hard Prompt evaluation with style control from its Creative Writing subset.

A.2 IMPLEMENTATION DETAILS

Experiments use the verl framework across four NVIDIA B200 and four NVIDIA H200 GPUs, with J-ZERO generally following R-Zero’s hyperparameters. Baselines retain their original configurations, including G-Zero’s LoRA training and larger Challenger batch size.

  • Training setup: J-ZERO trains with verl on four NVIDIA B200 and four NVIDIA H200 GPUs, generally following R-Zero’s hyperparameters, including a Challenger batch size of 16 and no weight decay.Full hyperparameter settings are provided in Table 4.
  • Baseline configurations: R-Zero uses a Challenger batch size of 16, whereas G-Zero retains LoRA training and a Challenger batch size of 128 from its original configuration.Each baseline otherwise keeps its own configuration.
  • Baseline configurations: Preliminary experiments found that full-parameter fine-tuning or reducing G-Zero’s Challenger batch size to 16 lowered performance.These comparisons motivated retaining G-Zero’s original LoRA-based setup and batch size.

B JUDGE IMPROVEMENTS · C PROMPTS

J-Zero’s Judge improves across RM-Bench domains and especially on hard preference pairs, while the prompt design specifies how the Challenger generates tasks, constructs subtask-amplification pairs, and supports Solver responses. The prompts emphasize diverse, predominantly non-mathematical open-ended requests and concise, self-contained subtask solutions.

  • B JUDGE IMPROVEMENTS: The Judge’s co-adaptation keeps supervision appropriate for the latest Solver rather than optimizing a static-benchmark reward model, without losing general reward-modeling capability.The Judge is evaluated at every iteration on RM-Bench (Liu et al., 2025b).
  • B JUDGE IMPROVEMENTS: 93.95 average RM-Bench accuracy, up 1.34 points from 92.61, with gains in all four domains and the largest improvement in Chat (+3.70).Math improves by +1.24 and Code by +0.34, while Safety is nearly unchanged (+0.10) from its already saturated 98.69 accuracy.
  • B JUDGE IMPROVEMENTS: 4.77 points: Hard-pair accuracy rises from 85.08 to 89.85, while Normal is unchanged (+0.01) and Easy decreases by 0.74 points.Hard comparisons are most important for improving an already strong Solver, and the Judge’s gains concentrate where response style conflicts with response quality.
  • C.1 CHALLENGER PROMPTS IN J-ZERO: The Challenger receives only the user prompt and generates one self-contained, non-trivial request sampled across general-domain task types, wrapped in <question> tags.Its prompt favors writing, explanation, advice, planning, analysis, coding, role-play, dialogue, and creative tasks, with non-math categories weighted heavily.
  • C.1 CHALLENGER PROMPTS IN J-ZERO: The Challenger’s decomposition prompt requests 3-5 essential, simpler, task-specific subtasks ordered so earlier answers support later ones.A separate composition prompt combines solved subtasks into one coherent final answer, resolving contradictions and correcting obvious errors.
  • C.2 SOLVER PROMPTS IN J-ZERO: The Solver answers each Challenger-generated question using a system prompt, while separate system and user prompts collect concise, self-contained answers for each decomposition subtask.The subtask prompt supplies the larger task as context and asks the Solver to answer only the specified subtask.

C.3 LLM-AS-A-JUDGE PROMPT FOR JUDGE TRAINING DATA ANALYSIS · D SUBTASK AMPLIFICATION EXAMPLES

The appendix specifies a forced-choice, domain-aware LLM judge that flags degeneration and emits strict JSONL, then illustrates how broad user prompts are decomposed into actionable subtasks across planning, travel, marketing, fiction, and coding.

  • C.3 LLM-AS-A-JUDGE PROMPT FOR JUDGE TRAINING DATA ANALYSIS: The judge must select exactly one winner, A or B, for two answers to the same question.It is framed as an impartial expert judge and never permits ties.
  • C.3 LLM-AS-A-JUDGE PROMPT FOR JUDGE TRAINING DATA ANALYSIS: The judge first classifies each question as math, code, explain, advice, writing, or other, then applies domain-specific quality criteria.Criteria prioritize correctness for math, specification compliance for code, factual accuracy for explanation, and usefulness for advice.
  • C.3 LLM-AS-A-JUDGE PROMPT FOR JUDGE TRAINING DATA ANALYSIS: Degeneration—such as repetition, echoed prompts, empty output, leaked scaffolding, or self-cutoff—loses heavily, while display truncation is not penalized.The judge evaluates only what is shown and distinguishes a display cutoff from model degeneration.
  • C.3 LLM-AS-A-JUDGE PROMPT FOR JUDGE TRAINING DATA ANALYSIS: Each candidate receives an independent deg_a or deg_b degeneration flag regardless of which answer wins.The output therefore records both comparative preference and answer-level degeneration status.
  • C.3 LLM-AS-A-JUDGE PROMPT FOR JUDGE TRAINING DATA ANALYSIS: The judge outputs one strict JSONL object per item containing the item id, domain, winner, degeneration flags, and a short reason.No prose or Markdown fences are permitted.
  • D SUBTASK AMPLIFICATION EXAMPLES: Subtask amplification decomposes broad prompts into ordered, actionable steps, as shown for office monitoring, Paris travel, coffee-shop marketing, historical fiction, and factorial coding.Examples include GDPR compliance, family-oriented activity planning, audience segmentation, character construction, and negative-input handling.
  • D SUBTASK AMPLIFICATION EXAMPLES: The examples span technical and creative tasks, with subtasks covering implementation, compliance, recommendations, strategy, narrative design, and explanation.The factorial example specifically separates function definition, computation, error handling, mathematical explanation, and the rationale for handling negative inputs.
Loading 2608.26582v1…