G-Zero: Self-Play for Open-Ended Generation from Zero Data

• Self-play has shown improvement on reasoning tasks with verifiable outcomes, but many real-world tasks lack objective oracles.Source: §1 · p. 1
• Judge-based self-evolution in unverifiable domains is limited by judge capability and vulnerable to reward hacking, while G-Zero derives its preference signal internally.The comparison describes majority voting as limited to verifiable tasks and judge scores as bounded by judge capability.Source: §1 · p. 1, Figure 1 · p. 2
• Hint-δ measures the shift in the Generator’s predictive distribution caused by a hint, and rewards the Proposer for challenging queries with informative hints.The Proposer is trained with GRPO; the Generator is trained with DPO to prefer hint-guided responses over its unassisted answers.Source: §1 · p. 2, Figure 2 · p. 3
• +3.74 points on AlpacaEval and +5.21 points in AIME 25; G-Zero improves across Qwen and Llama model families.The paper also reports robust improvements over several self-play iterations.Source: §1 · p. 2
• The paper proves a best-iterate suboptimality guarantee for an idealized standard-DPO variant, assuming sufficient Proposer-induced coverage and low δ-certified pseudo-label score noise.Source: §1 · p. 2
• G-Zero’s reasoning gains are reported not to stem from domain-specific memorization, but from internalizing logical depth that transfers from open-ended tasks to mathematical problem-solving.Source: §1 · p. 2

Direct Preference Optimization (DPO)

• DPO aligns policy πθ to preference triples (prompt, chosen response, rejected response) against frozen reference πref without a separate reward model.Its loss uses β to control deviation from the reference policy.Source: §2 · p. 3

Group Relative Policy Optimization (GRPO)

• GRPO efficiently updates policy πθ by maximizing a clipped objective over K sampled outputs, without an external value model.The clipping range is ϵ ∈ (0, 1).Source: §2 · p. 3
• GRPO omits the KL-divergence penalty and standardizes each output’s scalar reward within its sampled group to compute its advantage.The advantage is Ai = (r(oi) − µr)/σr.Source: §2 · p. 3

3 The G-Zero Framework

• Each round trains the Proposer to find challenging queries and informative hints, curates response pairs with Hint-δ, then updates the Generator with DPO for continued self-evolution.The Generator’s improved responses become the starting point for the next round.Source: §3 · pp. 3–4

3.1 The Intrinsic Learning Signal: Hint-δ

• Hint-δ is a per-token mean log-likelihood shift that is large when a difficult query and informative hint jointly reshape the Generator’s response distribution.Per-token normalization makes the signal invariant to the unassisted response’s length, while maximizing it targets the Generator’s blind spots without an external difficulty signal.Source: §3.1 · p. 4

3.2 Proposer Training

• The Proposer uses GRPO with Hint-δ as its reward to generate challenging query-hint pairs, while length and BLEU-duplication penalties discourage verbosity and repetition.The length penalty uses λ = 0.03 and penalizes hints beyond approximately 200 characters; question pairs with BLEU above 0.5 are clustered for duplication penalties.Source: §3.2 · pp. 4–5

3.3 Generator Training and Dataset Curation

• For DPO, the Generator prefers hint-assisted responses over unassisted ones, retaining the lower half of freshly sampled δ scores to favor fine-grained learning while limiting out-of-distribution updates.The curated pairs teach the Generator to reproduce hint-guided structural and stylistic improvements independently, without hints at inference.Source: §3.3 · pp. 5–6

3.4 Theoretical Analysis

• With sufficient post-filter exploration and low pseudo-label noise, Theorem 1 guarantees a best iterate using eO(d^2/ε^2) retained samples.The analysis attributes low noise to reliable preference pairs and exploration to coverage of feature directions the Generator has not learned.Source: §3.4 · p. 6

4.1 Experimental Setup

• Experiments tested Qwen3-8B-Base and Llama-3.1-8B-Instruct on AIME24/25, IFEval, and AlpacaEval 2.0.Reasoning uses mean@32 over 32 responses at temperature 0.7; IFEval uses greedy decoding and four standard accuracies; AlpacaEval reports length-controlled win rate against GPT-4-Turbo, judged by Qwen3-235B-A22B-Instruct-2507.Source: §4.1 · p. 6

4.2 Main Results

• After two rounds, average absolute performance rose from 33.95% to 35.43% on Qwen3-8B-Base and from 42.77% to 43.90% on Llama-3.1-8B-Instruct.The reported gains compound iteratively, with Llama-3.1’s modest Round 1 gain amplifying by Round 2.Source: Table 1 · p. 7, §4.2 · p. 7

4.1 Experimental Setup

• The δ-based data filter is supplemented with heuristic checks to remove DPO artifacts before forming the final dataset.Checks remove length inflation when lw/ll > 2.5, responses outside 100–10,000 characters, compression ratios below 0.15, prompt echoing with prefixes of at least 30 characters, and raw role markers.Source: §4.1 · p. 7

4.2 Main Results

• G-Zero targets different bottlenecks by model: Qwen3 improves in hard reasoning and strict formatting, while Llama-3.1 shows its largest gain in conversational alignment.AIME25 rises from 7.19% to 12.40% and IF-iS from 56.00% to 57.92% on Qwen3; AlpacaEval LC rises from 24.12% to 27.86% on Llama-3.1.Source: §4.2 · p. 7
• R-Zero’s targeted gains trade off other capabilities, whereas G-Zero R2 improves six Qwen3 metrics and every Llama-Instruct metric.R-Zero’s Qwen3 conversational performance falls from 8.94% to 8.04% and IFEval-pS from 43.07% to 37.56%; its Llama-3.1 average falls from 42.77% to 40.89%. G-Zero R2’s only Qwen3 regression is AlpLC (−0.47), while Llama-Instruct AlpLC gains +3.74.Source: §4.2 · p. 7

Structural Transfer from Non-Verifiable Tasks

• Non-verifiable tasks make up over 70% of the final DPO pool, versus less than 19% for math and code, while the highest δ signals come from structured writing and detailed explanations.The analysis attributes reasoning improvements to transferring logical depth and compositional complexity from the Challenger’s trajectories into mathematical problem-solving, rather than domain-specific memorization.Source: §5 · p. 8

The Necessity of the Lower-Half Filter

• The [0, 50] filter is balanced, not dominant: [0, 100] slightly improves IFEval and Chat but costs 1.20 pp Math, while [20, 80] raises Math but lowers IFEval.The upper half [50, 100] lowers IFEval from 52.78 to 51.97 while AlpacaEval LC reaches its highest value, 9.68; the surrounding [0, 50] ± 30 pp band gives broadly comparable averages.Source: §5 · p. 8

Capability Scaling Dynamics

• At N = 100, Math gains +1.24 toward +2.97 at N = 730, while Round 2 reaches +2.96; IFEval reaches +1.22 only in Round 2.Incremental IFEval rises from −0.96 to +0.25 at N = 730, while AlpacaEval LC stays within [+0.13, +0.38]; Round 2 ends at −0.47 pp on AlpacaEval LC and dominates incremental scaling on IFEval and the overall metric.Source: §5 · p. 8, Figure 3 · p. 9

The Shifting Distribution in Different Rounds

• Round 2 shifts the Hint-δ distribution rightward and raises its median, as the Proposer adapts to a stronger Generator by synthesizing more complex, impactful hints.The Proposer’s GRPO reward rises as it uncovers newly elevated blind spots, raising the difficulty ceiling and preventing Generator stagnation across iterations.Source: §5 · pp. 8–9, Figure 4 · p. 9

6.1 Self-Evolving Language Models

• Self-evolution progressed from self-generated fine-tuning to iterative self-play, multi-role co-evolution [15] [1], and dynamic self-challenging [40] [14] [17] and unsupervised post-training [30].Cross-verification [4] [22] mitigates feedback saturation; the field has shifted from supervised imitation [36] toward intrinsically verifiable, zero-data frameworks [10] [37] [8] [13], though filtering is needed to reduce synthetic-output model-collapse risk.Source: §6.1 · p. 9

6.2 Verifier-Free RL

• Verifier-free methods replace explicit rules with intrinsic generation-based rewards, progressing from reference-answer likelihoods [39] [18] to smoother dense rewards [12] [35] and internal step-wise signals [32] [19].These approaches target open-ended domains; likelihood-based refinements reduce gradient variance and help prevent reasoning degradation such as chain-of-thought shortening.Source: §6.2 · p. 9

7 Conclusion

• G-Zero uses Hint-δ to measure query difficulty and hint informativeness, letting a Proposer target Generator blind spots and the Generator internalize hints without external judges.The conclusion presents this continuous loop as a way for LLMs to self-improve in open-ended, unverifiable domains.Source: §7 · p. 9
• The conclusion argues that intrinsic feedback can enable autonomous self-evolution and self-aligning systems independent of human ground truth.This is the paper’s stated implication of deriving supervision from internal distributional dynamics.Source: §7 · p. 9

A Prompts and Templates

• The Proposer configuration uses Qwen3-8B-Base by default, temperature 1.0, and a maximum of 8,192 tokens.The passage identifies this as the base policy configuration.Source: A Prompts and Templates · p. 13
• The Proposer prompt requests one challenging general-domain user request and a short, useful hint, with broad task diversity and little emphasis on math.Requests should be self-contained and non-trivial; hints should guide the approach without giving away the answer, and both blocks use explicit tags.Source: A Prompts and Templates · p. 13
• For dual-response sampling, the Generator uses a round-R checkpoint, temperature 0.7, and a 16,384-token limit for no-hint and hinted responses.The first round uses the base model; the templates pair the question with no hint or with a hint.Source: A Prompts and Templates · p. 14
• AIME24/AIME25 evaluation uses mean@32 with 32 samples at temperature 0.7, while IFEval uses greedy decoding and a rule-based verifier.Both evaluations use the trained Generator with a 16,384-token maximum; AIME prompts request step-by-step reasoning and a boxed final answer.Source: A Prompts and Templates · p. 14
• The AlpacaEval LC judge is Qwen3-235B-A22B-Instruct-2507, runs at temperature 0.0 with a 32-token limit, and randomizes model position per pair.The randomized m/M ordering is intended to mitigate residual position bias.Source: A Prompts and Templates · p. 14
• The AlpacaEval prompt asks the judge to compare outputs for a given instruction and return only the identifier of the best model, m or M.The template supplies the instruction and both model outputs to the judge.Source: A Prompts and Templates · p. 15

B Pseudo-Code

• The Proposer samples query–hint rollouts, scores frozen-Generator responses with per-token mean Hint-δ plus structural penalties, and updates via group-relative advantages.This is Phase 1 of each co-evolutionary round.Source: B Pseudo-Code · p. 15
• For Generator training, the loop samples unassisted and hint-assisted responses, computes Hint-δ, retains the lower 50% of shifts, and applies quality heuristics to curate DPO pairs.The assisted response is the preferred answer and the unassisted response is the rejected answer.Source: B Pseudo-Code · p. 15
• The Generator is updated with DPO on the curated dataset against a frozen snapshot reference, and both updated models proceed to the next round.The loop returns the updated Proposer and Generator.Source: B Pseudo-Code · p. 15

• Table 4 lists the full G-Zero configuration used throughout, with values treated as defaults unless otherwise noted.Source: C Hyperparameters and Configuration · p. 15, Table 4 · p. 16

D Proof of Theorem 1

• At each round, an adaptive helper supplies question–hint pairs, the generator samples unassisted and hint-assisted responses, and a δ-filter retains data for DPO.The retained tuples are sampled conditionally independently given the past; the proof focuses on the retained distribution.Source: D Proof of Theorem 1 · p. 16
• The analysis assumes bounded linear rewards and features, Bradley–Terry clean preferences, a Gibbs policy class, δ-certified score noise, and exact cumulative standard-DPO updates.The retained-data noise condition permits wrong pseudo-labels but constrains their concentration in low-coverage feature directions.Source: D Proof of Theorem 1 · p. 17
• Helper coverage requires retained prompts to dominate the target distribution and generated question–hint pairs to expose feature directions where the current generator differs from the target policy.CQ measures target-distribution mismatch, while αS measures directional exploration.Source: D Proof of Theorem 1 · p. 17
• With probability at least 1 − ζ − ζδ, Theorem 1 guarantees a best-iterate target-suboptimality bound under Assumptions 1–6.The proof uses an elliptical-potential argument to obtain the best-iterate bound; the supplied passage does not show its formula.Source: D Proof of Theorem 1 · pp. 17–18
• The proof derives cumulative standard-DPO confidence, converts parameter confidence into target-distribution suboptimality, and applies an elliptical-potential argument to bound the best iterate.The confidence result holds under Assumptions 1, 2, 4, and 6; target suboptimality follows for every iterate on that event.Source: D Proof of Theorem 1 · pp. 18–19

E Limitation

• Reported results stop at R = 2: an exploratory Llama-3.1-8B-Instruct R3 run collapsed when the Phase 2 quality filter rejected all 1,994 candidate pairs for too-short responses.The R2-trained Generator's responses fell below the chosen_min_chars bound.Source: E Limitation · p. 19
• Reported runs consumed about US$2,000 in Tinker compute, making each Table 1 cell a single end-to-end run; three-seed reporting would roughly triple the cost.Tighter error bars, especially for AIME24/25 with n = 30 problems and 1σ ≈8 pp at p ≈0.3, are left for future work.Source: E Limitation · p. 19
• The collapse is attributed partly to reward hacking: idiosyncratic hints can raise Hint-δ without genuinely informative guidance, while length-normalized DPO mechanically favors shorter chosen responses.The authors leave multi-round-stable variants for future work.Source: E Limitation · pp. 19–20

Broader Impacts

• G-Zero could lower financial and computational barriers to AI alignment by replacing human labeling and proprietary judge APIs with intrinsic predictive signals.The authors say this could enable open-source communities and academic institutions to develop advanced reasoning models without centralized, closed-source infrastructure.Source: Broader Impacts · p. 20
• G-Zero demonstrates a path toward scalable oversight in which models identify and rectify logical blind spots without human-curated ground truth.The authors motivate this path by noting the increasing difficulty of external supervision as AI systems approach or surpass human expertise in complex tasks.Source: Broader Impacts · p. 20
• Internal distributional shifts could cause value drift, prioritizing structural depth over general helpfulness or safety alignment.The authors recommend final safety validation or lightweight human-in-the-loop checkpoints as mitigations.Source: Broader Impacts · p. 20
• Autonomous refinement in open-ended domains could enable harmful outputs such as sophisticated social engineering or cyberattack scripts.The authors recommend applying the framework to already safety-aligned base models and using output filtering to keep self-improvement within ethical boundaries.Source: Broader Impacts · p. 20

F Case Study

• In the Qwen3-8B-Base example, the hint prompts structural improvements, and the assisted response applies them while the unassisted response uses a generic template.The hint specifies an anecdote or statistic, investment-not-cost framing, and measurable outcomes.Source: Figure 5 · p. 21
• In the second example, the assisted response offers and explains one sustainability-and-comfort slogan, while the unassisted response gives ten generic options without commentary.The hint asks for emotionally resonant language; the assisted response names the brand EcoGentle.Source: Figure 6 · p. 22