G-Zero: Self-Play for Open-Ended Generation from Zero Data

Chengsong Huang; Haolin Liu; Tong Zheng; Runpeng Dai; Langlin Huang; Jinyuan Li; Zongxia Li; Zhepei Wei; Yu Meng; Jiaxin Huang

Source-linked AI summary

G-Zero: Self-Play for Open-Ended Generation from Zero Data

Chengsong Huang, Haolin Liu, Tong Zheng, Runpeng Dai, Langlin Huang, Jinyuan Li, Zongxia Li, Zhepei Wei, Yu Meng, Jiaxin Huang

arXiv:2605.09959v1May 11, 2026cs.LGcs.AIcs.CLcs.ET

TL;DR

Open-ended self-evolution lacks objective verifiers and can be bottlenecked by proxy LLM judges. G-Zero uses Hint-δ-driven co-evolution, achieving gains on both open-ended and verifiable tasks while proving a conditional best-iterate guarantee.

Problem
Open-ended tasks lack objective oracles, so self-evolving models often rely on LLM judges whose capabilities can bottleneck optimization.
Method
G-Zero co-evolves a Proposer and Generator using Hint-δ, contrasting unassisted and hint-conditioned responses to create preference pairs without external verifiers.
Results
+3.74 points on AlpacaEval and +5.21 points on AIME 25 accompanied robust improvements across open-ended and verifiable tasks and a conditional best-iterate guarantee.
Takeaways & Limitations
G-Zero demonstrates that intrinsic distributional feedback can support LLM self-improvement across unverifiable and verifiable domains without external supervision.
Takeaways & Limitations
Optimizing Hint-δ is vulnerable to Proposer reward hacking through excessively verbose hints, requiring a length penalty.

Abstract

from arXiv · show

Self-evolving LLMs excel in verifiable domains but struggle in open-ended tasks, where reliance on proxy LLM judges introduces capability bottlenecks and reward hacking. To overcome this, we introduce G-Zero, a verifier-free, co-evolutionary framework for autonomous self-improvement. Our core innovation is Hint-$δ$, an intrinsic reward that quantifies the predictive shift between a Generator model's unassisted response and its response conditioned on a self-generated hint. Using this signal, a Proposer model is trained via GRPO to continuously target the Generator's blind spots by synthesizing challenging queries and informative hints. The Generator is concurrently optimized via DPO to internalize these hint-guided improvements. Theoretically, we prove a best-iterate suboptimality guarantee for an idealized standard-DPO version of G-Zero, provided that the Proposer induces sufficient exploration coverage and the data filteration keeps pseudo-label score noise low. By deriving supervision entirely from internal distributional dynamics, G-Zero bypasses the capability ceilings of external judges, providing a scalable, robust pathway for continuous LLM self-evolution across unverifiable domains.

1 Introduction

G-Zero extends self-evolution beyond verifiable domains through verifier-free co-evolution between a Proposer and Generator, using Hint-δ as an intrinsic supervision signal. The framework is theoretically characterized and empirically validated across open-ended and verifiable tasks.

Motivation: G-Zero targets open-ended instruction following, dialogue, and creative writing, where objective programmatic oracles required by RLVR are unavailable.This avoids dependence on deterministic correctness signals that support self-evolution in mathematics and code generation.
Method: Hint-δ measures how much a self-generated hint shifts the Generator’s predictive distribution, creating an internal preference signal without external verifiers or judges.Hint-conditioned responses are preferred over unassisted responses, allowing supervision to arise from internal distributional dynamics.
Theory: The idealized standard-DPO variant has a best-iterate suboptimality guarantee when Proposer-induced coverage is sufficient and δ-certified pseudo-label score noise is low.These conditions control the theoretical bound for the co-evolutionary loop.
Empirical validation: +3.74 points on AlpacaEval demonstrates a substantial open-ended gain after several self-play iterations, alongside improvements on verifiable tasks.The introduction describes these gains as robust improvements within several self-play iterations.
Method: The Proposer generates query–hint pairs and receives Hint-δ as a GRPO reward to explore the Generator’s blind spots, while the Generator learns from hinted versus unhinted responses.The co-evolutionary loop alternates Proposer exploration with Generator improvement using internally generated supervision.

2 Preliminaries: Optimization Objectives

This section introduces DPO and GRPO as the optimization objectives used for preference alignment and reinforcement learning. DPO uses a frozen reference policy, while GRPO avoids an external value model and computes standardized group-relative advantages.

Direct Preference Optimization (DPO): DPO aligns a language-model policy with preference triples by optimizing against a frozen reference model without a separate reward model.The triples contain an input prompt, a preferred response, and a rejected response.
Direct Preference Optimization (DPO): The DPO objective uses a logistic function, with β controlling the policy’s deviation from the reference policy.
Group Relative Policy Optimization (GRPO): GRPO updates a policy from groups of K sampled outputs using a clipped objective, without requiring an external value model.The clip range is ϵ ∈(0, 1).
Group Relative Policy Optimization (GRPO): GRPO omits the KL-divergence penalty and standardizes scalar rewards within each sampled group to compute the advantages.This formulation follows prior work [20].

3 The G-Zero Framework

G-Zero is a verifier-free, iterative co-evolution framework that generates preference pairs from intrinsic hint-induced distribution shifts. A Proposer targets difficult queries while a Generator internalizes hint-guided improvements through filtered DPO updates, with theoretical progress depending on exploration and low pseudo-label noise.

Framework: G-Zero alternates Proposer training with dataset curation and Generator DPO, using hint-conditioned responses as preferred over unassisted baselines for continuous self-evolution.The improved Generator becomes the base model for the next round, while preference pairs are constructed without external verifiers.
Hint-δ: Hint-δ measures how strongly a self-generated hint shifts the Generator’s predictive distribution over its own unassisted response, jointly signaling query difficulty and hint informativeness.A large δ requires both a challenging query and a hint supplying missing knowledge or reasoning; per-token averaging removes direct response-length inflation.
Proposer Training: The Proposer is trained with GRPO to generate challenging query–hint pairs, while length and BLEU-duplication penalties limit verbose reward hacking and repetitive exploration.The reported settings use λ = 0.03, a hint budget of approximately 200 characters, and a BLEU merge threshold of 0.5.
Dataset Curation and Generator Training: The Generator is updated with reference-anchored DPO on curated pairs, selecting hint-assisted responses over unassisted responses to internalize useful structure and reasoning without explicit hints at inference.Curation resamples both responses and retains pairs whose δ lies in the lower half of that round’s empirical distribution.
Theoretical Guarantee: The informal guarantee requires sufficiently exploratory retained data and low pseudo-label noise, yielding a high-probability best-iterate bound after iterative DPO.Hint-δ and filtering control data quality, while exploration controls coverage of feature directions not yet learned by the Generator.

4 Experiments

Experiments evaluate G-Zero across base and instruction-tuned 8B models using reasoning, instruction-following, and conversational benchmarks. Across two rounds, G-Zero improves overall capability without external supervision, targets model-specific bottlenecks, and avoids the cross-capability trade-offs observed with R-Zero.

Experimental setup: The evaluation uses Qwen3-8B-Base [33] and Llama-3.1-8B-Instruct [5], covering AIME24/AIME25, IFEval [38], and AlpacaEval 2.0 [3].Reasoning uses mean@32 from 32 responses at temperature 0.7; IFEval uses greedy decoding and four standard metrics, while AlpacaEval reports length-controlled win rate against GPT-4-Turbo.
Main results: After two rounds, average absolute performance rises from 33.95% to 35.43% on Qwen3-8B-Base and from 42.77% to 43.90% on Llama-3.1-8B-Instruct.Table 1 reports absolute performance after rounds 1 and 2, with improvements compounding iteratively for Llama-3.1.
Experimental setup: Training settings are standardized across the iterative loop, using the Tinker API and Low-Rank Adaptation (LoRA) [9].Data curation also removes likely DPO artifacts, including response pairs with length inflation above lw/ll > 2.5 and responses outside absolute length bounds.
Capability-specific gains: G-Zero targets different bottlenecks by model: Qwen3 improves on AIME25 from 7.19% to 12.40% and IF-iS from 56.00% to 57.92%, while Llama-3.1’s AlpacaEval LC rises from 24.12% to 27.86%.The foundational model shows prominent gains in hard reasoning and strict formatting, whereas the instruction-tuned model shows its largest gain in conversational alignment.
Comparison with R-Zero: Compared with R-Zero, G-Zero preserves broader capability performance: R-Zero’s Qwen3 conversational score falls from 8.94% to 8.04%, IFEval-pS from 43.07% to 37.56%, and Llama-3.1’s 7-metric average from 42.77% to 40.89%.R-Zero gains on Qwen3 AIME24 from 10.42% to 14.92% but trades off other capabilities, whereas G-Zero R2 keeps per-metric movements small.

5 Analysis

G-Zero’s training pool is dominated by non-verifiable tasks, while capability gains depend on task-specific scaling and filtering choices. Across rounds, the Proposer adapts to the stronger Generator by generating increasingly challenging hints, sustaining co-evolutionary pressure.

Structural Transfer from Non-Verifiable Tasks: Over 70% of the final DPO pool comes from non-verifiable tasks, versus less than 19% from math and code, with the highest δ signals from structured writing and detailed explanations.The distribution supports structural transfer from open-ended tasks rather than relying primarily on verifiable data.
The Necessity of the Lower-Half Filter: The [0, 50] δ filter is supported by a dichotomy in which low-δ pairs provide structural or logical refinements, while high-δ pairs risk answer leakage; the observed magnitudes are modest.Retaining the upper half [50, 100] trades away the intended refinement signal according to the alternative-strategy analysis.
Capability Scaling Dynamics: Round 2 dominates incremental scaling on IFEval and the overall metric, whereas the two schedules coincide on Math and AlpacaEval LC remains within [+0.13, +0.38].IFEval recovers from −0.96 at N = 100 to +0.25 at N = 730, while Round 2 ends at −0.47 pp on AlpacaEval LC.
Capability Scaling Dynamics: Math gains reach +2.97 at N = 730, with +1.24 already at N = 100, while Round 2 reaches the same final value (+2.96).This indicates rapid saturation and that the Generator can absorb structural reasoning with relatively little data.
The Shifting Distribution in Different Rounds: The Hint-δ distribution shifts rightward with a higher median from Round 1 to Round 2, as the Proposer synthesizes increasingly impactful hints for the stronger Generator.This co-evolutionary arms race continually exposes harder blind spots, raises the difficulty ceiling, and helps prevent Generator stagnation.

6 Related Work

Related work traces LLM self-evolution from self-generated supervised trajectories through iterative self-play and multi-role co-evolution toward dynamic self-challenging and unsupervised post-training. Verifier-free methods likewise seek intrinsic rewards from generation processes to reduce reliance on explicit rules in open-ended domains.

Self-evolution: LLM self-evolution progressed from high-confidence self-generated trajectory fine-tuning to iterative self-play and multi-role co-evolution [15] [1].Cross-verification [4] [22] was introduced to mitigate feedback saturation, followed by dynamic self-challenging [40] [14] [17] and unsupervised post-training [30].
Verifier-free learning: Verifier-free research addresses RLVR’s reliance on explicit rules by extracting intrinsic rewards directly from the generation process for open-ended domains.Foundational methods [39] [18] bypass external verifiers by optimizing the conditional probability of reference answers.
Verifier-free learning: Likelihood-based verifier-free designs are refined to stabilize training and prevent reasoning degradation such as chain-of-thought shortening.

7 Conclusion · A Prompts and Templates · B Pseudo-Code

G-Zero enables verifier-free self-improvement in open-ended domains through the internal Hint-δ signal and a co-evolutionary Proposer–Generator loop. The appendices specify prompt templates, evaluation configurations, and the two-phase update algorithm.

7 Conclusion: G-Zero enables LLM self-improvement in open-ended, unverifiable domains without external judges, using Hint-δ to measure query difficulty and hint informativeness.The Proposer targets model blind spots, while the Generator internalizes the resulting hints through continuous self-evolution.
A Prompts and Templates: Proposer generation uses a base policy, temperature 1.0, and an 8,192-token limit, with no system message.The output must contain only <question> and <hint> tagged blocks; examples cover resignation writing and Kalman-filter explanation.
A Prompts and Templates: The Proposer prompt requests one challenging, general-domain user query and a short hint, emphasizing non-mathematical tasks whose quality depends on clarity, tone, structure, audience fit, or creativity.It requires self-contained, non-trivial requests and hints that guide rather than reveal the answer.
B Pseudo-Code: In Phase 1, GRPO trains the Proposer from rollout queries and hints, frozen-Generator responses, per-token Hint-δ signals, structural penalties, and group-relative advantages.The procedure updates πP by maximizing each rollout’s group-relative advantage.
B Pseudo-Code: In Phase 2, the Generator samples unassisted and assisted responses, retains the lower 50% of the Hint-δ distribution, applies quality checks, and forms preference pairs.The assisted response is preferred over the unassisted response for DPO training.
B Pseudo-Code: The Generator is updated with DPO against a frozen reference snapshot, and both updated models are returned for the next co-evolutionary round.The curated dataset uses x=q, yw=a_assisted, and yl=a_hard.

C Hyperparameters and Configuration

This section specifies the default G-Zero configuration used throughout the experiments, with the full hyperparameter settings provided in Table 4.

C Hyperparameters and Configuration: Table 4 provides the full set of default G-Zero hyperparameters used throughout the experiments, unless otherwise noted.The listed values are defaults unless an experiment explicitly indicates otherwise.

D Proof of Theorem 1

The proof analyzes an idealized standard-DPO version of G-Zero, where an adaptive helper supplies question–hint pairs and a δ-filtered retained distribution drives generator updates. Under bounded linear rewards, Bradley–Terry preferences, exploratory coverage, and exact cumulative DPO, it establishes a high-probability best-iterate guarantee on the target prompt distribution.

Helper-induced data: The analysis models each round by retaining helper-generated question–hint–response tuples through a measurable δ-filter, then sampling a conditionally independent batch for DPO.The retained prompt marginal ρt may differ from the target distribution Q, with the mismatch controlled by CQ.
Coverage assumptions: Exploration is formalized by requiring retained helper prompts to dominate Q and expose feature directions where the current generator differs from the target policy.CQ measures target-distribution mismatch, while αS measures exploratory coverage of relevant feature directions.
Theorem 1: Under Assumptions 1–6, with probability at least 1 − ζ − ζδ, some iterate t0 ≤ T satisfies the theorem’s target-distribution guarantee.The result relies on helper coverage, bounded features and linear rewards, Bradley–Terry clean preferences, controlled pseudo-label noise, and exact cumulative standard-DPO updates.
Proof strategy: The proof proceeds from coverage transfer and self-normalized clean-logistic concentration to a cumulative-DPO confidence bound, then converts parameter confidence into target suboptimality.The final step uses an elliptical-potential argument over helper-generated batches to obtain a best-iterate result.

E Limitation

G-Zero’s reported runs use roughly US$2,000 of Tinker compute, with each Table 1 cell representing a single end-to-end run. The method also exhibits reward hacking and length collapse, motivating multi-round-stable variants as future work.

E Limitation: Roughly US$2,000 of Tinker compute supports all reported runs, and every Table 1 cell is a single end-to-end run.The total covers two models, two rounds each, cutoff and Phase 1 ablations, a data-scaling sweep, and a temperature-0.7 AIME re-evaluation.
E Limitation: After accessible structural improvements are internalized, the Proposer can reward-hack Hint-δ with idiosyncratic hints that strongly shift log πG(a_hard) without genuinely informative guidance.This pathway is identified as a contributor to the observed limitation.
E Limitation: Length-normalized DPO further amplifies length collapse because shortening the chosen response mechanically raises the average per-token log-ratio.Multi-round-stable variants remain an open direction for future work.

Broader Impacts

G-Zero is presented as a verifier-free framework for autonomous LLM self-evolution with potential benefits for accessible AI alignment and scalable oversight. Its reliance on internal distributional shifts also raises value-drift and dual-use concerns, motivating safety validation and lightweight human oversight.

Broader Impacts: G-Zero’s broader implications center on verifier-free autonomous LLM self-evolution.The framework is introduced as a basis for the paper’s broader-impact discussion.
Positive Societal Impacts: Intrinsic predictive signals could lower the financial and computational barriers to AI alignment by reducing reliance on human labeling and proprietary judge APIs.This could enable open-source communities and academic institutions to develop advanced reasoning models without centralized, closed-source infrastructure.
Potential Risks and Mitigations: Internal-distribution-driven self-evolution may cause value drift, prioritizing complex structural depth over general helpfulness or safety alignment.The proposed mitigation is a final safety-validation phase or lightweight human-in-the-loop checkpoints.
Potential Risks and Mitigations: The framework also presents potential dual-use risks associated with autonomous capability development.The supplied passage identifies dual-use risks but does not provide the remainder of its mitigation discussion.

F Case Study

The case study illustrates Hint-δ guiding G-Zero toward targeted improvements in open-ended writing tasks. In two examples, assisted responses apply the hints more specifically than generic alternatives.

Case Study: For a non-profit funding pitch, Hint-δ = +0.056 directs the response toward an engaging opener, investment framing, and measurable outcomes.The query targets an education-focused, business-oriented audience seeking a concise, conversational pitch.
Case Study: In Figure 5, the assisted response applies all three structural improvements, whereas the hard response defaults to a generic template.The improvements are an anecdote or statistic, investment-not-cost framing, and measurable outcomes.
Case Study: For an eco-friendly diaper slogan, Hint-δ = +0.031 encourages balancing sustainability and comfort with emotionally resonant, nature-connected language.The intended audience is environmentally conscious parents aged 25–40, and the wording should avoid being overly technical or abstract.
Case Study: In Figure 6, the assisted response commits to one branded slogan and explains its fit, while the hard response produces ten generic options without commentary.The assisted slogan uses the brand name EcoGentle and addresses the brief’s sustainability–comfort balance.