Source-linked AI summary
The Chase Is the Curriculum, the Capture Anchors the Credit: Pursuit-Evasion Self-Play for Zero-Data LLM Reasoning
Jing Yu, Shengchao Chen, Yiyun Tan
TL;DR
Existing zero-data self-play still lacks adaptive task placement and dense verifier-derived credit for hard, self-generated reasoning tasks. LURE recasts training as pursuit-evasion, positioning tasks at a learned half-capture frontier and giving the solver capture-anchored process credit; across three environments and backbone families, it outperforms baselines and achieves stronger aggregate OOD zero-shot accuracy across nine benchmarks.
Problem
Existing zero-data self-play uses post-hoc rejection with fixed difficulty bands and sparse terminal rewards, leaving task placement non-adaptive and solver credit underinformed as competence changes.
Method
LURE trains a challenger to place tasks in a pursuit-evasion game, using a half-capture frontier and verifier-derived capture-anchored process credit for the planner-executor solver.
Results
Across three verifiable reasoning environments and backbone families, LURE outperforms trained baselines and attains stronger aggregate OOD zero-shot accuracy across nine benchmarks, though gains are modest and benchmark-dependent.
Takeaways & Limitations
The findings support learned curriculum positioning and verifier-derived process credit as a zero-data self-play approach within the tested verifiable reasoning settings.
Takeaways & Limitations
Reliable multi-role self-play evaluation requires auditing all role-conditioned prompts because leakage can originate from a role-specific prompt rather than the benchmark answer key.
Abstract
from arXiv · showhide
Reinforcement learning with verifiable rewards has become the dominant recipe for improving large language model reasoning, yet it presumes large human-curated task collections. Zero-data self-play removes this dependency, but existing methods vet learnability only by probing candidates and rejecting post hoc, never learning where along an environment's difficulty axis to place a task, and credit the solver with sparse terminal rewards alone. We recast zero-data self-play as a pursuit-evasion game: in LURE, an LLM evader positions tasks along each environment's difficulty axis to stay one step ahead of a planner-executor pursuer that hunts it down through verifiable interaction. The evader is trained on a capture-frontier reward that peaks when the solver captures it on exactly half of its rollouts, turning barely catchable into a learned positioning strategy rather than a hand-tuned rejection band. The pursuer earns capture-anchored dense process credit, in which monotone verifier progress is group-normalized jointly with the terminal capture under a round-anchored KL that keeps the co-evolution stable. Across three verifiable reasoning environments and three backbone families, LURE outperforms advanced baselines under unified/specialist settings, while the unified model attains stronger aggregate OOD zero-shot accuracy than all trained baselines across nine held-out benchmarks from three task families.
Introduction
The introduction frames zero-data self-play as limited by implicit, non-adaptive task difficulty and sparse terminal credit. LURE addresses both through a pursuit-evasion game in which an evader learns task placement while a planner-executor solver receives verifier-based training signals.
- Motivation: RLVR trains language models on multi-step verifiable reasoning by optimizing policies directly against programmatic checkers.Target tasks include multi-hop question answering, instruction following, and logic-grid deduction.
- Limitations: Existing zero-data self-play authors tasks in text space and vets learnability post hoc with fixed empirical solve-rate bands, leaving difficulty placement implicit and non-adaptive.The band edges do not adapt as solver competence changes.
- Limitations: Sparse terminal rewards credit an entire multi-step trajectory as one block, while group-relative optimization can produce vanishing gradients when every rollout fails.Process reward models would reintroduce external supervision that self-play aims to avoid.
- LURE: LURE recasts zero-data self-play as pursuit-evasion: an LLM evader positions tasks while a planner-executor pursuer hunts them through verifiable reasoning environments.The two roles co-evolve without human data across three heterogeneous environments with shared per-role weights.
- LURE: Its capture-frontier objective learns difficulty positioning from solver group-capture statistics, replacing post-hoc rejection with a learned curriculum policy.The evader acts as a learned lure, holding tasks barely catchable so they pull the solver toward its frontier.
Related Work
Prior work makes multi-step reasoning explicitly verifiable and develops self-evolving curricula through synthesized supervision or proposer–solver games. However, task generation remains open, while R-Zero retains candidates using a fixed hand-set solve-rate band.
- Verifiable Multi-Step Reasoning: Verifiable reasoning exposes intermediate computation through chain-of-thought, step-level scoring, and reinforcement learning with programmatic checkers.These advances also motivate verifiable benchmarks for instruction following, logic-grid deduction, and multi-hop relational question answering.
- LLM Self-Evolving: Self-evolving training bootstraps curricula from synthesized instructions, outcome-filtered rationales, or proposer–solver games.This line of work frames curriculum generation as model-driven supervision or competition between a task proposer and a solver.
- LLM Self-Evolving: R-Zero co-evolves an LLM challenger that authors problem text but keeps candidates only within a fixed hand-set solve-rate band.Earlier verifiable reasoning work improves solving but leaves task generation open.
Methodology
LURE formulates zero-data reasoning as a pursuit-evasion game in which an evader selects task difficulty and a planner-executor pursuer solves verifier-generated tasks. Capture-frontier rewards, verifier-based dense credit, and round-anchored KL updates coordinate curriculum placement, process supervision, and stable co-evolution.
- Pursuit-evasion setup: The evader places generated tasks along each environment’s difficulty axis, while the planner-executor pursuer attacks them through G verifier-scored rollouts.Each environment provides a task generator, multi-turn interaction protocol, terminal capture, and running best-so-far progress signals.
- Frontier-seeking evader: The evader’s capture-frontier reward peaks when the pursuer’s empirical capture rate is p = 1/2, with repetition penalties preventing signature collapse.Unlike post-hoc rejection filtering, the evader learns where to position tasks relative to measured pursuer competence.
- Dense-credit pursuer: The pursuer receives group-normalized dense process credit from terminal capture and verifier progress increments, rather than relying on sparse terminal rewards alone.The terminal reward is assigned once on the final executor turn; zero-progress costs distinguish stalling, while best-so-far progress ensures cumulative increments telescope to peak verified progress.
- Dense-credit pursuer: Planner credit is separately group-normalized over executor-stage potentials, rewarding how far execution advanced rather than whether the task was captured.This gives planner turns their own verifier-derived learning signal without adding a terminal term.
- Stabilized co-evolution: The evader, planner, and executor each take one clipped GRPO update using advantages from their group-normalized rewards and KL penalties to round-start snapshots.Round-start snapshots generate all samples, anchoring updates to the data-producing policies during non-stationary co-evolution.
Experiments
LURE outperforms trained baselines across three verifiable reasoning environments and adapts task difficulty toward a 0.5 capture frontier. Its learned curriculum improves efficiency, robustness, scaling behavior, and unified-model OOD generalization.
- Main Results: LURE outperforms all trained baselines across PhantomWiki, IFEval, and ZebraLogic, exceeding R-Zero by 15.0 points on PhantomWiki.Comparisons include Base, curated-task GRPO, frozen-pool GRPO-Zero, a non-adaptive curriculum, and R-Zero.
- Ablation Study: The learned challenger is the largest ablation loss at −10.6, while curriculum and stabilization removals degrade performance more than any credit-term removal.Every component removal lowers the mean against the full system.
- Hyperparameter Sensitivity: Performance is most sensitive to β and G: removing round-anchored KL substantially reduces accuracy, excessively large β degrades performance, and reducing G causes a marked decline.The capture rate is estimated from G rollouts, making group size important for reliable frontier estimation.
- Curriculum Dynamics: Held-out success rises from approximately 27% to 65.8% as LURE lowers emitted task difficulty toward the target capture rate p = 0.5.R-Zero maintains approximately fixed emitted difficulty and achieves lower final capture and held-out performance.
Conclusion
LURE recasts zero-data self-play as a pursuit-evasion game in which a challenger places tasks and a planner-executor solver hunts them down. A capture-frontier reward and capture-anchored dense process credit jointly stabilize their co-evolution.
- Method: LURE frames zero-data self-play as pursuit-evasion between a task-placing challenger and a planner-executor solver.The solver hunts down tasks through the environment, while the challenger positions them along the difficulty axis.
- Method: The challenger’s capture-frontier reward peaks at a capture rate of one half.This trains task placement around a barely catchable capture frontier rather than relying on post-hoc rejection.
- Method: The solver receives capture-anchored dense process credit under a round-anchored KL that keeps both players stable.Verifier progress is credited jointly with terminal capture to support stable co-evolution.
- Results: Across three verifiable reasoning environments and backbone families, LURE outperforms baselines under unified and specialist settings.The supplied conclusion passage reports this cross-environment and cross-backbone result without specifying numerical values.