PaperlayerView exact revision on arXiv

Source-linked AI summary

Search Self-play: Pushing the Frontier of Agent Capability without Supervision

Hongliang Lu, Yuhang Wen, Pengyu Cheng, Ruijin Ding, Jiaqi Guo, Haotian Xu, Chutian Wang, Haonan Chen, Xiaoxi Jiang, Guanjun Jiang

arXiv:2510.18821v3cs.LG

TL;DR

RLVR for LLM agents is difficult to scale because it depends on human-crafted, validated tasks and ground-truth answers, while synthetic tasks offer limited difficulty control. The paper introduces Search Self-play, where one LLM proposes and solves deep-search queries and uses proposer-collected results for RAG verification. SSP consistently improves search-agent performance across benchmarks in both from-scratch and continuous training, without external human supervision, though Proposer-Only training underperforms on complex multi-hop datasets.

  • Problem

    RLVR-based agent training depends on labor-intensive task queries and ground-truth answers, while existing synthesis methods have limited scalability and difficulty control.

  • Method

    Search Self-play trains one LLM as both a progressively challenging question proposer and a problem solver, using proposer-collected search results for RAG-based question verification.

  • Results

    SSP consistently improves search-agent performance across benchmarks and training setups, including a 26.4-point average gain for Qwen2.5-7B-Base and state-of-the-art results on five of seven benchmarks for Qwen2.5-32B-Instruct.

  • Takeaways & Limitations

    Self-play provides a scalable, data-efficient paradigm for improving agentic LLMs without external human supervision.

  • Takeaways & Limitations

    Proposer-Only training remains weaker on complex multi-hop reasoning than Solver-Only training, despite partial recovery on simpler benchmarks.

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) has become the mainstream technique for training LLM agents. However, RLVR highly depends on well-crafted task queries and corresponding ground-truth answers to provide accurate rewards, which requires significant human effort and hinders the scaling of RL processes, especially in agentic scenarios. Although a few recent works explore task synthesis methods, the difficulty of generated agentic tasks can hardly be controlled to provide effective RL training advantages. To achieve agentic RLVR with higher scalability, we explore self-play training for deep search agents, in which the learning LLM utilizes multi-turn search engine calling and acts simultaneously as both a task proposer and a problem solver. The task proposer aims to generate deep search queries with well-defined ground-truth answers and increasing task difficulty. The problem solver tries to handle the generated search queries and output the correct answer predictions. To ensure that each generated search query has accurate ground truth, we collect all the searching results from the proposer's trajectory as external knowledge, then conduct retrieval-augmentation generation (RAG) to test whether the proposed query can be correctly answered with all necessary search documents provided. In this search self-play (SSP) game, the proposer and the solver co-evolve their agent capabilities through both competition and cooperation. With substantial experimental results, we find that SSP can significantly improve search agents' performance uniformly on various benchmarks without any supervision under both from-scratch and continuous RL training setups. The code is at https://github.com/Qwen-Applications/SSP.

1. Introduction

Training LLM agents is difficult because supervised trajectories are scarce and tool-specific, while RLVR still depends on validated task queries and answers. Search Self-play addresses this by having one LLM generate and solve progressively difficult deep-search tasks, with RAG-based verification, and improves performance without human supervision.

  • Training Challenges: Supervised agentic training data is scarce, and trajectories for one tool set may not transfer to agents using another tool set.This makes human annotation especially difficult for heterogeneous LLM agents.
  • Training Challenges: RLVR reduces reliance on supervised data, but synthetic query methods still require rigorous question-answer validation that limits scalability.Existing synthesis methods can control difficulty through recursive information replacement, but validation remains a bottleneck.
  • Self-play Motivation: Self-play methods provide a supervision-free route to improving agent capabilities by learning from outcomes generated through interactions between agent roles.Prior self-play work motivates applying this paradigm to deep-search agents.
  • Search Self-play: Search Self-play makes the same LLM both a proposer of progressively difficult deep-search queries and a solver that answers them through multi-turn search.The proposer and solver roles are trained within one self-play game.
  • Search Self-play: RAG verification uses the proposer’s search results to test whether the solver can answer each generated query with the necessary information, preserving reward accuracy without human verification.The design combines autonomous task generation with answerability checking.

2. Related Work

Deep search agents extend RAG with multi-hop, dynamic, self-guided exploration, but prior task-generation and self-play methods do not adequately support adaptive agentic training. Search Self-play addresses this by coupling task generation and solving during training.

  • Deep Search Agents: Deep search agents use search engines and LLM reasoning for multi-turn retrieval, query reformulation, and exploration of complex questions.Their behavior differs from traditional one-pass RAG pipelines.
  • Deep Search Agents: Offline deep-search pipelines cannot adaptively control task difficulty to provide more effective RL advantages.The paper motivates self-play as a way to generate and solve tasks within the training process.
  • Search Self-play: Search Self-play co-evolves task-proposing and problem-solving abilities through self-generated tasks, reducing reliance on human annotation.The approach is designed for broader agentic scenarios than methods limited to internal model knowledge.
  • Prior Self-play: Earlier language-model self-play studies improve reasoning in simpler or offline settings, but do not address tool-using agentic environments.The cited word-based game remains confined to offline RL updates.

3. Methodology

Search Self-play models deep-search training as a game between a proposer and solver sharing one LLM policy. It verifies generated questions with proposer-collected search evidence, filters valid tasks, and updates both roles with outcome-based reinforcement learning.

  • Agent Formulation: A search-agent trajectory interleaves LLM outputs with search-tool observations, and the agent is modeled as a token-level Markov decision process.States are language sequences, actions are vocabulary tokens, and transitions append generated tokens or tool observations.
  • Self-play Modeling: The proposer receives a ground-truth answer, uses a dedicated prompt to generate a question, and the solver uses another prompt to answer it.The same policy acts in both roles through distinct system prompts.
  • RAG Verification: Proposer search results are collected as RAG materials so the solver can verify whether the generated question is correct and answerable before unrestricted deep search.This cooperation constrains the proposer to questions supported by the evidence it gathered.
  • Optimization: Joint optimization of competition and cooperation is inefficient, so rejection sampling retains questions whose RAG-based solver answer matches the ground truth.The filtered valid questions are then used for the adversarial training objective.
  • Implementation: Training filters proposer outputs by format and quality rules, mixes unrelated trajectory documents for robust RAG verification, and updates the solver and proposer with GRPO-style and REINFORCE-based objectives.The proposer is rewarded when solver success is low, encouraging increasingly difficult questions.
  • Design Rationale: The proposer’s external search interactions and verifiable RAG pipeline distinguish SSP from methods that rely only on the LLM’s internal knowledge or majority voting.External-environment information supports question generation while RAG verifies correctness.

4. Experiments

Experiments show that SSP consistently improves search-agent performance across models, training settings, and scales, while co-evolution and RAG verification are important to stable training.

  • SSP consistently outperforms baseline counterparts across question-answering benchmarks and diverse models, training paradigms, and scales.
  • Main Results: Qwen2.5-7B-Base gains 26.4 average points with SSP, including +40.4 on TriviaQA, while Qwen2.5-7B-Instruct improves by 8.0 average points.
  • Main Results: SSP further improves search-specialized models and reaches state-of-the-art results on five of seven benchmarks with Qwen2.5-32B-Instruct.
  • Self-play versus Fixed-Opponent Training: Complete SSP achieves the highest average score, outperforming Solver-Only and Proposer-Only fixed-opponent training schemes.
  • Self-play versus Fixed-Opponent Training: Solver-Only reward saturates near 0.9, after which held-out NQ and 2Wiki performance initially rises and then declines.
  • Ablation on RAG Verification: RAG verification prevents noisy or incorrect questions from training the solver, while noisy documents make verification more robust against proposer hacking.

5. Conclusion

SSP combines proposer–solver self-play with RAG verification to improve deep-search agents without external supervision. Experiments show consistent gains across benchmarks in both from-scratch and continual training.

  • SSP makes the LLM both a question proposer and problem solver, using multi-turn search, verifiable answers, progressive difficulty, and RAG-based correctness checks.
  • SSP consistently enhances search-agent performance across diverse benchmarks under both from-scratch and continuous training without external human supervision.
  • The approach positions self-play as a scalable, data-efficient paradigm for agentic LLM training.

A.1. Training Hyperparameter

The experiments use a predefined answer set and binary solver rewards, with tool-integrated rollouts enforcing structured responses. The setup samples public training answers and uses RAG-based filtering to support stable question generation.

  • Rewards Design: Solver rewards are binary, assigning 1 when the predicted answer matches the ground-truth answer.The ground-truth answer is checked using an indicator function for equality.
  • Rollout Configuration: Tool-integrated rollouts require structured <think>, <search>, <answer>, and <question> tags, while format deviations receive no reward.Search results are used as external information during interactive reasoning and tool invocation.
  • Baselines: The experiments use Search-R1, ZeroSearch, and R-Search checkpoints as baselines for continuous reinforcement-learning training.Training starts from the best-performing checkpoints reported for each baseline.
  • Sampling and Verification: Dynamic sampling replenishes batches after filtering reduces valid questions, making rewards less sparse for more stable training.The default RAG solver for verification is Qwen2.5-32B-Instruct.
  • Answer Set: The answer set D contains 50,000 ground-truth answers sampled from public training data, with an average word length of 14.53.The set covers topics including people, time and dates, geography and places, music, and sports.

B.1. Ablation on Batch Sampling Strategies

Batch sampling strongly affects SSP training: full reuse improves data exposure but risks overfitting, while periodic replay-buffer resets provide the best balance between reuse and novelty.

  • Replay Buffer (Periodic Reset): Periodic buffer resets every 10 training steps balance efficient reuse with data novelty.This strategy is the configuration used to reproduce the main experimental results.
  • Alternative Strategies: Dummy padding yields the smallest improvement because invalid proposals create severe reward sparsity.Dynamic resampling performs slightly better by collecting full batches of novel valid questions, but repeated generation is computationally expensive.
  • Replay Buffer (Full Reuse): Full replay improves the average score from 42.4 to 44.4 but can overfit the static question pool and degrade proposer learning.Repeated solver training on old questions diminishes the proposer’s learning signal.
  • Results: 48.7 average score makes Replay Buffer (Periodic Reset) the best strategy across nearly all benchmarks.Periodic clearing preserves data novelty while retaining sufficient exposure to generated questions.

B.2. Training Dynamics of Solver

Under SSP training with periodic replay-buffer resets, the solver increases search and response length while evaluation accuracy improves on GeneralQA and Multi-HopQA before later plateauing.

  • Tool Use and Responses: The average number of search calls per trajectory steadily increases, indicating more extensive multi-step search behavior.The solver also produces longer responses while prompt length remains stable.
  • Accuracy: Evaluation scores on GeneralQA and Multi-HopQA continuously improve during SSP training.The improvement slope decreases in later training stages.
  • Limitation: A maximum of 10 search steps constrains later improvement by limiting deeper reasoning paths.The authors suggest that scaling this search-step constraint could unlock further performance gains.

B.3. Training Dynamics of Proposer

During SSP training, the proposer increases search use, generates progressively harder questions, and preserves broad topic coverage, while punitive format-error rewards destabilize question generation.

  • Search Behavior: The proposer’s search-tool utilization rises throughout training, supporting increasingly complex question scenarios.The analysis also reports improving question validation and quality dynamics.
  • Question Difficulty: The proposer progressively increases generated-question difficulty, exhibiting adaptive curriculum learning.Difficulty is assessed with DeepSeek-V3.2 using a structured evaluation prompt.
  • Topic Coverage: Balanced topic distributions preserve broad domain coverage and avoid bias toward specific question types.LDA clustering is used to analyze topical distributions across training phases.
  • Reward Sensitivity: A -0.1 penalty for invalid questions causes valid-question generation to collapse toward 0 and destabilizes self-play.Higher policy entropy makes outputs more random, while the solver’s apparent reward increase reflects overfitting to a static buffer.

B.5. Ablation Study on RL Algorithms

The ablation compares RL algorithms for proposer and solver roles, showing that solver-side GRPO offers the best practical trade-off while GRPO–GRPO yields only marginal gains at much higher cost.

  • 50.9 average accuracy with GRPO–GRPO exceeds RF–GRPO’s 49.5, but generation time increases from 83.4 s to 504.4 s per step.This is approximately 6× slower, making GRPO–GRPO the most expensive configuration despite its small accuracy advantage.
  • RF–GRPO achieves the best average performance among configurations other than GRPO–GRPO, reaching 49.5 accuracy at 83.4 s per step.The authors retain RF–GRPO as the default because it balances effectiveness and efficiency.
  • Solver-side GRPO is more effective than proposer-side GRPO, while RF–RF produces the lowest average performance at 42.0 despite requiring only 9.1 s per step.The results associate multi-trajectory credit assignment during solving with better solution quality, whereas proposer exploration alone does not improve outcomes reliably.
  • Using GRPO for the proposer and RF for the solver lowers average accuracy to 46.5, although its generation time is only 50.1 s per step.This configuration underperforms RF–GRPO despite being faster.

E. Hacking Question Cases

The hacking cases expose that SSP verification can accept questions whose answers are non-unique, especially when retrieval documents bias the solver toward one candidate.

  • A question asking for a “Temptations singer” is ambiguous because the group had multiple singers, so it does not determine a unique answer.The example can be answered with several historically valid names rather than one unambiguous target.
  • Biased RAG documents prominently featuring Otis Williams can make an ambiguous question appear to have a single correct answer.The retrieval context may steer the solver toward one member even though the question itself permits multiple answers.
  • A solver without those specific documents would likely struggle to identify which singer is intended, revealing a boundary of RAG-based verification.The verification process can succeed because of limited and biased retrieval rather than because the question is intrinsically well-posed.
Loading 2510.18821v3…