PaperlayerView exact revision on arXiv

Source-linked AI summary

AceSearcher: Bootstrapping Reasoning and Search for LLMs via Reinforced Self-Play

Ran Xu, Yuchen Zhuang, Zihan Dong, Jonathan Wang, Yue Yu, Joyce C. Ho, Linjun Zhang, Haoyu Wang, Wenqi Shi, Carl Yang

arXiv:2509.24193v1cs.CLcs.AIcs.IRcs.LG

TL;DR

Complex reasoning with search remains challenging because retrieval may require multiple hops and answers must integrate evidence. AceSearcher trains one LLM to decompose questions and solve them using retrieved context. It reports gains over strong baselines across reasoning tasks and matches DeepSeek-V3 on document reasoning.

  • Problem

    Search-augmented LLMs face complex questions requiring multi-hop retrieval and evidence integration, while existing approaches also have limits in latency, memory use, or supervision breadth.

    Source: §1 · pp. 1–2

  • Method

    AceSearcher trains one LLM as a decomposer and solver using supervised fine-tuning on retrieval, reasoning, and decomposition data, followed by reinforcement fine-tuning based only on final-answer rewards.

    Source: §1 · p. 2

  • Results

    AceSearcher outperforms strong baselines across reasoning tasks, reports a 7.6% gain on multi-hop QA and fact verification, and matches DeepSeek-V3 on document reasoning.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The reported results show smaller open-source LLMs can search and reason more effectively, reducing reliance on proprietary or extremely large models.

    Source: Limitations and Impact Statement · p. 10

    Evaluation focuses on complex QA, fact verification, and document reasoning; the framework uses a fixed retriever and adds inference overhead.

    Source: Limitations and Impact Statement · p. 10

Abstract

from arXiv · show

Search-augmented LLMs often struggle with complex reasoning tasks due to ineffective multi-hop retrieval and limited reasoning ability. We propose AceSearcher, a cooperative self-play framework that trains a single large language model (LLM) to alternate between two roles: a decomposer that breaks down complex queries and a solver that integrates retrieved contexts for answer generation. AceSearcher couples supervised fine-tuning on a diverse mixture of search, reasoning, and decomposition tasks with reinforcement fine-tuning optimized for final answer accuracy, eliminating the need for intermediate annotations. Extensive experiments on three reasoning-intensive tasks across 10 datasets show that AceSearcher outperforms state-of-the-art baselines, achieving an average exact match improvement of 7.6%. Remarkably, on document-level finance reasoning tasks, AceSearcher-32B matches the performance of the DeepSeek-V3 model using less than 5% of its parameters. Even at smaller scales (1.5B and 8B), AceSearcher often surpasses existing search-augmented LLMs with up to 9x more parameters, highlighting its exceptional efficiency and effectiveness in tackling complex reasoning tasks. Our code will be published at https://github.com/ritaranx/AceSearcher and https://huggingface.co/AceSearcher.

1 Introduction

Complex reasoning with search requires both multi-hop retrieval and the integration of evidence, yet existing approaches face limits in latency, memory, or supervision breadth. AceSearcher trains one LLM as decomposer and solver, combining supervised and reinforcement fine-tuning; evaluations report gains over strong baselines.

1 Introduction

  • Existing methods often rely on single-turn retrieval, powerful closed-source models, latency-increasing tree search, memory-intensive RL, or QA-only supervision.Complex applications need multi-hop evidence gathering and reasoning that integrates information beyond span extraction.Source: §1 · pp. 1–2
  • AceSearcher trains a single LLM as a decomposer that creates subquestions and a solver that uses their answers and context to generate responses.The roles jointly address search and reasoning in complex tasks.Source: §1 · p. 2
  • Its two-stage fine-tuning first uses supervised data spanning retrieval, reasoning, and decomposition, then reinforces both roles using rewards from final answers alone.The framework does not require intermediate supervision and is studied at model sizes from 1.5B to 32B.Source: §1 · p. 2
  • 7.6% average gain across three tasks and ten datasets, with the 1.5B variant matching models 10× larger on QA tasks.The evaluation compares AceSearcher with strong baselines, including recent reasoning models and RL-enhanced search LLMs.Source: §1 · p. 2

2 Related Works

Prior work addresses reasoning-intensive search through multi-turn prompting, reasoning-focused supervision, inference-time search, query refinement, RL, and self-play. AceSearcher is positioned as collaborative self-play tailored to reasoning-intensive retrieval-augmented generation.

Reasoning-intensive Search/Retrieval

  • Multi-turn prompting, chain-of-thought supervision, reward-guided search, query refinement, and multi-agent fine-tuning are prior approaches to reasoning-intensive search.Some multi-agent approaches require serving multiple LLMs at deployment.Source: §2 · p. 2

Self-play Finetuning for LLMs

  • Earlier self-play work spans alignment, instruction following, theorem proving, and reasoning; AceSearcher instead targets collaborative problem solving for reasoning-intensive RAG.The paper distinguishes its application-specific self-play framework from these prior settings.Source: §2 · p. 2

RL for Search-augmented LLMs

  • Recent RL-for-RAG work uses open-domain QA datasets to create verification rewards, whereas AceSearcher uses a data-centric self-play fine-tuning pipeline for broader reasoning-intensive RAG tasks.The stated contrast is generalization beyond multi-hop QA.Source: §2 · p. 3

3 Overview of AceSearcher

AceSearcher frames reasoning-intensive search as generating answers from questions and retrieved passages, then assigns one LLM two cooperative roles. Its decomposer creates subquestions, while its solver answers them stepwise and produces the final response.

3.1 Problem Formulation

  • In the problem setup, a retriever selects passages from a corpus for a question, and an LLM generates an answer conditioned on both.The setup allows short- or long-form responses and notes that reasoning-intensive questions may require multiple steps.Source: §3.1 · p. 3

3.2 AceSearcher: A Cooperative Self-Play Framework

  • One LLM acts as decomposer and solver: the decomposer produces sequential subquestion templates, and the solver generates intermediate answers from retrieved passages before predicting a final answer.A later subquestion may depend on earlier answers; final generation uses the original question, decomposition, intermediate answers, and context.Source: §3.2 · pp. 3–4
  • The training objective maximizes the LLM’s probability of generating the final answer, approximating the otherwise intractable marginalization over decompositions and intermediate answers by sampling candidate paths.At inference, the decomposer generates a subquestion sequence and the solver uses intermediate answers to derive the final answer.Source: §3.2 · pp. 3–4

4 Two-Stage Finetuning for AceSearcher

AceSearcher first builds decomposition, reasoning, and context-use abilities through supervised fine-tuning, then uses preference-based reinforcement fine-tuning to improve decomposer and solver policies from final-answer feedback. Its training setup covers both retrieval-augmented QA and context-rich reasoning, with rewards based on answer accuracy and output format.

4.1 Stage I: Supervised Finetuning (SFT)

  • Supervised fine-tuning mixes context-rich QA, question-decomposition, and chain-of-thought data to establish foundational capabilities for the two roles.The SFT stage curates 180K examples and uses next-token prediction.Source: §4.1 · p. 4, §4 · p. 4

4.2 Stage II: Preference-based Reinforcement Finetuning (RFT)

  • Final-answer feedback trains the model to decompose questions and reason interactively without requiring intermediate decomposition or chain-of-thought annotations.The RFT prompts contain questions and final answers only.Source: §4.2 · p. 4
  • In retrieval-augmented QA, the decomposer generates subquestions, relevant documents are retrieved for each, and the solver produces intermediate answers before a final prediction.The RAG environment uses multi-hop QA and fact-verification datasets.Source: §4.2 · p. 4
  • The reward evaluates complete trajectories using final-answer exact match and a binary check for correctly structured subquestions, intermediate answers, and reasoning steps.Both RAG and context-rich reasoning use this final-answer-derived signal.Source: §4.2 · p. 5
  • Preference-based reinforcement fine-tuning jointly optimizes decomposer and solver policies using preference pairs for decompositions, intermediate subquestions, and final answers.The unified dataset combines Ddecompose, Dsubq, and Dfinal.Source: §4.2 · p. 6
  • For online optimization, the current policy samples preference pairs iteratively, while the implementation limits documents to 15 and gives full rationales only for final answers.It discards preference pairs when the best and worst responses receive the same reward.Source: §4.2 · p. 6

5 Experiments

AceSearcher is evaluated across multi-hop QA, fact verification, and document-level reasoning, with additional studies testing its components, efficiency, parameters, and question decomposition quality. It outperforms baselines while showing gains from its training components and practical trade-offs in data, inference time, and retrieval settings.

5.1 Experiment Setups

  • Evaluation spans multi-hop QA, fact verification, and document-level reasoning, with baselines including single-turn RAG, multi-step retrieval, search-trained models, and specialist LLMs.QA is evaluated with EM, accuracy, and F1; fact verification with EM; document-level reasoning with official-script accuracy.Source: §5.1 · pp. 6–7

5.2 Evaluation on QA and Fact Verification

  • AceSearcher-32B achieves the highest overall score (60.7), outperforming proprietary and open-source baselines by up to 7.6%.AceSearcher-1.5B matches or exceeds 8B baselines, while AceSearcher-8B outperforms baseline models with 70B parameters.Source: §5.2 · p. 7, Table 1 · p. 7

5.3 Evaluation on Document-level Reasoning

  • AceSearcher-32B and AceSearcher-8B outperform size-comparable DocMath-Eval baselines by 6.2% and 9.0%, while AceSearcher-32B matches DeepSeek-V3 accuracy with less than 5% of its parameters.AceSearcher-14B also exceeds baselines up to 72B parameters.Source: §5.3 · p. 8, Table 2 · p. 8

5.4 Additional Studies

  • Both SFT and RFT contribute to performance, while removing either reasoning or search data hurts results across QA, fact verification, and document-level reasoning.AceSearcher’s RFT performs best among the compared alternatives; AceSearcher-8B gains most in its first two iterations, with diminishing returns in the third.Source: §5.4 · pp. 8–9, Table 3 · p. 8, Figure 2 · p. 8

5.5 Efficiency Studies

  • With 2K SFT examples, AceSearcher matches strong baselines with up to 4× more parameters and surpasses them after RFT; 5K RFT prompts yield gains of 1% on QA and fact verification and 2% on document-level reasoning.AceSearcher incurs more latency than standard RAG, but achieves substantial performance gains and can outperform 32B models at comparable inference time.Source: §5.5 · p. 9, Figure 3 · p. 9

5.6 Parameter Studies

  • Performance improves as retrieved contexts increase, plateauing at k = 10; more sampled decompositions and final answers generally improve results but increase trajectory collection time.The study varies k, m, and m′.Source: §5.6 · p. 9, Figure 4 · p. 9

5.7 Quality Analysis of Question Decomposition Module

  • SFT and RFT significantly improve human-rated subquestion quality, and decomposition raises HotpotQA recall@10 by up to 25% over standard retrieval, surpassing a strong reranker.Human evaluation uses 40 randomly sampled subquestions per task on a 1–5 scale.Source: §5.7 · pp. 9–10, Figure 5 · p. 9

6 Conclusion

AceSearcher combines a single LLM’s decomposer and solver roles with two-stage fine-tuning for RAG and document-level reasoning. It reports gains across ten benchmarks, including stronger performance at smaller model scales.

6 Conclusion

  • 7.6%: AceSearcher outperforms state-of-the-art models on multi-hop QA and fact verification.The comparison is reported across ten benchmarks.Source: §6 · p. 10
  • AceSearcher matches DeepSeek-V3 on document reasoning with under 5% of its parameters.Source: §6 · p. 10
  • At 1.5B and 8B parameters, AceSearcher delivers competitive or superior performance to larger models.The conclusion describes the approach as efficient under resource constraints.Source: §6 · p. 10

Limitations and Impact Statement

The authors identify boundaries in task coverage, retrieval training, inference latency, and optimization choice, while describing potential benefits for low-resource and domain-specific applications.

Limitations and Impact Statement

  • Evaluation covers complex QA, fact verification, and document-level reasoning; open-ended generation, dialogue, and real-time tool use remain unexplored.The authors also note reliance on a fixed retriever and leave joint retrieval–reasoning optimization for future work.Source: Limitations and Impact Statement · p. 10
  • Decomposition adds inference overhead that may limit use in latency-sensitive settings.The authors report favorable tradeoffs in Figure 3(c) and note that many strong baselines also use multi-turn retrieval.Source: Limitations and Impact Statement · p. 10
  • Smaller open-source LLMs may reduce reliance on proprietary or extremely large models in low-resource or domain-specific applications.Examples given include finance, scientific discovery, and healthcare.Source: Limitations and Impact Statement · p. 10

A Derivation Step for Optimal Policy π∗and ρ∗

B.2 Main theorem

The appendix presents convergence results for the self-play optimization and connects its losses to DPO under stated conditions. It develops population and sample arguments before establishing the DPO convergence claim.

B.2 Main theorem

  • Under regularity conditions, the loss (B.2) minimizer is close to the loss (B.1) minimizer with high probability and converges to θ∗ as t increases.The theorem remark attributes the result to equivalence of the losses and convergence of the loss (B.1) maximizer.Source: §B.2 · p. 17

B.3 Population Version

  • Under its assumptions, the population operator is contractive; the authors describe self-play as progressively approaching the true distribution.They attribute easier convergence to an intermediate reasoning step that smooths the local optimization landscape and makes the loss approximately convex.Source: §B.3 · pp. 18–19

B.4 Sample Version

  • The sample-based operator is presented as having a convergence property analogous to the population version.Source: §B.4 · p. 19

B.5 On the Equivalence with DPO

  • With high probability, optimizing loss (B.8) on dataset D is equivalent to maximizing the original reward up to a scaling factor.The section frames this as establishing closeness between losses (B.6) and (B.8).Source: §B.5 · p. 20

B.6 Convergence Property of DPO

  • The authors state that the sequence of DPO minimizers converges as the iteration count increases.Source: §B.6 · p. 23

C Information for Test Datasets

The test-dataset section specifies which questions were evaluated and gives a retrieval setup for two long-context datasets.

C Information for Test Datasets

  • Evaluations use all StrategyQA and Bamboogle questions, the first 500 development-set questions for other datasets, and DocMathEval’s testmini set.The passage says these choices follow existing studies for the first two sampling rules.Source: C Information for Test Datasets · p. 23
  • For SimpLong and CompLong, the evaluation uses text-embedding-3 to retrieve the top 10 relevant contexts before answer generation.Source: Table 4 · p. 24

D Details of Training Data

These appendices document the training data and prompt formats used across retrieval, decomposition, subquestion answering, and final-answer tasks. They also report SFT implementation settings and identify a table of results by model size.

D Details of Training Data

  • The SFT and RFT data composition is detailed in Table 5, and overlapping NQ and SQuAD training IDs are removed to avoid data leakage.The authors follow MusiQue repository instructions for this decontamination step.Source: D Details of Training Data · p. 25, Table 5 · p. 25

E.1 Prompts for Direct RAG

E.2 Prompts for Decomposition

E.3 Prompts for subquestion answering

  • Subquestion-answering prompts use Python programs for document-level PoT and step-by-step reasoning with a boxed answer for CoT.The appendix also shows prompts for complex question answering and fact verification, and includes instructions to use available passages and tables.Source: Figure 13 · p. 27, Figure 15 · p. 28, Figure 16 · p. 28

E.4 Prompts for final answer generation

  • Final-answer prompts provide passages and, for the task-solving format, the subquestions and their answers; examples cover complex QA, fact verification, and document reasoning with PoT or CoT.The document-reasoning examples distinguish PoT and CoT formats.Source: §E.4 · p. 28, Figure 17 · p. 28, Figure 18 · p. 29

E.5 Prompts for InstructRAG

F.1 Implementation Details for SFT

  • SFT uses a batch size of 64 per example and a maximum sequence length of 2,560 tokens; Table 6 reports results by model size.The learning rate is listed in Table 7.Source: §F.1 · p. 30, Table 6 · p. 30

F.2 Implementation Details for RFT

The appendix specifies RFT settings and baseline evaluation configurations, including sampling choices and reproduction procedures.

F.2 Implementation Details for RFT

  • RFT uses m = 3, m′ = 4, t = 1.0 for multiple rollouts, β = 0.1, and two DPO iterations by default.Examples with identical maximum and minimum rewards are discarded; Table 7 reports RFT results by model size.Source: §F.2 · pp. 30–31, Table 7 · p. 31

F.3 Implementation Details for Baselines

  • Baseline settings vary by model and method, including Qwen-3 thinking and non-thinking sampling, 3-shot Plan-RAG prompts, and reproduced IRCOT and RAG-Star results.For IRCOT and RAG-Star, the number of retrieved passages is tuned over 5, 10, and 20, and the best performance is reported.Source: §F.3 · p. 31

G Additional Experimental Results

Additional experiments examine preference-optimization and retriever choices, while prompting comparisons and case studies offer further results and evaluation details.

G Additional Experimental Results

  • AceSearcher is generally robust to β, with β = 0.1 yielding slightly better performance.The parameter study uses Llama-3.1-8B as the backbone.Source: G Additional Experimental Results · p. 31
  • AceSearcher consistently outperforms baselines across the evaluated retrievers, while E5 achieves the best overall performance.The study compares retrievers including Dragon and Contriever; the authors suggest stronger retrieval yields more relevant passages and better answer quality.Source: G Additional Experimental Results · pp. 31–32
  • PoT scores exceed CoT scores across four document-level reasoning settings; for AceSearcher-32b, DMCompLong averages are 43.0 vs. 33.0 and DMSimpShort averages are 89.5 vs. 73.5.The passage reports the advantage across models and describes the tasks as requiring multi-step reasoning.Source: G Additional Experimental Results · p. 32, Table 8 · p. 32

H Case Studies

  • The human-study form asks evaluators to rate decomposition quality from 1 to 5 using relevance, completeness, coherence, and usefulness criteria.The top rating describes highly relevant, complete, well-structured subquestions that refer appropriately to earlier answers.Source: H Case Studies · p. 32

Details of Case Study

  • Across examples from MusiQue, 2WikiMHQA, and DocMathEval, AceSearcher decomposes questions, answers subquestions, and performs multi-step reasoning to reach the correct final answer.The passage contrasts this with Qwen3-32B-reasoning struggling with complex questions despite its larger size.Source: Details of Case Study · pp. 32–33

Related papers

Loading 2509.24193v1…