PaperlayerView exact revision on arXiv

Source-linked AI summary

CAFE: Self-Improving Search Agents Need Co-Evolving Feedback

Boyang Liu, Senjie Jin, Peixin Wang, Zhangyue Yin, Yibo Wang, Yuhao Zhou, Xinbing Liang, Shizheng Zhu, Yuhui Wang, Jingqi Tong, Zhiheng Xi, Jiazheng Zhang, Clive Bai, Clarenceai, Blaze Chen, Tao Gui, Qi Zhang, Xuanjing Huang

arXiv:2608.24794v1cs.AI

TL;DR

Outcome-supervised search agents lack localized correction for intermediate errors, motivating CAFE’s shared-parameter agent–critic framework. CAFE combines online feedback-seeking optimization with offline rollout-derived critic refinement, and achieves the strongest average performance among evaluated RL-based search agents while transferring across six out-of-domain benchmarks and reducing hallucinations.

  • Problem

    Terminal rewards do not localize intermediate search errors, while learned feedback must adapt as the agent’s failure patterns change.

  • Method

    CAFE uses a shared model that alternates agent and critic roles, training feedback seeking online and feedback generation offline from rollout-derived preference pairs.

  • Results

    CAFE achieves the strongest average performance among evaluated RL-based search agents, transfers across all six out-of-domain benchmarks, and reduces answer-level hallucinations.

  • Takeaways & Limitations

    A self-improving search agent needs feedback that evolves with the policy it guides rather than a fixed supervisor tied to earlier failures.

Abstract

from arXiv · show

Outcome-supervised search agents learn when and how to retrieve evidence, but terminal rewards neither localize intermediate errors nor redirect an ongoing trajectory before those errors compound. Treating corrective feedback as a learned in-trajectory intervention couples the two roles: the agent must decide when to request and use feedback, while the critic must infer useful corrections from outcome-confounded rollouts whose failure patterns shift as the agent improves. We introduce CAFE (Coupled Agent--Feedback Evolution), a framework in which a shared-parameter model alternates between search-agent and critic roles. CAFE initializes feedback-conditioned recovery from trajectories built around the base agent's own failures, then couples online and offline optimization. During online RL, a comparative feedback estimate uses a prompt-level call--skip success gap to shape request returns, while feedback-aware advantage shaping reweights token advantages before and after feedback. Offline, rollout-derived preference optimization learns feedback from matched successful and unsuccessful trajectories. On seven agentic search benchmarks, CAFE outperforms the evaluated RL-based search agents on average, retains its gains across all six out-of-domain benchmarks, and reduces answer-level hallucinations. One-sided ablations show that improving only the agent or only the critic eventually plateaus, whereas alternating the two updates continues to improve performance. These findings suggest that a self-improving search agent needs feedback that co-evolves with the policy it guides.

1 INTRODUCTION

CAFE frames self-improving search as a coupled agent–feedback problem: terminal rewards do not localize errors or redirect active trajectories, so feedback must evolve with the policy it guides. It alternates learned feedback seeking and critic refinement, achieving strong average performance and robust transfer.

  • Motivation: Long-horizon search errors can compound before terminal reward reveals where the trajectory went wrong.Retrospective credit signals evaluate outcomes but do not directly diagnose or correct the active trajectory.
  • Results: CAFE achieves the best average performance among evaluated RL-based baselines, transfers across all six out-of-domain benchmarks, and reduces hallucinations.The reported gains include sustained improvements in trajectory quality and search performance.
  • CAFE: CAFE treats self-improving search as a coupled agent–feedback learning problem with one model serving both roles.The framework alternates online agent updates with offline critic refinement.
  • CAFE: CAFE uses comparative feedback estimation and advantage shaping to learn when to request interventions and how to assign credit around them.The comparative estimate uses the call–skip success gap, while advantage shaping weights tokens before and after feedback.
  • CAFE: Rollout-derived preference optimization learns feedback from prefix-matched successful and unsuccessful trajectories, keeping the critic aligned as the agent evolves.The preference pairs are mined from recent rollouts.

2 METHODOLOGY

CAFE implements in-trajectory recovery with a shared role-conditioned model and trains feedback seeking and generation from the agent’s own evolving rollouts. Online credit shaping is alternated with offline preference optimization so both capabilities remain aligned.

  • Self-Feedback Search Agent: CAFE augments search with an optional feedback request that lets a critic diagnose the current trajectory and propose a corrective next step.The same shared model switches between agent and critic roles during the rollout.
  • Self-Feedback Search Agent: Recovery demonstrations preserve the base agent’s erroneous prefix, insert a feedback request, and retain a successful continuation.This initializes feedback use at states the policy actually visits.
  • Online Agent Optimization: CFE compares feedback-requesting and feedback-skipping rollouts to estimate whether requesting help improves prompt-level success.The estimate is combined with the task reward and a penalty for requests beyond the first.
  • Online Agent Optimization: Feedback-aware advantage shaping separates token credit before and after the first request so repaired continuations are not credited identically to the preceding error.The method applies the adjustment under positive-advantage and positive-gap conditions, with clipping controlled by b.
  • Co-Evolution: CAFE alternates online agent optimization with RDPO critic refinement using preference pairs mined from the latest on-policy rollouts.Shared parameters let each update alter the experience used by the other role in the next iteration.

3 EXPERIMENTS

CAFE is evaluated on seven agentic SearchQA benchmarks, using EM and token-level F1 across in-domain and out-of-domain datasets. Its main results include the strongest average EM among evaluated methods, gains over RL baselines, and continued improvements beyond standard SearchQA.

  • Evaluation setup: The evaluation covers seven agentic SearchQA benchmarks, using Exact Match and token-level F1, with 2WikiMultihopQA in-domain and six out-of-domain datasets.The out-of-domain benchmarks are HotpotQA, MuSiQue, PopQA, Bamboogle, Natural Questions, and TriviaQA.
  • Main results: CAFE achieves the highest average EM of 52.5 and second-highest average F1 of 60.7 among evaluated methods with a 7B backbone.It outperforms the strongest RL-based baseline, IGPO, by 2.1 EM and 1.3 F1.
  • Main results: CAFE improves both metrics over GRPO on every benchmark, including all six out-of-domain datasets.Relative to Search-R1, average gains are 7.4 EM and 5.9 F1 across four multi-hop benchmarks, versus 1.3 EM and 1.3 F1 across three single-hop benchmarks.
  • Scale and generalization: On BrowseComp-Plus, performance improves at each training stage, with CAFE achieving the best result.This extends the reported trend beyond standard SearchQA benchmarks.

4 ABLATION AND ANALYSIS

The ablations examine online credit assignment, offline feedback learning, agent–critic alignment, and feedback evolution. Results support alternating optimization and show that feedback becomes increasingly focused on higher-level search and reasoning errors.

  • Online optimization: Advantage shaping raises average EM/F1 from 49.7/58.0 to 51.2/59.4, exceeding the more modest improvement from CFE alone to 50.8/58.4.CFE changes the return associated with requesting feedback, while advantage shaping differentially weights tokens before and after feedback.
  • Offline optimization: RDPO consistently outperforms rollout-derived SFT after five iterations by comparing matched successful and failed rollouts rather than retaining only successful feedback.The matched comparison provides a cleaner signal for feedback quality when success may result from the search prefix or later actions.
  • Agent–critic alignment: From iterations 3 through 5, each agent performs best with the critic from the same iteration, indicating alignment between critic checkpoints and evolving agent failures.With the final agent fixed, replacing the SFT critic with the iteration-5 critic raises EM from 80.6 to 84.0 and F1 from 86.6 to 89.2.
  • Feedback evolution: Feedback shifts from retrieval and grounding errors early to evidence verification in the middle and residual search and reasoning inefficiencies late in training.The progression includes misread results and conflated entities early, valid answers and explicitly stated in the middle, and repeated query, redundant tool calls, and logic fails late.
  • Answer grounding: CAFE lowers average answer-level hallucination from 29.9% for the base model and 17.6% after outcome-reward GRPO to 12.6%.Hallucination means an answer contains at least one factual claim unsupported by evidence retrieved along its trajectory.

5 RELATED WORK

Related work spans search-agent credit assignment, prompted and trained corrective feedback, and self-evolving agents. CAFE studies how feedback used within trajectories and updated from rollout outcomes can co-evolve across timescales.

  • Search agents and credit assignment: Search agents evolved from fixed retrieval pipelines to outcome-supervised policies, but long trajectories retain sparse credit because terminal correctness does not identify useful intermediate decisions.Recent methods provide finer signals through information gain and confidence changes.
  • Self-reflection and corrective feedback: Corrective feedback has been explored through prompted inference-time loops and trained critique or self-correction models, including evidence judgments and replanning in search settings.These approaches address feedback generation or use in ways that motivate CAFE’s coupled treatment.
  • Self-evolving agents: Self-evolving agents update task policies, supervisory signals, or system components, with prior work optimizing different parts of the improvement loop.Examples include alternating supervised fine-tuning with reinforcement learning and jointly improving task policies and supervisory signals.
  • Coupled evolution: CAFE couples two timescales: feedback is requested and used within a trajectory, while recent rollout outcomes update feedback generation across iterations.These updates change the experience available to both roles in the next round.

6 CONCLUSION

CAFE jointly adapts an agent’s ability to use feedback and a critic’s ability to generate it. Across search benchmarks, this coupling improves performance, transfers out of domain, reduces hallucinations, and supports feedback that evolves with the policy it guides.

  • CAFE jointly adapts an agent’s ability to use feedback and its ability to generate feedback.
  • Online RL trains feedback requesting and use, while rollout-derived preference optimization updates the critic from recent trajectories.
  • Across seven search QA benchmarks, CAFE achieves the strongest average performance among evaluated RL-based agents, transfers to six out-of-domain datasets, and reduces answer-level hallucinations.
  • The conclusion is that acting and critiquing form a coupled learning system, so feedback should evolve with the policy rather than remain tied to an earlier failure distribution.

A.1 DATASET DETAILS

The evaluation uses established multi-hop and single-hop search benchmarks, with dataset-specific test-set sampling protocols.

  • RL training selects prompts where feedback-assisted success contrasts with incorrect no-feedback behavior.Eight rollouts are sampled per prompt, retaining prompts with at least one correct feedback-requesting rollout and one incorrect no-feedback rollout.
  • Multi-hop evaluation covers 2WikiMultihopQA, HotpotQA, MuSiQue, and Bamboogle under specified test-set protocols.The first three use 500 examples each; Bamboogle uses all 125 provided test examples.
  • Single-hop evaluation includes Natural Questions, PopQA, and TriviaQA using their corresponding FlashRAG test sets.

A.2 TRAINING DETAILS

CAFE trains a shared agent–critic model through feedback-augmented initialization followed by alternating online agent optimization and offline feedback optimization.

  • Optimization settings: Training uses Qwen2.5-7B-Instruct as the shared backbone and an alternating schedule of 100 online RL steps followed by one RDPO update.The default schedule runs for five iterations, totaling 500 online RL steps.
  • Offline preference optimization: Offline preference pairs match called-correct and called-incorrect rollouts using token-set Jaccard similarity and an LLM quality check.Pairs are retained at similarity threshold τsim = 0.7 before invalid or semantically mismatched pairs are removed.
  • Critic behavior: The prompt template instructs the critic to provide brief, actionable, evidence-grounded feedback focused on the next plan or tool call.It also requires preserving the task objective, avoiding unsupported evidence, and returning feedback in a single XML block.
  • Training pipeline: The training loop alternates online rollout-based agent updates with rollout-derived offline preference optimization.Algorithm 1 describes shared-parameter role switching, online GRPO-style updates, matched feedback-pair construction, and RDPO updates.
  • Training pipeline: CAFE initializes training from base-agent failures by preserving erroneous prefixes, inserting feedback requests, and retaining successful repaired trajectories.The resulting feedback-augmented data is used for supervised initialization.

C.2 PRESERVATION OF THE TASK-UPDATE DIRECTION

The analysis shows that CAFE’s feedback-aware online update remains aligned with the original outcome-only task update under bounded perturbation conditions.

  • Update construction: The CAFE update combines comparative feedback estimation and feedback-aware advantage shaping with the outcome-only GRPO update.Comparative feedback estimates request utility across rollouts, while shaping changes token-level credit around feedback.
  • Conditions: The preservation result assumes bounded rewards, feedback utilities, request counts, shaping coefficients, and nonzero normalization denominators.The theorem also assumes finite score-weighted policy-gradient norms.
  • Conclusion: Under the stated perturbation condition, the online CAFE update remains positively aligned with the original outcome-only task update.The proof invokes the resulting positive inner product after bounding the induced perturbation.
  • Conditions: Advantage shaping changes eligible pre- or post-feedback token advantages by at most λb and leaves other tokens unchanged.

D.1 MODEL SIZE ABLATION

The model-size ablation indicates that CAFE’s gains are not confined to the 7B backbone and remain substantial at 3B scale.

  • 3B-scale results: 48.8/57.4 average EM/F1 is achieved by CAFE at 3B scale, outperforming both evaluated 3B baselines.The result is reported across all seven benchmarks.
  • 3B-scale results: At 3B scale, CAFE’s performance is comparable to several 7B search agents.The comparison supports evaluating the coupled-feedback approach beyond the larger backbone.
  • Table interpretation: Table 4 reports results for 3B-scale models, with best and second-best results bolded and underlined, respectively.

D.2 DETAILED HALLUCINATION RESULTS

CAFE reduces answer-level hallucinations across seven benchmarks, improving over outcome-reward GRPO on every benchmark.

  • CAFE improves over GRPO on every benchmark, with the largest reductions on NQ and MuSiQue.The largest reductions are 10.8 percentage points on NQ and 9.4 points on MuSiQue.

D.3 ITERATION SCHEDULE ABLATION

The iteration-schedule ablation favors more frequent alternation between online RL and RDPO, while BrowseComp-Plus results extend the training-stage improvement trend to longer-horizon search.

  • Iteration schedule ablation: 100 × 5 achieves the highest average EM and F1 and performs best on most datasets among the tested schedules.Each schedule uses the same budget of 500 online RL steps; 100 × 5 is therefore selected as the default.
  • Iteration schedule ablation: A schedule H × K consists of H online RL steps followed by one RDPO update, repeated for K iterations.The schedule notation describes the alternation pattern used in the ablation.
  • BrowseComp-Plus: BrowseComp-Plus evaluates deep-research agents on short-answer questions requiring persistent browsing for hard-to-find and interconnected evidence.The fixed curated corpus and shared local retriever support controlled evaluation.
  • BrowseComp-Plus: Performance improves steadily across BrowseComp-Plus training stages, with CAFE achieving the best result.This evaluation uses the 7B checkpoints from the main experiments.

E TRAINING DYNAMICS

CAFE’s training dynamics combine improving accuracy with sustained feedback-routing diversity, and its recovery example shows feedback redirecting a misread search toward concrete trail attributes.

  • Training dynamics: Adding feedback-aware advantage shaping yields the highest final test accuracy and a more sustained improvement than outcome-only GRPO or CFE.Outcome-only GRPO improves rapidly but fluctuates after roughly 200 steps, while CFE raises late-stage accuracy.
  • Training dynamics: 0.496 bits: CAFE retains higher final feedback-routing entropy than GRPO at 0.221 and GRPO+CFE at 0.099.The entropy measures how selectively prompts request feedback across rollouts.
  • Recovery example: The BrowseComp-Plus recovery query asks for a trail matching distance, dimensions, elevation gain, and a nineteenth-century structure constraints.The ground-truth trail is Old Kiln Trail.
  • Recovery example: The initial search identifies Grand Junction as a plausible Colorado location but does not combine it with the trail’s remaining attributes.The agent’s plan is to identify the location from aerial distances and then search for the matching trail.
  • Recovery example: After the agent requests feedback, the critic recommends re-examining the existing Colorado airport results and combining them with the trail constraints.The feedback supplies a concrete next step rather than only diagnosing the misread.

3. Search combining geography and trail attributes

The case study shows how combining a geographic clue with all trail attributes corrected a location error and recovered Old Kiln Trail. The BASE agent instead repeated unsupported Denver-focused searches and returned an incorrect answer.

  • Recovery: Old Kiln Trail matches the requested trail evidence, including a one-mile round-trip distance, an old lime kiln, and a location south of Grand Junction.The trail description also reports a 0.80-mile length, 1–3-foot width, 153-foot elevation gain, and a historic limestone kiln dating from the 1800s.
  • Recovery: The revised query combined Colorado’s Rocky Mountains with the trail’s length, width, elevation gain, and 1800s structure.This query retrieved Old Kiln Trail with matching attributes: 0.80 miles, 1–3 feet wide, 153 feet of elevation gain, and a limestone kiln dating to the 1800s.
  • Role of feedback: Feedback converted recognition of the search error into a specific Colorado-focused query, leading to the correct answer (EM = 1).The recovery relied on the Colorado location constraint and four trail attributes, without independently verifying both airport distances.
  • Failure analysis: BASE fixed on Denver after an unverified location assumption and repeated searches without rechecking the airport distances.It changed source types and query wording but retained the same location constraint.
  • Failure analysis: BASE made 25 calls and issued 26 query strings, including eight verbatim repeats, while never combining all four trail attributes in one query.Four malformed or empty tool-call turns added no evidence, and the final answer lacked support (EM = 0).
Loading 2608.24794v1…