PaperlayerView exact revision on arXiv

Source-linked AI summary

Agent-Reflex: Teaching Language Agents to Act Reflectively

Guanting Dong, Xinyu Yang, Jiajie Jin, Yuyang Hu, Xiaoshuai Song, Yifei Chen, Xiaoxi Li, Jiejun Tan, Yutao Zhu, Tat-Seng Chua, Ji-Rong Wen, Zhicheng Dou

arXiv:2610.17953v1cs.CL

TL;DR

Real-world tool use brings noisy feedback and user corrections that existing training environments do not adequately model. Agent-Reflex mines reusable reflection skills and trains them through reflection supervision and noisy-interaction reinforcement learning with on-policy distillation; across six benchmark suites, it improves tool use and noise robustness with stable training dynamics.

  • Problem

    Training environments often assume stable feedback and minimal user intervention, despite real-world tool noise and user-guided correction.

    Source: §1 · p. 2

  • Method

    Agent-Reflex mines reusable skills from matched success/failure trajectories and uses them for reflection, skill-recovery, and user-correction supervision alongside noisy-interaction reinforcement learning and reflection-guided on-policy distillation.

    Source: §5 · p. 10

  • Results

    Across six challenging agent benchmark suites, Agent-Reflex improves long-horizon tool use and robustness under tool and user noise while maintaining stable training dynamics.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    The reported robustness and stable training dynamics support scalable agent training.

    Source: §1 · p. 3, §5 · p. 10

    The offline skill library comes from a fixed environment snapshot and may require periodic re-mining or maintenance as tools, services, or environment dynamics change.

    Source: F Limitations · p. 26

Abstract

from arXiv · show

Large language models have evolved from single-turn chatbots into interactive agents capable of long-horizon reasoning. In real deployments, MCP tool services can return noisy feedback and users may intervene to correct the agent, but existing methods provide limited supervision for reflection learning and failure recovery. To bridge this gap, we propose Agent-Reflex, a general training framework for incentivizing proactive reflection in agents. Specifically, Agent-Reflex comprises three stages. First, Environment Reflection Skill Discovery (RSD) injects grounded tool-feedback perturbations into environment scaling, samples a capability-diverse LLM pool, mines matched success-failure pairs, and summarizes reusable reflection meta-skills. Building on this skill library, we then apply Reflective Supervised Fine-Tuning (ReSFT) that converts each skill into supervision for self-reflection, skill recovery, and user-centric correction, strengthening complementary reflective behaviors. Finally, we propose On-Policy Reflective Policy Optimization (ORPO) to combine agentic reinforcement learning under noisy interactions with reflection-guided on-policy distillation to further elicit proactive reflection. Built on Qwen3-8B, Agent-Reflex reaches 58.6 overall on five long-horizon tool-use benchmarks and 29.1 on AgentNoiseBench, surpassing strong environment-scaling and reflection baselines. Experiments on six challenging agent benchmark suites demonstrate the effectiveness of Agent-Reflex. Further analyses show that Agent-Reflex improves agents' robustness to noise while maintaining stable training dynamics, supporting scalable agent training. Our code is available at https://github.com/RUC-NLPIR/Agent-Reflex.

1 Introduction

Agent training environments often assume stable feedback and little user intervention, leaving reflection learning under real-world noise and correction insufficiently addressed. Agent-Reflex mines reusable reflection skills and trains agents to reflect, recover skills, and respond to user corrections.

1 Introduction

  • Existing training often assumes stable, well-controlled environments with minimal user intervention, unlike real deployment.Real tool services can return timeouts and errors, while user simulators have limited coverage of corrective instructions and interaction styles.Source: §1 · p. 2
  • Agent-Reflex treats reflection as a learnable meta-skill mined from execution failures and progressively internalized through verifiable reflection learning.This framework targets proactive reflection under noisy external feedback.Source: §1 · p. 2
  • Across six long-horizon tool-use and noisy-interaction benchmarks, Agent-Reflex demonstrates effectiveness while improving noise robustness and maintaining stable training dynamics.The paper presents these properties as support for scalable general-agent training.Source: §1 · p. 3
  • RSD injects real-tool failures into executable environments and mines reflection meta-skills from matched success/failure trajectories sampled from a diverse LLM agent pool.The framework anchors 557 real-world environment themes and mines 1,170 skills across 557 environments.Source: §1 · pp. 2–3
  • ReSFT internalizes skills through self-reflection, skill-recovery, and user-centric supervision, while ORPO combines noisy-agent reinforcement learning with reflection-guided on-policy distillation.ORPO re-scores the same action tokens under a privileged skill context to provide confidence-weighted supervision for reflection decisions.Source: §1 · pp. 2–3

2 Related Work

Related work spans inference-time reflection, training-time trajectory repair, noisy-interaction training, and scalable environment synthesis. Agent-Reflex distinguishes itself by mining environment-indexed recovery skills from cross-model success/failure contrasts under perturbations.

2 Related Work

3 Methodology: Agent-Reflex

Agent-Reflex builds reflection skills from noisy tool failures, converts them into verifier-validated training data, then combines noisy-interaction reinforcement learning with skill-guided on-policy distillation.

3.1.1 Noise-Grounded Environment Scaling

  • RSD builds executable environments from 557 MCP specifications, injects observed tool failures, and mines reusable reflection skills from matched successful and failed trajectories.The failures are grouped into eight families, and the resulting library contains 1,170 skills across 557 environments.Source: §3.1.1 · p. 4, §3.1.2 · p. 5

3.1.2 Reflection Meta-Skill Extraction

  • A capability-diverse pool samples trajectories; an analyzer and environment verifier label them, and paired successes and failures yield environment-specific recovery rules.Pairs come from the same task, with limits on each task's contribution and at least five distinct tasks represented per environment.Source: §3.1.2 · pp. 4–5

3.2 Stage 2: Reflective Supervised Fine-Tuning (ReSFT)

  • ReSFT turns matched skills into verifier-validated supervision for local self-repair, global recovery from the original task, and user-guided correction.For self-reflection and skill recovery, the skill prompts are removed before examples are stored; user corrections remain in the stored input.Source: §3.2 · p. 5

3.3 Stage 3: On-Policy Reflective Policy Optimization (ORPO)

  • ORPO combines verifier-rewarded learning on mixed clean and noisy rollouts with token-level distillation that re-scores the same actions under environment-specific skills.The skill-conditioned score supplies a confidence-weighted reflection signal, while gradients flow through the ordinary policy branch.Source: §3.3 · pp. 6–7

4 Experiments

Agent-Reflex achieves the highest overall score among strong baselines on five long-horizon agent benchmarks and leads the compared 8B methods on AgentNoiseBench. The experiments also compare policy-optimization methods and ablate framework components.

4.1 Experimental Setups

  • Evaluation covers five long-horizon tool-use benchmarks and AgentNoiseBench settings, using standard MCP formats, task templates, and official metrics; trained methods share a Qwen3-8B backbone.The baselines span reflection, environment-scaling, and trajectory-level RL methods.Source: §4.1 · p. 8

4.2.1 Results on Long-Horizon Tool Use.

  • Agent-Reflex achieves the highest overall score among strong baselines on five long-horizon agent benchmarks.The section also reports gains over reflection methods, noisy-agent training, and environment-scaling methods on selected benchmarks.Source: §4.2.1 · p. 8

4.2.2 Robustness under Interaction Noise.

  • 29.1% Overall on AgentNoiseBench gives Agent-Reflex the highest score among compared 8B methods, versus 25.9% for NoisyAgent-8B.Table 2 reports performance under tool and user noise; its Overall is the mean of the two benchmark averages.Source: §4.2.2 · p. 8, Table 2 · p. 9

4.3 Does On-Policy Reflective Optimization Improve Robustness?

  • With matched Qwen3-8B starts, task pool, budgets, and 100 RL steps, ORPO holds an advantage over five representative algorithms on benchmark accuracy and real interaction noise.Figure 4(a) compares held-out final accuracy; the reported advantage includes GRPO, DAPO, REINFORCE++, SEED, and noisy Agent RL.Source: §4.3 · p. 8, Figure 4 · p. 9
  • ORPO's mean rollout reward separates early from GRPO and SEED and keeps rising after both flatten; the authors attribute this stability to confidence weighting.Confidence weighting emphasizes tokens with positive skill-induced shifts and down-weights weak or negative shifts.Source: §4.3 · p. 9, Figure 4 · p. 9

4.4 Ablation Study

  • Removing any component degrades both AgentNoiseBench settings, and removing self-reflection supervision causes the largest drop, followed by skill recovery and user-centric reflection.The authors describe failure awareness and local repair as the foundation of reflective behavior.Source: §4.4 · p. 9
  • The skill library alone is insufficient: removing OPD hurts performance, and removing its confidence weighting also causes a decline.The authors attribute the gains to the token-level reflection signal rather than skill exposure alone.Source: §4.4 · pp. 9–10, Table 3 · p. 9

5 Conclusion

Agent-Reflex combines mined reflection meta-skills, complementary reflective supervision, and noisy-interaction policy optimization. Across six challenging agent benchmark suites, it improves long-horizon tool use and robustness under tool and user noise with stable training dynamics.

5 Conclusion

  • RSD mines reusable reflection meta-skills from matched success–failure trajectories under grounded tool-feedback perturbations.The framework presents reflection as a learnable meta-skill.Source: §5 · p. 10
  • ReSFT turns each skill into complementary supervision for self-reflection, skill recovery, and user-centric correction.Source: §5 · p. 10
  • ORPO combines agentic reinforcement learning under noisy interaction with reflection-guided on-policy distillation to elicit proactive reflection.Source: §5 · p. 10
  • Across six challenging agent benchmark suites, Agent-Reflex improves long-horizon tool use and robustness under tool and user noise with stable training dynamics.The authors state that these results support scalable agent training.Source: §5 · p. 10

AI use statement

The authors describe generative AI use as limited to literature reading and proofreading or polishing author-written text. They state that the tools were not used to develop or conduct the research and that they reviewed the assisted content.

AI use statement

  • Generative AI tools assisted only with literature reading and proofreading or polishing author-written text.Source: AI use statement · p. 10
  • The tools were not used to develop research ideas, design experiments, conduct analyses, generate results or citations, formulate questions, or create or modify figures.Source: AI use statement · p. 10
  • The authors reviewed all AI-assisted content and take responsibility for the final manuscript, claims, and artifacts.Source: AI use statement · p. 10

Reproducibility statement

The reproducibility statement points to Appendix D for implementation and evaluation details. The related-work discussion distinguishes reflection approaches by where their corrective signal resides and positions Agent-Reflex as using reusable environment-indexed skills.

Reproducibility statement

  • Appendix D documents environment construction, failure mining and noise injection, trajectory sampling and skill extraction, training procedures, evaluation protocols, and contamination controls.Source: Reproducibility statement · p. 10

A.1 Agent Reflection Learning

  • Inference-time reflection methods leave parameters unchanged, using verbal critiques, iterative self-feedback, or retrieved cross-task insights; the authors note that a failed policy may not contradict a well-formed but wrong tool return.The passage reports that Reflexion and ExpeL fall below their own backbone on the longest-horizon benchmark.Source: §A.1 · p. 17
  • Correction-trajectory methods train on episode-specific repairs, which the authors contrast with accumulating environment-level knowledge reusable in later episodes.The passage identifies LoongReflect as closest to ORPO in supervision type, but its privileged context is an episode-level hindsight rewrite.Source: §A.1 · p. 17
  • NoisyAgent perturbs user instructions and tool responses and increases noise difficulty, but its scalar-return supervision does not specify which recovery a particular failure admits.Source: §A.1 · p. 17
  • Agent-Reflex mines matched success/failure contrasts into environment-indexed skills reusable across episodes, then uses them for ReSFT and ORPO supervision.The skills are privileged during training but removed from stored inputs, so deployment needs neither a skill prompt nor an auxiliary reflector.Source: §A.1 · p. 17

A.2 Scalable Environment Synthesis for Agent Training

Environment-synthesis methods provide executable settings but leave failure diagnosis unresolved; ORPO builds on agentic RL and on-policy distillation by supplying mined environment-level skills as privileged context.

A.2 Scalable Environment Synthesis for Agent Training

  • Executable environments make transitions and outcomes checkable, but existing approaches do not identify which tool call failed or what recovery it permits.Agent-Reflex targets this residual rather than environment construction itself.Source: §A.2 · pp. 17–18

A.3 Agentic Reinforcement Learning

  • ORPO reuses SEED’s on-policy distillation estimator, supplying an environment-level skill card mined from tool failures as privileged context.This distinguishes its contribution from methods that reshape rollouts or credit assignment.Source: §A.3 · p. 18

B Datasets

The section describes the evaluation benchmarks, comparison methods, and training data and stages used to assess Agent-Reflex. Its training combines mined skills, reflective supervision, and noisy-interaction RL with reflection-guided distillation.

B.1 General Function Calling •

  • The evaluation covers function-calling, stateful conversational, dual-control, and cross-scenario workflow benchmarks; BFCL-v4 and ACEBench averages follow official scoring procedures, whereas VitaBench reports official Avg@4.ToolSandbox uses a stratified, reproducible 200-task subset; τ 2-Bench success requires the final database state to match the target.Source: §B.1 · p. 18, §B.2 · p. 18

B.3 Noisy Interaction

  • AgentNoiseBench evaluates tool- and user-side perturbations with Stability-Gated Success, crediting only trajectories that remain valid throughout the task.Its perturbations are out of distribution: their taxonomy and injection process differ from training, and its environments are not used for training.Source: §B.3 · pp. 18–19

B.4 Training Task Pools

  • The RSD environment pool supplies ReSFT data, while ORPO reuses 2.5K EnvScaler RL tasks; these are distinct task pools aligned by environment.The EnvScaler pool is used to compare RL objectives under the same task coverage.Source: §B.4 · p. 19, §D.4 · p. 22

C Baselines

  • Comparisons use Qwen3-8B as the common backbone and include reflection, environment-scaling, and agentic policy-optimization baselines.RL algorithms are compared on the same 2.5K EnvScaler tasks; baseline results are mostly reported from their papers, with some local reproductions.Source: §C.1 · p. 19, C Baselines · p. 19, §C.4 · p. 20

D.2 Stage 1: Environment Construction and Skill Discovery

  • RSD builds 557 database-backed executable environments with 60 verifier-backed tasks each and injects tool-return failures drawn from eight observed families.Failure mining yields 24,510 errors among 177,700 tool returns; perturbations are retained only when recovery remains verifiable.Source: §D.2 · p. 20, Table 4 · p. 21
  • RSD samples capability-diverse trajectories, pairs successes and failures within tasks, and summarizes 1,170 reflection meta-skills.Pairs require verifier agreement, and disagreements are discarded.Source: §D.2 · p. 21

D.3 Stage 2: Reflective Supervised Fine-Tuning

  • ReSFT uses Qwen3-8B as the continuation-rollout actor and training backbone; its verifier-filtered corpus contains 54K trajectories, combining verified successes with reflective-correction samples across SR, KR, and UR.Source: §D.3 · p. 22

D.4 Stage 3: On-Policy Reflective Policy Optimization

  • ORPO combines noisy agentic RL with token-level reflection-guided on-policy distillation, using the frozen skill library to re-score action tokens.Its skill branch is a forward pass with detached gradients rather than an additional environment rollout.Source: §D.4 · p. 22

D.5 Evaluation Protocol and Contamination Control

The evaluation follows benchmark protocols and reports three-run means, with explicit controls against training–evaluation contamination. Supplementary results further break down noise performance by diagnostic axis.

D.5 Evaluation Protocol and Contamination Control

  • Scores use each benchmark’s official metric and standard MCP format, with means over three independent runs; some baseline results are locally reproduced when unavailable.The protocol also specifies the noise buckets reported in Table 2.Source: §D.5 · p. 22
  • Training and evaluation are separated by environment-family splits, near-duplicate checks, and withholding evaluation answers from analyzer, teacher, and user-simulator prompts.Figure 5 normalizes thirteen diagnostic axes by their three-model mean and marks leads clearing paired-bootstrap intervals against both baselines.Source: §D.5 · p. 23, Figure 5 · p. 23

E Supplement Experimental Results

E.1 Does Agent-Reflex Act Reflectively Under Noise?

Agent-Reflex leads across all thirteen AgentNoiseBench diagnostic axes, with its largest reported margins on selected tool- and user-side noise types. The section also describes why this evaluation tests transfer to unseen perturbations and gives examples of environment-specific recovery skills.

E.1 Does Agent-Reflex Act Reflectively Under Noise?

  • Agent-Reflex leads on all thirteen axes, with margins of 13.9 on misleading lure, 10.9 on incomplete information, and 12.9 on cognitive overload.Cognitive overload is the only user-side axis also clearing the interval against EnvScaler-8B.Source: §E.1 · p. 23
  • The two largest tool-side margins occur for well-formed misleading and incomplete returns, while hard-failure templates yield gains over the backbone of 12.3–13.5.Well-formed returns evade schema checks, so their faulty content must be identified from evidence in the payload.Source: §E.1 · p. 23
  • Agent-Reflex is derailed on 28.5% of rollouts, versus 44.8% for EnvScaler-8B and 56.1% for the backbone; the trajectory-validity gate attributes gains to repair, not tolerance.A correct answer after a trajectory has been derailed does not count as success under this metric.Source: §E.1 · pp. 23–24

E.2 Why AgentNoiseBench Is Out of Distribution for Agent-Reflex

  • AgentNoiseBench is out of distribution: its noise sources, environments, and injection process differ from training, and none of its environments, tasks, or injector is used in training.The authors characterize performance as transfer to unseen perturbations rather than recall of a seen noise distribution.Source: §E.2 · p. 24

E.3 Representative Environment-Specific Skills

  • Representative environment-specific skills emphasize checking identifiers, constraints, and state after operations before writing records, booking, or reporting data.The examples include re-reading updated records, verifying hotel sessions and rates, and reconciling returned rows against requested filters.Source: §E.3 · pp. 24–25

F Limitations

The authors identify limits in keeping reflection skills aligned with changing environments and in generalizing skills from online updates. They also outline extensions to the failure taxonomy and training recipe, and discuss broader impacts and ethics.

F Limitations

  • The library is mined offline from a fixed environment snapshot, so changed interfaces or service behavior may require periodic re-mining or maintenance.The authors identify 557 executable environments anchored in live MCP services as the current recipe's scope.Source: F Limitations · p. 26, G Broader Impacts and Future Directions · p. 26
  • Online summaries may remain specific to local environments or failure patterns when available tasks lack sufficient diversity; the proposed link to stronger generalization over SEED remains unisolated.The authors say testing that explanation would require further controlled experiments.Source: F Limitations · p. 26
  • Future work will investigate online skill summarization and incremental updating across larger, more diverse environment collections to incorporate new environments and emerging failure patterns.The stated aim is to support cross-environment generalization.Source: F Limitations · p. 26

G Broader Impacts and Future Directions

  • The recipe is exercised across 557 executable environments, eight mined tool-failure families, and six evaluation suites; its three stages are designed to be replaceable independently.The stated scope spans long-horizon tool use and noisy interaction.Source: G Broader Impacts and Future Directions · p. 26
  • Proposed extensions include temporal failure signals, testing skill transfer across model capacities and environments, and distilling artifacts beyond reflection rules through ORPO.Examples of temporal signals include latency, concurrency, and partial writes; richer artifacts could include verified sub-plans or tool contracts.Source: G Broader Impacts and Future Directions · pp. 26–27
  • Recovery can make agents more consequential when wrong; the authors use reversibility fields, distinguish user queries from unilateral action, and include incorrect advice in training.Environments use public service specifications without real user data, and released error information is aggregated by counts and families rather than service identity.Source: G Broader Impacts and Future Directions · p. 27

H Ethics statement

Related papers

Loading 2610.17953v1…