PaperlayerView exact revision on arXiv

Source-linked AI summary

ReflectPO: Reflection-Guided Policy Optimization for Long-Horizon LLM Agents

Rumei Li, Xunliang Cai

arXiv:2610.30933v1cs.CL

TL;DR

Long-horizon execution errors can alter later states and histories, making autonomous recovery difficult. ReflectPO uses branch differences and repeated visits to guide recovery trajectories, with tree-structured credit assignment; experiments report the best overall average performance at both model sizes.

  • Problem

    Long-horizon execution errors alter subsequent states and interaction histories, and existing approaches provide limited support for agents to identify and recover from their own deviations.

    Source: §1 · pp. 1–2

  • Method

    ReflectPO constructs recovery hints from branch differences and repeated visits, generates recovery trajectories, and applies tree-structured credit assignment to sampled trajectories.

    Source: §1 · p. 2

  • Results

    ReflectPO achieves the best overall average performance at both model sizes, and analyses suggest agents can autonomously identify errors and recover from deviations to some extent.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    ReflectPO improves proactive error recovery while supporting successful task execution and maintaining sampling efficiency.

    Source: §6 · p. 9

Abstract

from arXiv · show

With external tools, large language models (LLMs) can tackle complex tasks through sustained environmental interaction. LLM-based agents are therefore increasingly used to tackle complex long-horizon tasks in real-world environments. However, these tasks involve sequential, interdependent decisions and changes in environmental states. Early execution errors affect subsequent interaction histories and environmental states, and their effects accumulate along the decision chain. To address this challenge, we propose Reflection-Guided Policy Optimization (ReflectPO), a reinforcement learning framework for autonomous error recovery in long-horizon tasks. ReflectPO comprises two core components. The first is Reflection-Guided Sampling. We use branch differences and repeated visits in interaction trajectories to construct hints for error recovery. The second is Tree-Structured Credit Assignment. We organize sampled trajectories into trees and assign action-level credit using parent--child reward differences and recovery gains. Experiments over two long-horizon tool-use benchmarks show that ReflectPO outperforms multiple existing baselines in most domains. It also achieves the best overall performance on average. Further analyses show that ReflectPO effectively improves agents' ability to recognize their own deviations, recover from erroneous states, and continue making progress toward task completion.

1 INTRODUCTION

Long-horizon dependencies make autonomous error detection and recovery difficult because execution errors can alter later states and interaction histories. ReflectPO combines recovery-trajectory sampling with tree-structured credit assignment, and improves average performance on two tool-use benchmarks.

1 INTRODUCTION

  • Execution errors can alter subsequent states and histories, accumulate across decisions, and remain difficult for agents to detect and recover from autonomously.Existing recovery approaches may require external test-time assistance or externally supplied error locations and guidance.Source: §1 · pp. 1–2
  • ReflectPO constructs recovery hints from differences between execution paths and repeated visits, then generates recovery trajectories from selected anchors.Its contrast-based strategy organizes same-task trajectories in a shared prefix tree; its loop-based strategy builds a local interaction graph for each trajectory.Source: §1 · p. 2
  • Tree-structured credit assignment uses sampled-trajectory structure to capture local decision effects and gains from recovery.The resulting training signals reinforce effective task execution and error recovery.Source: §1 · p. 2
  • ReflectPO improves average performance over conventional methods on two long-horizon tasks, while analyses report clear error-recovery behaviors.Evaluations on two tool-use benchmarks find consistent average-performance improvements and suggest agents can identify errors and recover from deviations to some extent.Source: §1 · p. 2

2 RELATED WORK

Prior work documents accumulating errors in long-horizon execution and explores monitoring and recovery, while leaving autonomous deviation detection and correction insufficiently addressed. ReflectPO uses branch differences and repeated visits to train reflection and recovery from natural trajectories.

2.1 FAILURES AND RECOVERY IN LONG-HORIZON AGENT EXECUTION

  • Long action sequences compound short-term errors and complicate exploration and credit assignment; existing monitoring and diagnosis approaches do not directly train autonomous deviation detection and correction.Prior work also reports weaker detection and recovery of agents’ own errors than recognition of external errors.Source: §2.1 · p. 2

2.2 AGENTIC REINFORCEMENT LEARNING FOR LONG-HORIZON TASKS

  • Agentic RL methods improve sampling, advantage estimation, reflection, or recovery, but give limited attention to discovering latent deviations in natural rollouts and learning when to reflect.ReflectPO uses branch differences and repeated visitation patterns to identify key states and construct recovery trajectories for autonomous reflection and recovery.Source: §2.2 · p. 3

3 PRELIMINARY

The paper models tool interaction as a task-conditioned partially observable process in which actions shape later states and histories. Its preliminary analysis finds errors are widespread while autonomous recovery is limited.

3.1 PROBLEM FORMULATION

  • The formulation models each task with an initial state, environment, and context, while the policy acts over tool calls and natural-language responses based on interaction history.Actions update environment state and return observations; trajectories end with a final response or at the step limit, and earlier actions can shape later decisions.Source: §3.1 · p. 3

3.2 PRELIMINARY EXPERIMENT

  • Error trajectory rates reach 79.38 on BFCL-v3 and 96.04 on τ 2-bench, while step recovery rate is 18.74 and complete recovery rate is 14.19.The authors conclude that the base policy does not consistently detect and recover from errors; reflection-guided recovery trajectories are incorporated into policy optimization to help agents resume progress.Source: §3.2 · pp. 3–4

4 METHOD

ReflectPO identifies points in original trajectories for review, generates recovery trajectories from corresponding environment states, and trains with reflection-guided sampling and tree-structured credit assignment.

4 METHOD

  • ReflectPO combines contrast-based and loop-based recovery sampling with tree-structured credit assignment for long-horizon agent tasks.Sampling generates recovery trajectories from selected points; credit assignment derives action advantages from original and recovery outcomes.Source: §4 · p. 4

4.1 INTERACTION NODE REPRESENTATION

  • Interaction nodes represent recent action–observation pairs; normalized signatures support matching, while saved environment snapshots enable resumed execution.The node window can retain the full interaction history when its length covers the task so far.Source: §4.1 · p. 4

4.2 REFLECTION-GUIDED SAMPLING

  • At shared-prefix branch points, contrast sampling prioritizes local behavior groups with larger terminal-reward differences and pairs a lower-reward target with a different, no-worse reference.The guide model uses the shared prefix and both local segments to produce a reflection hint; recovery begins from the latest valid snapshot in the target segment.Source: §4.2 · pp. 4–5
  • Loop-based sampling detects repeated finite-window nodes, uses the loop and preceding context to suggest an alternative, then resumes from the post-loop state with a hint.The loop recovery anchor is after the repeated node’s second visit; the target trajectory’s environment snapshot is restored for the new rollout.Source: §4.2 · p. 5

4.3 TREE-STRUCTURED CREDIT ASSIGNMENT

  • For original actions, branch-node advantages compare mean terminal rewards through a child and its parent, with the latest branch advantage inherited along the path.Raw advantages are scaled and clipped using batch-level statistics.Source: §4.3 · p. 6
  • For recovery actions, paired terminal-reward differences assign a shared advantage to post-anchor actions, while policy updates remove hints from recovery contexts.The recovery trajectory is paired with its target original trajectory.Source: §4.3 · p. 6

5 EXPERIMENTS

ReflectPO is evaluated on two long-horizon tool-use benchmarks, with results covering overall performance, recovery behavior, component ablations, and training efficiency.

5.1 EXPERIMENTAL SETUP

  • The evaluation uses BFCL-v3 Multi-Turn and τ 2-Bench, comparing direct inference with Qwen3 models and policy-optimization baselines.Training uses Qwen3 models and RL data from EnvScaler.Source: §5.1 · p. 7

5.2 MAIN RESULTS

  • ReflectPO leads all reported metrics at both model sizes, with overall scores of 36.6 for Qwen3-4B and 45.4 for Qwen3-8B.Its gains over the base models are larger for Qwen3-8B overall and especially on BFCL-v3 Multi-Turn.Source: §5.2 · p. 7, Table 2 · p. 8

5.3 ERROR RECOVERY ANALYSIS

  • For Qwen3-8B, ReflectPO reduces execution errors and raises step and complete recovery rates by 2.49 and 2.70 percentage points over Reinforce++.Across five error categories, it exceeds the base model in all five and Reinforce++ in four.Source: §5.3 · pp. 8–9, Figure 4 · p. 8

5.4 ABLATION STUDY

  • Tree-based credit assignment beats standard Reinforce++ by 1.91 points overall, and contrast- and loop-based sampling add 2.04 and 1.11 points over that variant.Full ReflectPO scores 45.36 overall and 37.41 on τ 2-Bench; combining sampling strategies improves τ 2-Bench by 1.08 points but reduces BFCL-v3 Multi-Turn by 0.25 points versus contrast-only.Source: §5.4 · p. 9, Table 3 · p. 9

5.5 TRAINING EFFICIENCY ANALYSIS

  • ReflectPO achieves higher validation performance than Reinforce++ under the same cumulative rollout response-token budget.For ReflectPO, the budget includes tokens from direct rollouts and both recovery-sampling strategies.Source: §5.5 · p. 9

6 CONCLUSION

ReflectPO samples recovery trajectories from execution-branch differences and repeated visits, then applies tree-structured credit assignment. It achieves the best average performance at both model sizes, with analyses reporting improved proactive recovery and sampling efficiency.

6 CONCLUSION

  • ReflectPO constructs reflection hints from branch outcome differences and repeated visits, generates recovery trajectories from corresponding environment states, and assigns action-level credit across original and recovery trajectories.Its tree-structured credit signals support optimization of actions in both trajectory types.Source: §6 · p. 9
  • ReflectPO achieves the best overall average performance at both evaluated model sizes on long-horizon tool-use benchmarks.The conclusion describes the benchmarks as challenging.Source: §6 · p. 9
  • Further analyses report improved proactive error recovery while maintaining training sampling efficiency.The paper states that ReflectPO supports successful task execution alongside these outcomes.Source: §6 · p. 9, Figure 5 · p. 9

A.1 IMPLEMENTATION DETAILS FOR TRAINING AND EVALUATION

The appendix specifies training configuration and recovery-sampling limits, and describes evaluation handling for API failures.

A.1 IMPLEMENTATION DETAILS FOR TRAINING AND EVALUATION

  • Training uses one node with 8 H800-141G GPUs, batches of 32, three epochs, and results from step 210; the RL set contains 2457 samples.Samples with faulty MCP tool code that could not execute were removed before training.Source: §A.1 · p. 16
  • Recovery sampling retains at most 128 contrast-based and 64 loop-based branches per training step, using a two-node window for loop graphs.The reflection prompt uses at most three candidate contrast branches per task and a local segment length of Hc = 5.Source: §A.1 · p. 16
  • Evaluation allows up to three retries for invalid user-simulator API responses and excludes samples that fail because of network fluctuations.The simulator is DeepSeek-V4-Pro, accessed through API calls.Source: §A.1 · p. 16

A.2 IMPLEMENTATION DETAILS OF SECTION 5.3

The appendix documents error-category analysis, distributed batch handling, baseline methods, and prompts for agent interaction and recovery-hint generation.

A.2 IMPLEMENTATION DETAILS OF SECTION 5.3

  • Because of resource constraints, Figure 4b’s error-category recovery analysis covers only BFCL-v3 Multi-Turn and uses five top-level categories.The categories are formed by recursively grouping fine-grained clusters.Source: §A.2 · p. 16

A.3 BATCH ALIGNMENT AND DISTRIBUTED NORMALIZATION

  • Expanded batches are padded to a multiple of distributed processes, with padding rows assigned zero training weight and excluded from credit assignment and loss averaging.Normalization uses the global count of valid trajectories, and updates are skipped if that set is empty.Source: §A.3 · p. 16

A.4 BASELINES

  • The baselines span value-model and group-relative advantage estimation, global advantage normalization, state-grouped action credit, external tool feedback, and diagnostic feedback for failed trajectories.The appendix describes PPO, GRPO, REINFORCE++, GiGPO, CRITIC, and Fission-GRPO.Source: §A.4 · pp. 16–17

B.2 PROMPTS FOR AGENT INTERACTION

  • Agent-interaction prompts distinguish conversation from non-conversation tasks and instruct the assistant to use tools, request essential missing information, and end with a concise response.Conversation user simulation follows scenario instructions and discloses information progressively.Source: §B.2 · pp. 17–19

B.3 PROMPTS FOR CONTRAST-BASED SAMPLING

  • Contrast-based hints are generated by comparing target and reference branches after their shared prefix and should suggest concrete checks or revisions.The prompt permits hints that may lead to correct or incorrect reflection.Source: §B.3 · p. 19

B.4 PROMPTS FOR LOOP-BASED SAMPLING

  • Loop-based hints use the context before repeated behavior and the loop segment to suggest a concrete next step that avoids repeating the behavior.The prompt asks the guide to focus on what the agent should inspect, change, verify, or try next.Source: §B.4 · p. 20

B.1 PROMPTS FOR PRELIMINARY EXPERIMENT

The preliminary prompt defines how trajectory steps are judged as erroneous and successfully corrected, then presents contrast-based and loop-based recovery examples. The examples show step indices, actions, and observations.

B.1 PROMPTS FOR PRELIMINARY EXPERIMENT

  • The prompt asks the evaluator to identify erroneous steps and assess whether subsequent steps corrected them.The preliminary-experiment section introduces this prompt for evaluating errors and corrections.Source: §B.1 · p. 17
  • A clear error is an agent action, judgment, or statement that was incorrect given information available at that time.Reasonable but unsuccessful exploration, tool or environment failures, and information not yet knowable are excluded.Source: §B.1 · p. 17
  • A correction counts only when a later step takes remedial action and the trajectory confirms the error or its effects were eliminated.Repeating the error, unrelated actions, failed remedies, or merely claiming success count as uncorrected.Source: §B.1 · p. 17
  • The output format requests a separate item for each erroneous step, with a has-been-corrected field and an empty error-step list when none exist.The prompt also requires valid JSON.Source: §B.1 · p. 17
  • The sample output marks step A3 as corrected.The example uses a true value for the has-been-corrected field.Source: §B.1 · p. 17

C CASE STUDY OF SAMPLING IN RL

  • The case study presents two actionable recovery-guidance examples from contrast-based and loop-based sampling, retaining steps relevant to recovery decisions.Intermediate interactions that do not affect the conclusions are omitted.Source: C CASE STUDY OF SAMPLING IN RL · p. 21
  • Table 4 presents a contrast-based sampling example, with each trajectory step showing its index, action, and observation.Source: Table 4 · p. 21
  • Table 5 presents a loop-based sampling example, with each trajectory step showing its index, action, and observation.Source: Table 5 · p. 22

D CASE STUDY OF ERROR RECOVERY EXAMPLE

The example follows a tool-use interaction in which the agent sets a budget limit, books a flight, and then closes a ticket. The booking succeeds, but the displayed reasoning does not verify its cost before asserting that it meets the requested limit.

D CASE STUDY OF ERROR RECOVERY EXAMPLE

  • Table 6 gives the full interaction sequence, with the first four reasoning passages excerpted and the final two reproduced in full.Steps are grouped by tool calls, with two calls in Step 1.Source: D CASE STUDY OF ERROR RECOVERY EXAMPLE · p. 23
  • The interaction sequence illustrates a budget-limit correction using the converted USD amount before subsequent user requests.The caption identifies this as the focus of the full interaction sequence.Source: Table 6 · p. 23
  • The agent sets the budget limit to 2,857.14 USD, equivalent to 20,000 RMB, before the flight request.The interaction then requests a business-class JFK-to-LAX flight on February 28, 2026, for no more than $2,500.Source: D CASE STUDY OF ERROR RECOVERY EXAMPLE · p. 24
  • The booking response reports success, but the reasoning notes that no cost is available to confirm the fare is under $2,500.The assistant nevertheless says the booking aligns with the budget and that the cost is within the allowable range.Source: D CASE STUDY OF ERROR RECOVERY EXAMPLE · p. 24
  • The agent subsequently closes ticket 83912, and the tool response reports successful closure.Source: D CASE STUDY OF ERROR RECOVERY EXAMPLE · p. 24

Related papers

Loading 2610.30933v1…