PaperlayerView exact revision on arXiv

Source-linked AI summary

From Reaction to Anticipation: Proactive Failure Recovery through Agentic Task Graph for Robotic Manipulation

Sheng Xu, Ruixing Jin, Huayi Zhou, Bo Yue, Guanren Qiao, Yunxin Tai, Yueci Deng, Kui Jia, Guiliang Liu

arXiv:2605.11951v1cs.RO

TL;DR

Robotic manipulation remains difficult to execute reliably because failures are inevitable in dynamic environments, while reactive detect-reason-recover approaches can be slow. AgentChord adds anticipated recovery branches to directed task graphs and uses compiled monitors to trigger pre-planned responses. Across long-horizon manipulation tasks, it improves success rates and reduces execution time relative to reactive re-planning or backtracking baselines.

  • Problem

    Reliable robotic manipulation remains challenging because failures are inevitable in dynamic, unpredictable environments, and stepwise reactive recovery can be ineffective for real-time response.

    Source: §I · pp. 1–2

  • Method

    AgentChord augments directed task graphs with anticipated, forward-moving recovery branches and compiles them into monitored behaviors that can be triggered when failures occur.

    Source: §I · p. 2, §VI · p. 9

  • Results

    Across long-horizon single-arm and bimanual tasks in simulation and real-world settings, AgentChord improves task success rates and reduces execution time compared with reactive re-planning or backtracking baselines.

    Source: §VI · p. 9

  • Takeaways & Limitations

    Recovery-augmented task graphs provide a practical and extensible representation for integrating failure anticipation, monitoring, and recovery in robotic manipulation.

    Source: §VI · p. 9

    The system may miss rare, compound, or non-hand-induced failures, and some feature extractors can be unreliable for irregular, deformable, or transparent objects.

    Source: §V · p. 9

Abstract

from arXiv · show

Although robotic manipulation has made significant progress, reliable execution remains challenging because task failures are inevitable in dynamic and unstructured environments. To handle such failures, existing frameworks typically follow a stepwise detect-reason-recover pipeline, which often incurs high latency and limited robustness due to delayed reasoning and reactive planning. Inspired by the human capability to anticipate and proactively plan for potential failures, we introduce AgentChord, an agentic system that models a manipulation task as a directed task graph. Before execution, this graph is enriched with anticipatory recovery branches that specify context-aware corrective behaviors, enabling immediate and targeted responses when failures occur. Specifically, AgentChord operates through a choreography of specialized agents: a composer that structures the nominal task graph, an arranger that augments the graph with anticipatory recovery branches, and a conductor that compiles and coordinates executable transitions using low-latency monitors to detect deviations and trigger pre-compiled recoveries without re-planning. Empirical studies on diverse long-horizon bimanual manipulation tasks demonstrate that AgentChord substantially improves success rates and execution efficiency, advancing the reliability and autonomy of real-world robotic systems. The project page is available at: https://shengxu.net/AgentChord/.

I. INTRODUCTION

Robotic manipulation remains difficult to execute reliably amid inevitable failures in dynamic environments. AgentChord addresses reactive recovery by anticipating failures in a task graph and coordinating agents to prepare monitored recovery behaviors before execution.

I. INTRODUCTION

  • Failures in dynamic, unpredictable environments make reliable robotic manipulation and automatic failure recovery essential for real-world systems.Existing approaches face slow MLLM inference, coarse language grounding, and costly or unreliable re-execution and backtracking.Source: §I · pp. 1–2
  • AgentChord augments a directed task graph with forward-moving recovery branches, aiming to correct anticipated failures without regressive resets or re-planning.The paper reports its highest average success rate and execution efficiency on diverse long-horizon bimanual tasks in simulation and real-world settings.Source: §I · p. 2

III. METHOD

  • AgentChord’s structuring, orchestration, and compilation agents prepare recovery branches and monitored transitions for immediate execution without re-planning.Compiled monitors evaluate multimodal constraint violations and trigger pre-compiled behaviors.Source: §I · p. 2

A. Agentic Systems for Robotic Manipulation

  • Agentic robotic systems coordinate perception, reasoning, planning, and execution agents, but existing systems do not fully support proactive failure anticipation and recovery.Perception agents include CLIP, Grounding DINO, and SAM3; planning agents include ProgPrompt and Code-as-Policy.Source: A. Agentic Systems for Robotic… · pp. 2–3

B. Failure Detection and Recovery in Robotic Manipulation

  • Foundation-model recovery approaches use executable corrective actions or language instructions, while compiled monitors improve detection but leave recovery to pipeline re-invocation or backtracking.The cited approaches’ detectors can remain tied to specific constraint points, limiting flexibility and generalization.Source: B. Failure Detection and Recovery in… · p. 3

A. Problem Formulation and System Interfaces

  • The system’s task-dependent features combine perception outputs and robot signals, avoiding reliance on a fixed intermediate representation for failure detection.Features can include object geometry, joint configuration, and gripper state.Source: A. Problem Formulation and System… · p. 3

B. Directed Graph for Task Execution

AgentChord represents tasks as directed graphs of semantic sub-goals and constrained motion transitions. Before execution, it adds anticipated recovery routes and constrains them to preserve progress toward the terminal goal.

B. Directed Graph for Task Execution

  • Task-graph nodes represent semantic sub-goals, while directed edges represent constrained motion transitions; branching and merging encode alternative strategies and recovery routes.A task structuring agent synthesizes the graph topology and constraint specifications from the task instruction and initial observation.Source: B. Directed Graph for Task Execution · p. 3
  • The framework compiles nominal and recovery transitions so detected failures can switch execution to a pre-compiled recovery branch without re-invoking full task planning.The execution compilation agent prepares executable, interruptible programs with monitors.Source: Figure 2 · p. 4

C. Proactive Failure Anticipation and Recovery

  • Anticipated failure modes receive online-detectable triggers and recovery intents, with branches routing from the failure context through recovery behavior to a feasible downstream merge node.A recovery intent can reuse an existing nominal sub-goal when its semantics and constraints align; otherwise, an explicit recovery node is added.Source: C. Proactive Failure Anticipation and… · pp. 4–5
  • Recovery branches must not increase the remaining graph distance to the goal, filtering out routes whose cost-to-go exceeds that of the failure state.The recovery orchestrator proposes merge targets, and the forward-progress constraint filters the candidates.Source: C. Proactive Failure Anticipation and… · p. 5

D. Graph Compilation via Hierarchical Constrained Solvers

AgentChord compiles nominal and recovery transitions through a two-level solver structure, separating sub-goal realization from transition realization. It monitors anticipated failure modes during execution and switches to pre-compiled recovery edges when triggered.

D. Graph Compilation via Hierarchical Constrained Solvers

  • Node synthesis computes a target end-effector pose for each semantic sub-goal, while edge synthesis plans transitions that satisfy path constraints.Node objectives can include collision avoidance, reachability margins, and bimanual coordination; path constraints include collision avoidance, reachability, and task-relevant geometric relations.Source: D. Graph Compilation via Hierarchical… · p. 5
  • Edge transitions use receding-horizon refinement, updating the perceptual estimate and re-solving periodically while executing only the first M commands.The solver evaluates constraints with the latest estimate because future observations are unavailable.Source: D. Graph Compilation via Hierarchical… · p. 5
  • For each active edge, executable failure functions monitor anticipated modes; a positive value indicates detection, while a violation must persist beyond a robustness margin and short window to activate.The functions distinguish normal execution from failure conditions such as object displacement, tilt, loss of grasp, or relational misalignment.Source: D. Graph Compilation via Hierarchical… · p. 5
  • When a monitor activates, AgentChord immediately executes the mapped pre-compiled recovery edge without re-invoking full task planning.The compilation agent builds executable behaviors for both nominal and recovery transitions using the same node-versus-edge decomposition.Source: D. Graph Compilation via Hierarchical… · p. 5

E. Agentic Coordination and Execution Workflow

Three agents construct, augment, and compile the task graph; during execution, monitors determine whether to continue nominally or take a recovery edge.

E. Agentic Coordination and Execution Workflow

  • The task structuring agent builds the base graph with semantic sub-goals and constraints, and the recovery orchestration agent adds recovery nodes, edges, and candidate merge targets.The resulting recovery-augmented graph is denoted Gaug.Source: E. Agentic Coordination and Execution… · pp. 5–6
  • The execution compilation agent compiles nominal and recovery transitions into interruptible edge programs with monitors; execution switches to recovery when a monitor triggers, otherwise advancing when target constraints are met.The system runs the active edge program while evaluating its monitor.Source: E. Agentic Coordination and Execution… · p. 6

IV. EMPIRICAL EVALUATION

The evaluation describes tests across simulation and real-world settings, along with analyses of failure cases and recovery-data policy training. The experimental setup specifies a dual-arm platform, six tasks, varied object instances, four baselines, and success and efficiency metrics.

IV. EMPIRICAL EVALUATION

  • The evaluation asks whether AgentChord solves diverse manipulation tasks with higher success and reduced execution time, what causes its failures, and whether its recovery data can train a robust recovery policy.The questions cover simulation and real-world environments.Source: §IV · p. 6

A. Experimental Setup

  • The hardware is a fixed-base dual-arm robot with two 6-DoF arms, parallel-jaw grippers, and two binocular cameras providing complementary RGB-D views.The platform supports synchronous and asynchronous bimanual tasks.Source: A. Experimental Setup · p. 6
  • The six tasks span single- and dual-arm pouring, table rearrangement, block handover, towel folding, and coffee-tray setup; each has 20 trials with varied object instances and randomly sampled initial poses.Fig. 3 presents the objects collected for the tasks to assess generalizability across instances.Source: A. Experimental Setup · p. 6, Figure 3 · p. 6
  • Four failure-detection and recovery baselines are compared: Inner Monologue, DoReMi, ReKep, and Code-as-Monitor; all methods use GPT-5 and shared functional tools.The baselines differ in detection timing and recovery behavior, including VQA queries, constraint detectors, and backtracking.Source: A. Experimental Setup · p. 6
  • Task success rate is the primary metric; efficiency is measured by average execution time and action-step count, including recovery actions.Execution time covers the interval from initial visual and instruction inputs through task completion, including perception, planning, action, and MLLM inference.Source: A. Experimental Setup · p. 6

B. Performance in Simulated Manipulation Tasks

Across simulated and real-world evaluations, AgentChord outperforms baselines on success and execution efficiency. Failure analysis identifies reasoning, inverse-kinematics, perception, and monitoring-compilation errors as remaining sources of failure.

B. Performance in Simulated Manipulation Tasks

  • Across simulated disturbance tasks, AgentChord consistently outperforms baselines in success rate, execution time, and episode steps.Simulation used single-arm pour water, dual-arm pour water, and rearrange table, with disturbances introduced stochastically.Source: B. Performance in Simulated… · pp. 6–7, Table I · p. 7
  • CaM’s repeated MLLM calls increase execution time despite similar episode steps, while ReKep’s backtracking struggles with long-horizon bimanual coordination and adds redundant steps.ReKep nevertheless achieves the second-best execution time overall by invoking the full pipeline only at initialization.Source: B. Performance in Simulated… · p. 7

C. Performance in Real-world Manipulation Tasks

  • Under sustained perturbations, AgentChord shows robustness by switching to a pre-compiled recovery branch and rejoining the nominal task graph.The illustrated dual-arm pour-water execution uses this recovery transition.Source: C. Performance in Real-world… · p. 7, Figure 5 · p. 9

D. Deep Analysis on the Failure Cases

  • In long-horizon tasks, reasoning may miss failure modes during graph construction or select an incorrect recovery node, as in a displaced-block handover.The authors expect stronger reasoning models to improve anticipation accuracy and recovery selection.Source: D. Deep Analysis on the Failure Cases · pp. 7–8

C. Performance in Real-world Manipulation Tasks

  • Across six real-world tasks with disturbances, AgentChord achieves the highest average success rate and shortest execution time; episodic steps are omitted as unreliable.The table reports results on six real-world tasks with disturbances.Source: C. Performance in Real-world… · p. 7, Table II · p. 8

D. Deep Analysis on the Failure Cases

  • A reasonable corrective plan can remain infeasible when recovery IK conflicts with joint limits or reachability, particularly when re-grasping fallen or tipped objects.This exposes a gap between high-level recovery intent and low-level kinematic feasibility.Source: D. Deep Analysis on the Failure Cases · p. 8
  • Inaccurate object perception can degrade grasp planning, while incorrect monitoring-tool choices can mismatch intended recovery logic and executed checks.The authors identify perception robustness and monitoring logic as future-work directions.Source: D. Deep Analysis on the Failure Cases · p. 8

E. Leveraging Generated Data for Policy Learning

AgentChord improves success rates and reduces execution time across long-horizon manipulation tasks, and its recovery trajectories also benefit policy learning. The paper identifies reasoning, monitoring-feature, and perception or compilation limitations that can constrain recovery.

E. Leveraging Generated Data for Policy Learning

  • In 50 simulation trials with the cup moved during execution, a policy fine-tuned with AgentChord-generated recovery trajectories shows improved robustness and recovery for that failure scenario.The comparison uses a fixed fine-tuning data amount, replacing half of the successful trajectories with failure-recoverable ones.Source: E. Leveraging Generated Data for… · p. 9

V. LIMITATIONS

  • Unanticipated rare, compound, or non-hand-induced failures may go uncovered by pre-compiled monitors or recovery branches, especially in long-horizon tasks.The paper suggests invoking an MLLM-based verifier at sub-goal boundaries or after an execution timeout as a possible fallback.Source: §V · p. 9
  • Monitoring reliability depends on feature-extractor quality, while perception errors and incorrect monitoring tools can also contribute to occasional failures.The conclusion separately notes that AgentChord-generated recovery trajectories can benefit downstream policy learning.Source: §V · p. 9, §VI · p. 9

VI. CONCLUSION

  • Across long-horizon single-arm and bimanual tasks in simulation and real-world settings, AgentChord improves task success rates and reduces execution time versus reactive re-planning or backtracking baselines.The conclusion also describes recovery-augmented task graphs as a practical and extensible representation for integrating anticipation, monitoring, and recovery.Source: §VI · p. 9

APPENDIX A HARDWARE SYSTEM

The appendix describes a dual-arm robot and stereo-camera setup, six manipulation tasks, and the algorithm that builds, compiles, and executes a task graph with recovery branches.

APPENDIX A HARDWARE SYSTEM

  • The hardware uses an AgileX CobotMagic dual-arm robot with two 6-DoF arms and two RGB-D binocular stereo cameras capturing calibrated images at 10 Hz.The fixed-base robot supports coordinated dual-arm actions; cameras provide top and front views.Source: Appendix A · p. 12, Figure 6 · p. 12

APPENDIX B DETAILED TASK DESCRIPTIONS

  • The evaluation defines six tasks: single- and dual-arm pouring, table rearrangement, block handover, towel folding, and coffee-tray setup.The tasks include asynchronous and synchronous dual-arm collaboration and rigid or deformable objects.Source: Appendix B · pp. 12–13

A. Algorithm

  • AgentChord constructs a nominal graph, predicts likely failures per edge, adds recovery branches and monitors, filters branches, and compiles executable programs.During execution, the robot follows the active edge program and switches to a mapped recovery edge when a monitor triggers, without re-planning.Source: A. Algorithm · p. 13

B. Robot Control

Robot behaviors are composed from atomic actions that specify target poses and execute joint trajectories. Geometric scene and robot features support constraint generation and monitoring.

B. Robot Control

  • Atomic actions specify target end-effector poses, optionally use perception to estimate targets, and execute through inverse kinematics, joint interpolation, and position control.Target poses are clipped to a predefined workspace for kinematic feasibility and safety; the joint-trajectory representation supports single-arm and bimanual behaviors.Source: B. Robot Control · pp. 13–14

C. Atomic Actions

  • The extensible atomic-action library can adapt a grasp by re-grasping and lifting an object when perception detects that it has fallen.Actions represent semantically meaningful skills and can be augmented for more complex tasks.Source: B. Robot Control · p. 13, C. Atomic Actions · p. 14

D. Implementation Details of Geometric Feature Extraction

  • The feature vector combines object geometry and robot proprioception to encode scene state and evaluate active constraint functions.Features include object position, size, orientation, relational quantities, and gripper state; the pipeline uses open-vocabulary segmentation and tracking, point-cloud geometry, and task-relevant attributes.Source: D. Implementation Details of Geometric… · p. 14

E. Implementation Details of Sub-Goal Solver

The sub-goal solver searches for feasible target poses using a hybrid global-local strategy and soft penalties. The path solver generates constrained motion through receding-horizon optimization.

E. Implementation Details of Sub-Goal Solver

  • For each semantic sub-goal, the solver finds a target end-effector pose satisfying its constraints, using a coarse global search followed by local refinement.The objective includes soft penalties for collision avoidance, reachability, pose regularization, and consistency; bimanual coordination adds arm-arrangement regularizers and self-collision penalties.Source: E. Implementation Details of Sub-Goal… · pp. 14–15

F. Implementation Details of Path Solver

  • The path solver formulates transitions between sub-goals as constrained optimal control solved in a receding horizon.It optimizes H future steps, executes M steps before observing again, and uses M = 5 in this paper; the horizon covers the remaining edge or a few seconds of motion.Source: F. Implementation Details of Path Solver · p. 15
  • Path optimization penalizes pose changes, collision proximity, and inverse-kinematics infeasibility to encourage smooth, safe, reachable motions.The horizon is continuously refined using updated robot state and perception.Source: F. Implementation Details of Path Solver · p. 15

G. Implementation Details of Proactive Failure Anticipation and Recovery

AgentChord separates edge constraints from failure monitors, then uses persistent violations to trigger recovery. Its monitors cover geometric and relational failure conditions, with tolerances balancing noise robustness and responsiveness.

APPENDIX D MORE RESULTS IN SIMULATION EXPERIMENTS

The appendix illustrates recovery in simulation and varied real-world settings, and compares forward-moving recovery branches with backtracking. Across the described trials, AgentChord completes tasks despite variation and clutter.

APPENDIX D MORE RESULTS IN SIMULATION EXPERIMENTS

  • In three simulated tasks, AgentChord anticipates failures at initial planning and follows recovery branches back to the nominal graph after detection.Source: Appendix D · p. 16, Figure 7 · p. 17

APPENDIX E MORE RESULTS IN REAL-WORLD EXPERIMENTS

  • Appendix E presents additional real-world visualizations and points readers to qualitative videos on the project page.Source: Appendix E · p. 16

A. Manipulate Different Objects with Randomness

  • Across six real-world tasks, AgentChord completes trials despite variation in object instances, poses, and feasible disturbance configurations.Figure 8 shows initial and successful final frames; its rows are trials and its columns are initial and final scenes.Source: A. Manipulate Different Objects with… · p. 16, Figure 8 · p. 18

B. Manipulate in Clutter Scenes

C. Ablation on Forward-Moving Recovery Graph

  • AgentChord achieves higher success rates and lower execution time than its backtracking variant, whose repeated nominal execution increases time.The comparison is reported in Table V; recovery-graph construction adds upfront reasoning cost, while forward-moving branches reduce redundant execution after failures.Source: C. Ablation on Forward-Moving Recovery… · p. 16, Table V · p. 16

APPENDIX F PROMPTS FOR AGENTS AND EXAMPLE OUTPUTS

  • Appendix F directs readers to the code repository for prompts and example outputs, including a dual-arm coffee-bottle-to-paper-cup pouring task.Source: Appendix F · pp. 16, 18

Related papers

Loading 2605.11951v1…