PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboHarness: Memory-Driven Orchestration of Heterogeneous Robot Policies for Long-Horizon Planning

Jinbang Huang, Zhiyuan Li, Yuanzhao Hu, Ran Qi, Yixin Xiao, Mark Coates, Tongtong Cao, Zhanguang Zhang, Yingxue Zhang

arXiv:2607.18060v3cs.RO

TL;DR

Long-horizon tasks require capabilities that no single robot policy reliably provides, while heterogeneous policy coordination faces uncertain applicability and incompatible handoffs. RoboHarness wraps policies as reusable skills and combines capability-aware routing with Memory Bridge. Its evaluations report gains across long-horizon, memory-dependent, and out-of-distribution tasks.

  • Problem

    No single policy reliably meets long-horizon task demands, while planning methods often assume fixed skill applicability and do not resolve uncertain capability boundaries or policy handoffs.

    Source: §1 · p. 1

  • Method

    RoboHarness wraps heterogeneous policies as reusable skills and uses execution evidence for capability-aware task decomposition and routing, with Memory Bridge for policy handoffs.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Across benchmarks and trials, RoboHarness improves long-horizon, memory-dependent, and out-of-distribution performance; on LIBERO-LoHo, success rate averages 96.0%.

    Source: §1 · p. 2, §6 · p. 8, §9 · p. 9

  • Takeaways & Limitations

    Evaluation across public benchmarks, customized tasks, and real-robot trials supports heterogeneous-policy orchestration for long-horizon manipulation.

    Source: §9 · p. 9

    RoboHarness remains bounded by the capabilities in its policy library, and its reasoning and planning stages add execution time.

    Source: §8 · p. 9

Abstract

from arXiv · show

Long-horizon robotic tasks require a breadth of capabilities beyond what any single existing robot control policy can reliably provide. Combining heterogeneous policies with complementary strengths offers a promising solution, but introduces two key challenges: uncertain capability boundaries and distribution mismatches during policy handoffs. These challenges remain largely unaddressed by existing planning methods, which typically assume homogeneous, predefined skills with fixed applicability. We propose RoboHarness, a unified framework that encapsulates independently developed heterogeneous policies, including vision-language-action models (VLAs), world-action models (WAMs), reinforcement learning (RL) policies, and task and motion planners (TAMP), as reusable agentic skills. RoboHarness integrates understanding, memory, and evolution skills to reason about policy capabilities and support capability-aware task decomposition and policy routing. To mitigate distribution mismatches during policy handoffs, we introduce Memory Bridge, a plug-in policy-chaining mechanism that enables reliable transitions between heterogeneous policies without joint retraining. Extensive experiments across five public benchmarks, 500 customized tasks across 10 classes, and 135 real-robot trials demonstrate substantial gains in long-horizon and memory-dependent tasks, as well as robustness to out-of-distribution conditions.

1 INTRODUCTION

Long-horizon tasks call for capabilities no single robot policy reliably provides, motivating coordination among heterogeneous policies. RoboHarness addresses uncertain capability boundaries and incompatible handoffs through capability-aware orchestration and Memory Bridge.

1 INTRODUCTION

  • No single existing policy reliably meets the instruction-alignment, generalization, logical-consistency, and noise-robustness demands of long-horizon tasks.VLAs, WAMs, RL policies, and TAMP offer complementary strengths but also have distinct limitations.Source: §1 · p. 1
  • Policy applicability varies with context and execution history, making capability-aware decomposition and policy assignment necessary.Independently developed policies can have overlapping applicability and differ in architecture, interfaces, and assumptions.Source: §1 · p. 1
  • A selected policy may leave the robot in a state outside the next policy’s feasible region or input distribution, causing failures and cascading errors.Orchestration therefore must address both capability-aware planning and reliable handoffs.Source: §1 · p. 2
  • RoboHarness exposes independently trained or hand-designed policies as reusable agentic skills through a common interface that accommodates policy-specific inputs and outputs.It uses contextual and historical execution evidence to guide policy assignment and the incoming policy’s retrieved experience to construct compatible handoffs.Source: §1 · p. 2

2 RELATED WORK

Prior planning and robotic harness approaches leave unresolved the joint challenge of context-dependent policy selection and handoffs among independently developed policy families. RoboHarness combines capability-aware planning with memory-guided bridging without joint retraining.

Harness Systems for Agentic and Robotic Execution

  • Existing robotic harnesses orchestrate learned primitives or combine a VLA with analytic primitives, but do not explicitly address handoffs among independently developed policy families.RoboHarness combines capability-aware task decomposition and routing with explicit policy bridging.Source: §2 · p. 2

Long-Horizon Planning and Policy Chaining

  • Long-horizon planners commonly assume specified skill applicability and compatibility between successive skills, leaving context-dependent routing and uncertain capability boundaries unresolved.Skill-chaining approaches address transitions through mechanisms such as learned initiation sets or terminal-state regularization, typically requiring dedicated training or policy regularization.Source: §2 · p. 2

3 PROBLEM DEFINITION

The problem is to complete a high-level instruction by decomposing it into subtasks, assigning heterogeneous policies according to contextual applicability, and bridging between policies when needed.

3 PROBLEM DEFINITION

  • The setting allows online adaptation and experience accumulation during evaluation, but assumes neither task-specific orchestration pretraining nor joint retraining of constituent policies.The policy library may contain policies with context-dependent applicability regions that are not explicitly known.Source: §3 · pp. 2–3
  • The framework integrates heterogeneous policies with understanding, memory, and evolution skill libraries for planning, routing, handoffs, and adaptation.Understanding skills process multimodal observations; memory skills retrieve execution evidence and construct Memory Bridge; evolution skills update policies and the harness using online evidence.Source: Figure 1 · p. 3
  • RoboHarness seeks subtasks, policy assignments, and bridge trajectories whose joint execution completes the instruction with achievable subtasks and compatible execution conditions.A bridge starts from the preceding policy’s post-execution state and aims to establish an observation–subtask pair applicable to the next policy.Source: §3 · p. 3

4 METHODOLOGY

RoboHarness combines heterogeneous policies with understanding, memory, and evolution skills to assess applicability, route subtasks, construct handoffs, and adapt from execution evidence.

4.1 UNDERSTANDING SKILLS

  • Understanding skills assess policy applicability from task instructions, visual observations, object poses, robot states, and intermediate execution results.Their outputs provide complementary evidence rather than calibrated estimates of policy success.Source: §4.1 · pp. 3–4

4.2 MEMORY SKILLS AND MEMORY BRIDGE

  • Policy-specific episodic memories support history-dependent decisions and provide successful execution experience, while the execution log records interactions and feedback from successes and failures.These records support capability assessment, failure diagnosis, and evolution.Source: §4.2 · p. 4

4.2.1 MEMORY MANAGEMENT

  • Each policy’s episodic memory stores successful trajectories as linked nodes containing subtask instructions, observations, robot states, gripper states, and text and visual embeddings.Retrieval proceeds by text similarity and then visual similarity, and successful executions update the corresponding memory bank.Source: §4.2.1 · p. 4

4.2.2 MEMORY BRIDGE

  • Memory Bridge retrieves the incoming policy’s experience and selects reachable, memory-supported handoff configurations without joint policy retraining.The coding agent reassesses task and scene context after the bridge before invoking the next policy or revising routing.Source: §4.2.2 · pp. 4–5, Figure 2 · p. 5
  • For each retrieved anchor, Memory Bridge fits a local progress estimator using neighboring trajectory nodes and constrains candidate handoffs to a locally supported state region.The constraint limits unsupported score extrapolation; the selected target balances predicted progress against motion cost and must be reachable through a feasible plan.Source: §4.2.2 · pp. 4–5

4.3 HETEROGENEOUS POLICIES AS AGENTIC SKILLS

  • Policies retain their native implementations, inputs, and outputs; policy cards record capabilities, interfaces, constraints, assumptions, training tasks, and execution statistics.The implementation includes VLAs, WAMs, RL policies, and a TAMP planner.Source: §4.3 · p. 5

4.4 EVOLUTION SKILLS

  • Evolution skills use execution evidence to adapt policies, refine orchestration, and update capability descriptions for subsequent decomposition and assignment.Validated changes persist across evaluation episodes, allowing accumulated experience to guide later execution without task-specific orchestration pretraining.Source: §4.4 · pp. 5–6

5 EXPERIMENTAL SETUP AND BASELINES

The evaluation combines five public benchmarks, a customized benchmark for heterogeneous coordination, and real-robot trials. RoboHarness is tested with independently developed policies and compared against end-to-end and hierarchical baselines.

Evaluation scope

  • RoboHarness is evaluated on five public benchmarks and in 135 real-robot trials.The five benchmarks probe task following, robustness and routing, extended horizons, memory-dependent manipulation, and mobile manipulation.Source: §5 · p. 6

Policy implementations

  • The system incorporates four independently developed policies: π0.5, Faster-WAM, OpenVLA-OFT, and a TAMP planner.Policy inclusion depends on compatible checkpoints and benchmark requirements; learned-policy memory banks start with successful dataset trajectories.Source: §5 · p. 6

Evaluation scope

  • LIBERO-Complex adds 500 randomized instances across 10 long-horizon task classes requiring heterogeneous policy coordination.The authors note that existing benchmarks do not explicitly require policy switching or evaluate handoffs.Source: §5 · p. 6

Evaluation procedure

  • Execution logs are continuously updated, successful executions update episodic memory, and experience is retained only across trials of the same task.The evaluation also injects 5–10% noise into simulator-provided object poses to approximate real-world estimation errors.Source: §5 · pp. 6–7

Real-robot experiments

  • Real-robot evaluation covers 135 trials across five assembly task classes, with 20–30 subtasks and a 10-minute limit per trial.RoboHarness combines TAMP assembly planning with π0.5 for cabinet opening and exploration, under four execution-time disturbances.Source: §5 · p. 7

Baselines

  • Baselines include end-to-end policies and hierarchical methods that provide intermediate guidance for long-horizon execution.The listed end-to-end baselines span VLAs, world-action models, and other policies; results come from published evaluations or publicly available checkpoints.Source: §5 · p. 7

6 RESULTS

RoboHarness performs strongly across perturbation robustness, long-horizon execution, memory-dependent tasks, and mobile manipulation. Ablations also indicate contributions from understanding, evolution, and Memory Bridge.

RQ1: Capability-aware policy routing

  • VLA use generally increased where its standalone performance was stronger, but sensor noise blurred observations and impaired the understanding skill’s assessment.The VLA was used more under lighting, background, and layout changes than under robot-state perturbations.Source: §6 · p. 7

RQ2: Robustness to perturbations

  • 91.9% average success rate ranked RoboHarness first in six of seven perturbation categories and above every underlying policy overall; under sensor noise, it reached 90.3%.Degraded observations impair the understanding skill’s assessment of policy suitability.Source: §6 · pp. 7–8

RQ3: Coordination of heterogeneous policies for long-horizon tasks

  • On LIBERO-LoHo, RoboHarness achieved the highest progress score and success rate on every task, averaging 98.2% and 96.0%, respectively.Individual policies often made partial progress but rarely completed the full task; hierarchical guidance alone remained substantially below RoboHarness.Source: §6 · p. 8

RQ4: Memory-dependent task execution

  • RoboHarness achieved success rates of 77.0% on M(1) and 77.5% on M(n), with 77.2% overall across nine RMBench tasks.It outperformed Mem-0 at both memory levels, with a larger advantage on M(n); its performance was comparable across the two levels.Source: §6 · p. 8, Table 2 · p. 8

RQ5: Long-horizon mobile manipulation

  • On six BEHAVIOR-1K tasks, RoboHarness achieved 0.433 success rate and 0.658 progress score, exceeding RLC in success and G0.5 in progress.The tasks required coordination of navigation, grasping, placement, and precise interactions across spatially distributed locations.Source: §6 · p. 8, Table 2 · p. 8

RQ6: Ablation Study

  • Across 500 instances and 10 task classes, removing understanding caused the largest performance drop; evolution removal reduced performance, while removing Memory Bridge left partial progress but fewer completions.Figure 4 compares average success rates and progress scores for the full system and variants with individual components removed.Source: §6 · pp. 8–9, Figure 4 · p. 8

7 REAL ROBOT EXPERIMENTS

Across 135 real-robot trials, RoboHarness handled several disturbances, but re-hidden blocks caused the largest degradation and reasoning and planning added execution time.

7 REAL ROBOT EXPERIMENTS

  • Reasoning and planning calls added 67.4 seconds per trial on average across 135 real-robot trials, with timeouts the primary failure mode.Each API call averaged 4.6 seconds; tasks comprised 20–30 subtasks.Source: §7 · p. 9
  • Re-hiding blocks caused the largest performance degradation because it required repeated exploration and replanning.RoboHarness also tolerated partial dismantling, pose-estimation errors, and distractors.Source: §7 · p. 9
  • Performance was consistent across five structures, with lower success on the Taller Bridge due to its complexity and physical instability.Trials included four disturbance settings and required VLA-based block identification and retrieval before TAMP assembly.Source: §7 · p. 9

8 LIMITATIONS AND FUTURE WORK

RoboHarness coordinates policies but remains limited by the capabilities in its library, and its reasoning and planning add execution time.

8 LIMITATIONS AND FUTURE WORK

  • RoboHarness cannot perform tasks that require capabilities absent from its entire policy library.Future work proposes broadening the library.Source: §8 · p. 9
  • Reasoning and planning stages add to overall execution time.The authors propose using execution trajectories collected by RoboHarness for further robot-policy training.Source: §8 · p. 9
  • Additional modalities, such as tactile feedback, may improve contextual assessment.Source: §8 · p. 9

9 CONCLUSION

RoboHarness combines capability-aware policy assignment with memory-guided handoffs, and experiments and ablations support the complete system and its components.

9 CONCLUSION

  • RoboHarness couples capability-aware policy assignment with memory-guided handoffs for long-horizon manipulation.Contextual assessments, policy metadata, and execution logs support decomposition and routing; Memory Bridge uses retrieved experience to select reachable, memory-supported handoff configurations.Source: §9 · p. 9
  • Results across five public benchmarks, 500 randomized instances across 10 task templates, and 135 real-robot trials demonstrate the effectiveness of the complete system.The benchmarks include selected BEHAVIOR-1K tasks.Source: §9 · p. 9
  • Component ablations support the complementary value of contextual assessment, online refinement, and bridging for heterogeneous policy orchestration.The framework requires no task-specific orchestration pretraining or joint policy retraining, while permitting online adaptation and experience accumulation during evaluation.Source: §9 · p. 9

A LIBERO-COMPLEX CUSTOMIZED TASKS

A.1 SIMULATED TASKS

The customized tasks test long-horizon coordination across heterogeneous policies, including contexts where capabilities are exclusive or overlapping and instructions require state-aware reasoning.

A.1 SIMULATED TASKS

  • Approximately 20% of subtasks can be completed by only one of the four policy types, while the rest have overlapping coverage but differing success rates.The 500 simulated instances derive from 10 templates evaluated under 50 randomly initialized tabletop configurations.Source: §A.1 · p. 15
  • The templates test whether RoboHarness selects a policy with the required capability and an effective policy when multiple options are available.Source: §A.1 · p. 15
  • The ten templates include multi-step tasks such as retrieving objects, manipulating appliances, and rearranging bowls and other items.Source: §A.1 · p. 15
  • Tasks require grounding relational or state-dependent references, coordinating policies, and respecting ordering dependencies as intermediate actions change the scene.Examples include opening containers before inserting objects and activating appliances at the appropriate stage.Source: §A.1 · p. 15

B REAL ROBOT EXPERIMENT

The real-robot experiment evaluates structure construction with hidden-block retrieval and execution-time disturbances. Its implementation combines TAMP planning with VLA cabinet interactions and uses multiple assessments to inform policy selection.

B REAL ROBOT EXPERIMENT

  • 135 UR5e trials cover five structure-construction tasks and four execution-time disturbance settings.Required blocks may be hidden in a cabinet, so the robot must identify and retrieve missing components before assembly.Source: B REAL ROBOT EXPERIMENT · pp. 15–16
  • TAMP plans geometrically precise assembly actions, while π0.5 handles cabinet interactions and exploration through Memory Bridge handoffs.After exploration, control returns to TAMP to update object state and resume assembly.Source: B REAL ROBOT EXPERIMENT · p. 16
  • Four execution-time disturbances test replanning and recovery: re-hiding blocks, dismantling structures, pose-estimation noise, and workspace distractors.The pose estimates receive 5%-10% relative percentage noise.Source: B REAL ROBOT EXPERIMENT · pp. 16–17

C.1 UNDERSTANDING SKILLS •

  • Understanding skills assess temporal pose stability, visual similarity, and semantic relevance to inform uncertainty-aware policy selection.Visual and semantic assessments compare current observations and task instructions with policy-specific memory.Source: §C.1 · pp. 17–18
  • Input-quality and state-policy compatibility assessments evaluate observation reliability and whether the current robot configuration suits a candidate policy.Image quality combines exposure, sharpness, contrast, and entropy; compatibility draws on retrieved policy memory and a local progress estimator.Source: §C.1 · p. 18

C.2 MEMORY SKILL IMPLEMENTATION DETAILS

This material details memory retrieval and Memory Bridge construction, self-evolution, and the heterogeneous policies and benchmarks used in evaluation. On LIBERO-LoHo, RoboHarness leads compared methods in progress score and success rate across five tasks.

C.2 MEMORY SKILL IMPLEMENTATION DETAILS

  • Successful policy trajectories are stored as temporally linked memory nodes, while the execution log records interactions and feedback from successful and failed runs.Nodes include observations, subtask instructions, robot and gripper states, and visual and semantic embeddings.Source: §C.2 · pp. 18–19
  • A pairwise ranking SVM estimates local progress from retrieved trajectory segments, and Memory Bridge filters handoff targets by memory support, progress, and motion feasibility.Candidate acceptance uses a score threshold that can be lowered during sampling; bridge construction does not jointly retrain constituent policies.Source: §C.2 · pp. 19–20

C.3 EVOLUTION SKILL IMPLEMENTATION DETAILS

  • Self-evolution combines policy adaptation, harness refinement, and metadata updates using execution evidence, with validated changes retained across evaluation episodes.Parameter tuning uses directional choices over discretized values rather than exhaustive grid search.Source: §C.3 · pp. 20–21

C.3.1 UNDERLYING HETEROGENEOUS POLICIES

  • The heterogeneous library includes π0.5, Faster-WAM, RL-post-trained OpenVLA-OFT, and TAMP, whose symbolic planning remains bounded by its predefined task domain and execution capabilities.The policies contribute language-conditioned, video-model-based, reinforcement-learned, and symbolic-geometric capabilities while retaining native interfaces.Source: §C.3.1 · pp. 21–22

D.1 HETEROGENEOUS POLICY CONFIGURATIONS ACROSS BENCHMARKS

  • Table 3 marks policy availability by benchmark, not guaranteed invocation; routing determines which available policies are used during execution.LIBERO, LIBERO-Plus, RMBench, and BEHAVIOR-1K use π0.5 and TAMP; LIBERO-LoHo and the ablation include all four policy families.Source: §D.1 · p. 22, Table 3 · p. 22

D.2 RMBENCH

  • RMBench covers nine dual-arm tasks: five M(1) tasks requiring limited historical information and four M(n) tasks requiring reasoning over multiple past observations.Evaluation uses 20 binary success/failure episodes per task.Source: §D.2 · pp. 22–23, Table 4 · p. 23

D.3 BEHAVIOR-1K

  • BEHAVIOR-1K evaluation selects six household tasks, uses 10 episodes per task for RoboHarness and several compared methods, and reports mean success rate separately from progress score.G0.5 is represented by published per-task progress scores because per-task binary success rates are unavailable.Source: §D.3 · p. 23, Table 5 · p. 23

D.4 LIBERO-LOHO

  • 98.2% average progress score and 96.0% success rate on LIBERO-LoHo, outperforming all compared methods on both metrics across five tasks.Table 6 reports progress score / success rate for each entry.Source: §D.4 · p. 24, Table 6 · p. 24
Loading 2607.18060v3…