PaperlayerView exact revision on arXiv

Source-linked AI summary

Let the Mind Conceive, Let the Hand Achieve: Following Multimodal Instructions for Long-Horizon Robotic Manipulation

Youpeng Wen, Gang Wang, Tianyu Huang, Weiwei Zhang, Zhijian Huang, Wei Chen, Xieyuanli Chen, Yunhui Liu

arXiv:2610.18352v1cs.ROcs.AI

TL;DR

Long-horizon manipulation requires robots to preserve subtask order and ground actions to the right objects as scenes change, while tracking progress. Orchestra coordinates task planning, grounding, action generation, and subtask transitions through shared context; across simulated benchmarks and real-world tasks, it reports improved success and Average Success Length over the stated baselines.

  • Problem

    Long-horizon tasks remain challenging because progress must stay aligned with changing object states and goals, while operations must preserve order and target the intended objects and destinations.

    Source: §1 · pp. 1–2

  • Method

    Orchestra decomposes multimodal instructions into ordered subtasks and maintains shared context for task records, active-subtask grounding, and observations, with an action policy and independent progress evaluator guiding execution and transitions.

    Source: §1 · pp. 2–3

  • Results

    Orchestra achieved 68.22% mean RMBench success, 26.22 percentage points over Mem-0; its RMBench-Compose ASL was 7.35, 2.46× HarnessVLA, and real-world mean ASL improved 25.0% over the base π0.5 policy.

    Source: §1 · p. 3, §6 · p. 11

  • Takeaways & Limitations

    The results support shared-context coordination as a complement to local action capabilities.

    Source: §6 · p. 11

    Performance remains task-dependent, and simulation relies on reference grounding; the authors motivate evaluation on longer real-world sequences with predicted grounding.

    Source: §6 · p. 11

Abstract

from arXiv · show

A robot following a multi-step instruction needs to act on the right objects in the right order, even as the scene changes. It must also maintain awareness of task progress and determine when to advance. To address these challenges, we introduce Orchestra, an agent harness that separates task planning and execution management from action generation. Orchestra decomposes a complete multimodal instruction into ordered subtasks, tracks completed steps, and updates the locations of relevant objects and destinations during execution. It maintains a shared context containing the active subtask, spatial grounding, and recent observations. An action policy uses this context to generate actions, while a separately trained progress evaluator estimates when to advance to the next subtask. To evaluate whether robots trained on atomic instructions and short trajectories can execute longer subtask sequences, we introduce a benchmark named RMBench-Compose. Orchestra achieves a mean success rate of $68.22\%$ across nine RMBench tasks, outperforming Mem-0 by $26.22$ percentage points. On RMBench-Compose, it achieves an Average Success Length (ASL) of $7.35$ subtasks, $2.46$ times that of HarnessVLA. Across four real-world multi-step tasks, it improves mean ASL by $25.0\%$ over Orchestra using the base $\pi_{0.5}$ policy. These results show that Orchestra helps robots carry out longer sequences of subtasks while preserving the requested subtask order and acting on the specified objects. Our code and project page are available at https://youpengwen.github.io/Orchestra.

1 Introduction

Long-horizon instruction following requires continuity across changing task states and correct object grounding, not only capable local actions. Orchestra coordinates task interpretation, shared context, action generation, and subtask transitions, and is evaluated on benchmarks and real-world tasks.

1 Introduction

  • Memory mechanisms alone do not ensure that perceptual history and task records stay aligned with the active subtask throughout a complete instruction.Task continuity requires tracking completed and pending operations while retaining relevant observations.Source: §1 · p. 1
  • Grounding must be refreshed as objects move and goals change, because a successful action can still target the wrong object or destination.Grounding and continuity are coupled: each operation must target the intended entity at the correct point in the sequence.Source: §1 · p. 2
  • Orchestra converts complete multimodal instructions into ordered subtasks and maintains shared context for the active subtask, grounding, task records, and recent observations.At transitions, task records persist while local observation buffers and spatial grounding reset.Source: §1 · pp. 2–3
  • RMBench-Compose tests whether policies trained on atomic instructions and short trajectories can execute longer subtask sequences as object states change.The benchmark adapts five RMBench environments for variable-length, state-feasible compositions of atomic skills.Source: §1 · pp. 2–3
  • An adapted VLA policy uses subtask instructions, spatial prompts, and recent visual history, while an independent evaluator assesses subtask completion.The separate evaluator estimates progress to guide transitions.Source: §1 · pp. 2–3
  • Orchestra improves mean RMBench success by 26.22 percentage points over Mem-0, achieves 2.46× HarnessVLA’s Average Success Length on RMBench-Compose, and raises real-world mean ASL by 25.0% over the base-policy version.The evaluations span RMBench, RMBench-Compose, and real-world manipulation tasks.Source: §1 · p. 3

2 Related Work

Prior work advances local VLA capabilities, temporal memory, and agent-level task organization, while Orchestra emphasizes coordination through a shared execution context. Its method decomposes instructions, grounds entities, conditions action generation on spatial and visual context, and uses estimated progress to manage transitions.

2 Related Work

  • Prior work improves local VLA capabilities, incorporates temporal context into policies, or organizes learned policies in agent hierarchies; Orchestra adds harness-level task coordination and transition control.Its execution context aligns active subtasks, spatial grounding, observation history, and progress-driven transitions.Source: §2 · pp. 3–4

3.1 Problem Formulation and Overview

  • The shared context gives the action policy and progress evaluator a common execution interface, with completed-task records, bounded timestamped observation history, and an active-subtask index.At transitions, the controller retains task records while reinitializing visual history and spatial grounding.Source: §3.1 · pp. 4–5

3.2 Task Orchestration and Visual Grounding

  • Orchestra converts multimodal instructions into ordered executable subtasks and entity bindings, then grounds task-relevant entities in the current scene.It supports image-paired, language-only, and point- or box-referenced instructions; grounding is initialized at subtask boundaries and updated during action replanning.Source: §3.2 · p. 5

3.3 Spatially and Temporally Conditioned Action Policy

  • The adapted π0.5 policy conditions action chunks on the active subtask, spatial prompts, proprioception, and recent RGB observations, using spatial and temporal attention for visual history.Temporal fusion adds computation without increasing the visual-token count passed to the language model.Source: §3.3 · pp. 5–6

3.4 Progress Estimation and Closed-Loop Execution

  • An independent evaluator predicts temporal progress from the active subtask, spatial conditions, proprioception, and visual history; a terminal category advances the plan and resets local history and grounding.The evaluator is trained with cross-entropy on temporally labeled subtask demonstrations.Source: §3.4 · p. 6

4 RMBench-Compose Benchmark

RMBench-Compose adapts five environments to test long-horizon instruction following across feasible subtask sequences. It varies sequence length and order and evaluates whether systems preserve the requested order and select intended objects and destinations.

4 RMBench-Compose Benchmark

  • RMBench-Compose varies the length and order of feasible operations across five adapted RMBench environments.The benchmark tests complete instructions over sequences longer than those seen during atomic-skill training.Source: §4 · p. 6
  • Each environment defines atomic subtasks with language instructions, relevant entities, feasibility conditions, and success predicates.Feasibility depends on scene state; for example, an uncovering operation requires the target block to be covered.Source: §4 · pp. 6–7
  • A composed task is an ordered sequence of feasible subtasks, with each feasibility set determined by the current scene state.Sequence length and order are configurable subject to available skills and state-dependent feasibility.Source: §4 · p. 7
  • Evaluation begins with a complete instruction after training on atomic instructions and short trajectories, preserving scene state between subtasks.Episodes end when the sequence is completed or a subtask exhausts its action budget.Source: §4 · p. 7
  • Atomic SR measures isolated-task success, while ASL counts consecutive subtasks completed in order and SSR measures attempted-subtask success.SSR excludes subtasks not reached after an earlier failure.Source: §4 · p. 7

5 Experiments

The experiments evaluate memory-dependent manipulation, composition beyond atomic-training trajectories, and execution on a physical robot. The setup compares benchmark baselines and Orchestra configurations under simulated and real-world grounding conditions.

5 Experiments

  • Experiments cover RMBench, RMBench-Compose, a physical robot, and component ablations.RMBench-Compose evaluates longer subtask sequences after training on atomic instructions and short trajectories.Source: §5 · p. 7, §5.1 · p. 7

5.1 Experimental Setup

  • RMBench evaluation compares nine reported baselines with reproduced HarnessVLA and Orchestra; RMBench-Compose compares five policy and orchestration configurations.The Qwen variant changes the orchestration agent, while Orchestra with π0.5 uses the base action policy.Source: §5.1 · p. 7
  • Simulation uses environment-provided ground-truth segmentation, whereas real-world deployment initializes grounding at subtask boundaries and tracks masks during replanning.The temporally conditioned policy uses six history frames, sampled every three frames.Source: §5.1 · p. 7
  • Table 1 reports success rates on RMBench and marks the best result in each column.The evaluated RMBench tasks and rollout protocol are specified for the atomic-training, composed-evaluation setting.Source: §5.1 · p. 7, Table 1 · p. 8

Method

Orchestra performs strongly on RMBench and long-horizon compositions, while results vary by task and configuration. Ablations and real-world comparisons assess the contributions of orchestration, memory, grounding, and the conditioned policy.

5.2 Evaluation on RMBench

  • 68.22% mean success rate versus Mem-0’s 42.00% gave Orchestra the highest mean among evaluated RMBench methods.On Put Back Block it remained below Mem-0, and Swap T success was 10%.Source: §5.2 · p. 8

5.3 Compositional Execution on RMBench-Compose

  • With the same base π0.5 policy, Orchestra retained 85.0% Atomic SR while raising SSR from 47.36% to 70.78% and ASL from 0.932 to 2.884.The ASL was 3.09× the base policy’s value.Source: §5.3 · p. 8, Figure 3 · p. 9
  • 7.350 ASL was achieved by complete Orchestra, 2.46× HarnessVLA’s 2.986 ASL, across RMBench-Compose.Orchestra exceeded HarnessVLA in ASL in all five environments; Pick Up Object remained difficult at 2.11 ASL despite 81% Atomic SR.Source: §5.3 · p. 8, Figure 3 · p. 9
  • Replacing the default orchestration agent with Qwen yielded 85.74% SSR and 7.188 ASL, close to the default’s 86.69% and 7.350.Atomic SR remained 95.2%; this substitution does not establish robustness to arbitrary agent choices.Source: §5.3 · p. 8, Figure 3 · p. 9

5.4 Ablation Studies

  • Removing orchestration, short-horizon visual memory, or spatial grounding reduced mean ASL from 7.350 to 1.502, 2.500, and 5.814, respectively.The largest effect was from removing orchestration; grounding improved Cover and Uncover Block from 2.81 to 5.82 ASL.Source: §5.4 · pp. 9–10, Figure 4 · p. 9
  • 7.374 mean ASL with state-based oracle transitions was only 0.024 above Orchestra’s 7.350 with its learned progress evaluator.Absolute per-environment differences were at most 0.15, without establishing equivalent completion detection.Source: Figure 4 · p. 9, §5.4 · p. 10

5.5 Real-World Evaluation

  • Complete Orchestra improved SSR and ASL on all four real-world tasks; mean SSR rose from 72.77% to 82.09%, while mean ASL rose from 2.20 to 2.75 (25.0%).Source: §5.5 · p. 10

6 Conclusion

The conclusion presents Orchestra as a shared-context harness and RMBench-Compose as a benchmark for long-horizon composition after atomic-skill training. The paper also outlines appendices covering implementation, training, evaluation, and further analysis.

6 Conclusion

  • Orchestra coordinates task orchestration, spatial grounding, working memory, and progress-driven transitions through a shared execution context.The conclusion also introduces RMBench-Compose to evaluate long-horizon composition after atomic-skill training.Source: §6 · pp. 10–11
  • The conclusion reports 68.22% mean success on RMBench, 7.35 ASL on RMBench-Compose, and a 25.0% real-world mean-ASL improvement over the base-π0.5 configuration.It notes task-dependent performance and that simulation relies on reference grounding, motivating longer real-world evaluation with predicted grounding.Source: §6 · p. 11

Appendix Overview

  • The appendices cover framework details, policy and evaluator implementation, data and training, evaluation protocols, and additional results and failure analysis.Appendix E includes per-environment compositional results and qualitative execution examples.Source: Appendix Overview · p. 15

A.1 Task interpretation and registered capabilities

Orchestra turns a complete instruction and scene into ordered subtasks, then manages execution through shared context and progress-driven transitions. Its interfaces keep task interpretation and execution management distinct from the policy and evaluator that act on the context.

A.1 Task interpretation and registered capabilities

  • The complete instruction, current scene, and registered capabilities are converted into ordered executable subtasks with descriptions of their relevant entities.The prompt preserves operation order, dependencies, expected scene changes, and explicit repetitions without adding unsupported objects or capabilities.Source: §A.1 · p. 15, §A.2 · p. 15, Figure 6 · p. 16

A.2 Agent prompt and output schema

  • Each JSON output entry pairs an executable manipulation instruction with descriptions of relevant entities; downstream components handle localization, action generation, and completion detection.The prompt supplies the current scene, instruction, and capabilities and specifies seven rules for decomposition and entity identification.Source: §A.2 · p. 15, Figure 6 · p. 16

A.3 Shared context and spatial grounding

  • A shared context holds the active instruction, observations, recent visual history, task records, and grounded entities, while separate adapters construct policy- and evaluator-specific inputs.MolmoPoint-8B localizes entities and SAM3 initializes masks at subtask boundaries; subsequent replanning calls track them, and transitions reset grounding for the next instruction.Source: §A.3 · p. 17

A.4 What the agent remembers

  • The agent retains completed-subtask names and start and end frames across transitions, clearing these records for a new episode.Recent observations, current proprioception, grounded entities, and model-side buffers have distinct roles and follow the active subtask’s lifecycle.Source: §A.4 · pp. 17–18

A.5 Progress-driven runtime algorithm

  • At replanning calls, Orchestra predicts an action chunk and progress in parallel; a terminal prediction discards the chunk and advances the ordered plan.Otherwise, it executes up to He actions before replanning; subtask transitions reset local histories and grounding while retaining completed-subtask records.Source: §A.5 · pp. 18–19

B Policy and Evaluator Implementation

The action policy and progress evaluator use visual history and spatial conditioning, with separate model designs and training targets. The policy adds causal temporal mixing without learned memory parameters, while progress targets are generated from demonstration step positions.

B.1 Action-policy architecture

  • The π0.5-initialized policy predicts action chunks from the active subtask, spatial conditions, recent visual observations, and current proprioception.Its temporal attention reuses spatial-attention parameters and fixed temporal encodings, adding no learned memory parameters.Source: §B.1 · p. 19

B.2 Entity selection and spatial prompts

  • Training selects task-relevant entities and varies spatial and image inputs; inference uses contour-plus-bounding-box conditioning and disables those training dropouts.Spatial conditions are dropped with probability 0.2, while image dropout is applied with probability 0.1 during training.Source: §B.2 · pp. 19–20

B.3 Temporal window and encoder

  • Each camera provides six frames sampled at stride three; causal temporal attention is inserted after visual layers 4, 8, 12, 16, and 20, then historical tokens are discarded.The VLM receives 256 visual tokens per camera, matching the single-frame baseline, although history increases visual-encoder work.Source: §B.3 · p. 20, Figure 7 · p. 21

B.4 Progress-evaluator configuration

  • A learned query aggregates the evaluator’s multimodal prefix, and layer normalization plus a linear projection produces 11 progress logits.The evaluator uses an independent Gemma-300M backbone and the same six-frame, stride-three window as the policy.Source: Figure 8 · p. 21, §B.4 · p. 22

B.5 Automatically generated progress targets and transitions

  • The evaluator’s 11 progress categories are generated by rounding the normalized within-subtask step position; deployment advances when the predicted class is terminal bin 10.Completed-state frames occupy approximately the final 1/11 to 1/9 of demonstration segments, and this interval begins before the terminal-category boundary near 0.95.Source: §B.5 · p. 22

C.1 Real-world training data

The experiments combine real-world demonstrations and RMBench-Compose training on atomic instructions, then evaluate execution from complete instructions. The specified simulation setup uses persistent scene states across subtask boundaries and a fixed evaluation length per environment.

C.1 Real-world training data

  • Real-world demonstrations for four tasks were collected at 20 Hz and combined for joint training, with collection counted in complete-task units assembled from subtask-level data.Number Ordering covers all six permutations, with 24 demonstrations per order.Source: §C.1 · p. 22, Table 2 · p. 23

C.2 RMBench-Compose training data

  • For RMBench-Compose, each environment uses 50m atomic demonstrations for training and evaluates composed instructions of length 2m.Here, m is the number of task steps needed to complete one operation for every object in the environment.Source: §C.2 · p. 23, Table 3 · p. 24

C.3 Training configuration

  • The policy and evaluator use independent weights and distinct objectives—flow matching for action prediction and cross-entropy for progress classification—with 50,000 training steps and global batch size 20.The shared configuration spans two Nvidia A6000 Pro GPUs.Source: §C.3 · p. 23

D.1 Composed instructions and execution budgets

  • Each RMBench-Compose configuration is evaluated for 100 episodes using complete instructions, with scene states persisting across subtask boundaries and one specified evaluation length per environment.The terminal-category prediction advances the complete system; environment-state completion is used as the switching signal only in the oracle-transition ablation.Source: §D.1 · pp. 23–24, Table 3 · p. 24

D.2 Metrics and real-world evaluation

Evaluation uses metrics for isolated atomic tasks, attempted subtasks, and consecutive in-order completion, with any subtask failure ending a rollout. The RMBench baseline comparison combines officially reported results with reproduced HarnessVLA and Orchestra evaluations.

D.2 Metrics and real-world evaluation

  • Average Success Length measures the mean consecutive subtasks completed in requested order, while Subtask Success Rate counts successful attempted subtasks and excludes unreached subtasks.Atomic Success Rate measures isolated atomic-task execution; any subtask failure terminates the rollout.Source: §D.2 · p. 24

D.3 Baseline comparability

  • The RMBench comparison uses official results for nine baselines and reproduces HarnessVLA (Codex), alongside Orchestra evaluations.HarnessVLA uses Codex as its default agent, so the table combines reported benchmark results with reproduced evaluations.Source: §D.3 · p. 24
  • Orchestra generates the RMBench subtask sequence and entity descriptions, while the simulator resolves descriptions to instances and supplies ground-truth masks for spatial conditioning.For example, the description “the left cover” is resolved to an object instance and its segmentation mask.Source: §D.3 · p. 24

D.4 Physical robot platform

  • Real-world experiments use a dual-arm robot with head and wrist cameras; its 16-dimensional proprioceptive state and padded 32-dimensional action representation are shared across policy configurations.The platform has two 7-DoF arms, and RGB images are resized to 224 × 224.Source: §D.4 · p. 25

E.1 Per-environment compositional results

The appendix contrasts isolated and composed-task performance, describes ablation configurations, and illustrates execution on benchmark and real-world tasks. These results and visualizations distinguish task-level progress from isolated skill success.

E.1 Per-environment compositional results

  • On Swap Block, base π0.5 matched Orchestra at 98.0% Atomic SR, while composed-task SSR rose from 53.27% to 86.09% and ASL from 1.14 to 6.19.Figure 10 compares Atomic SR and composed-task SSR; isolated-task success rate is not a per-step success probability in a composed rollout.Source: §E.1 · pp. 25–26, Figure 10 · p. 25

E.2 Ablation implementation details

  • The ablations remove subtask decomposition, short-horizon visual memory, or spatial grounding individually; an oracle-transition variant substitutes an environment-state completion signal.The no-orchestration policy is retrained on long-horizon instructions and demonstrations while retaining visual memory and spatial grounding.Source: §E.2 · p. 26

E.3 Qualitative visualizations

  • Qualitative visualizations of RMBench-Compose and real-world tasks complement the quantitative results in Sections 5.3 and 5.5.These visualizations are presented as execution examples alongside the quantitative evaluation.Source: §E.3 · p. 26

E.3.1 RMBench-Compose

  • Selected chronological RMBench-Compose sequences show successive subtasks changing the scene, with later actions operating on configurations created by earlier actions.The five examples show selected portions, not every step of the full evaluation instructions.Source: §E.3.1 · p. 26, Figure 11 · p. 26

E.3.2 Real-world execution

  • Real-world visualizations cover four tasks on the custom dual-arm platform, combining RGB observations, spatial masks, and within-subtask progress traces.Masks are initialized at subtask boundaries and tracked at later action-chunk replanning calls; transitions use terminal-bin argmax, not the plotted progress trace.Source: Figure 9 · p. 25, Figure 12 · p. 27, §E.3.2 · p. 28

E.4 Failure case analysis

The failure analysis identifies action execution as the largest share of recorded failures, alongside errors in planning, grounding, and progress estimation. Its examples show how failures at different stages can leave the overall task incomplete.

Task decomposition

  • A generated plan can repeat one object and omit another, so executing every listed subtask may still leave the instruction incomplete.The example duplicates the yellow-edged snack bag and omits the white-and-green box.Source: §E.4 · p. 29

Object localization and segmentation

  • Grounding errors include selecting the wrong object and extracting only part of a target region; a valid point or nonempty mask alone is insufficient.The examples show a wrong-object localization and a partial mask on the blue mat.Source: §E.4 · p. 29

Progress estimation

  • Delayed completion detection can leave a physically completed placement unrecognized, preventing transition until the evaluator predicts the terminal progress category.In the example, bread is placed but the evaluator does not trigger the next subtask promptly.Source: §E.4 · p. 29

Action execution

  • Task orchestration and spatial conditioning do not eliminate inaccurate grasp poses, unintended contact, or unstable placement.Examples include a misplaced fork, a failed bottle grasp, an apple knocked away, and a toppled number plate.Source: §E.4 · p. 29

Failure distribution and implications

  • Action execution accounts for 87.7% of 65 recorded failures, versus 6.1% for progress estimation, 4.6% for localization, 1.5% for task decomposition, and 0.0% for segmentation.The segmentation tally remains 0.0% despite an incomplete-mask example; the distribution describes this evaluation’s recorded failures.Source: §E.4 · pp. 28–29, Figure 13 · p. 28

Related papers

Loading 2610.18352v1…