PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring the Limits of Combinatorial Generalization in Embodied Manipulation

Xupeng Zhang, Yantai Yang, Chang Guo, Zhaokai Yin, Zhipeng Zhang

arXiv:2610.15425v1cs.RO

TL;DR

Long-horizon manipulation requires VLAs to recombine atomic skills, but evidence on execution of unseen ordered compositions without composed training trajectories is limited. The paper introduces RoboCombine and HSE to evaluate and diagnose atomic-only recombination, and PRTI to broaden atomic start-state coverage. The authors report that PRTI improves ordered execution without composed training trajectories, while the study is limited to recombining familiar atomic task labels.

  • Problem

    Long-horizon manipulation requires composing atomic skills, but the paper targets the limited evidence on unseen ordered compositions trained without composed trajectories.

    Source: §1 · pp. 1–2

  • Method

    The paper introduces RoboCombine for atomic-only composition evaluation, HSE for diagnosing execution failures, and PRTI for broadening atomic start-state coverage.

    Source: §1 · p. 2

  • Results

    PRTI broadens atomic start-state support and improves ordered execution without composed training trajectories, while finite full-instruction training mainly fits seen compositions rather than recombining unseen ones.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Reliable ordered execution requires downstream atomic skills to work from handoff states, not only canonical atomic starts.

    Source: §1 · p. 2, §6 · p. 10

    The study concerns recombining familiar atomic task labels, not generalization to unseen atomic tasks, new embodiments, or open-ended real-world environments.

    Source: A Limitations and future work · p. 13

Abstract

from arXiv · show

In embodied manipulation, Vision Language Action (VLA) models can execute atomic skills, but long-horizon manipulation requires recombining these skills without demonstrations for every skill combination. The full instruction route treats skill chaining as long-horizon imitation. In a controlled study, full instruction training on limited skill combination sequences reaches 40.0% ordered success on covered combinations but only 7.7% on recombination, suggesting memorization of sequence templates rather than robust skill recombination. We introduce Hierarchical Subtask Execution (HSE), a diagnostic framework that exposes skill recombination through a shared modular interface over the planner, trigger, and policy, decomposing long-horizon failures into separable modes. We introduce RoboCombine, a benchmark for skill recombination without combined skill training trajectories, with metrics for ordered success and transition conversion across subtask boundaries. RoboCombine reveals that atomic competence alone does not ensure ordered execution. Failures concentrate at transitions where downstream subtasks launch from handoff states from preceding subtasks rather than canonical atomic starts. We identify this mismatch as the start state gap. To address it, we introduce Pose Robust Task Initialization (PRTI), a lightweight atomic task post-training strategy that broadens start state coverage beyond canonical starts. Under oracle timed HSE on RoboCombine, PRTI reaches 18.16% ordered success, compared with 4.21% under random perturbation and 9.47% under a robot-reset oracle, without combined skill training trajectories. These results show that combinatorial generalization requires handoff robust atomic skills that convert first stage progress into reliable downstream execution, not merely decomposition into known subtasks. Code: https://github.com/AutoLab-SAI-SJTU/RoboCombine.

1 Introduction

Full-instruction training performs poorly on novel skill compositions, while RoboCombine and HSE make atomic-only recombination and its failure modes measurable. The introduction identifies downstream handoff states as a gap in atomic training and proposes broader start-state coverage through PRTI.

1 Introduction

  • 7.7% average ordered success across partially overlapping and disjoint compositions exposes the limits of full-instruction training on finite sequence combinations.The reported performance degrades from seen to increasingly novel compositions.Source: §1 · p. 2, Figure 1 · p. 2
  • RoboCombine evaluates unseen ordered compositions using atomic-only training and reports ordered success and transition success across subtask boundaries.Its simulation setting trains on 50 atomic tasks and tests 19 novel compositions; the real-robot setting trains on 16 atomic tasks and tests 13 compositions.Source: §1 · p. 2
  • HSE separates the planner, trigger, and policy to diagnose failures in subtask sequencing, boundary timing, and downstream execution from handoff states.The shared modular interface treats hierarchy as a measurement interface rather than only an execution architecture.Source: §1 · p. 2
  • The start state gap is the mismatch between canonical atomic training starts and downstream handoff states; PRTI broadens atomic start-pose coverage to address it.PRTI preserves an atomic-only data regime while targeting poses relevant to downstream subtasks.Source: §1 · pp. 2–3

2 Related Work

Related work spans full-instruction long-horizon VLA training and hierarchical execution of known atomic skills. The paper focuses on whether unseen ordered compositions can be recombined under atomic-only training, a boundary existing benchmarks do not isolate explicitly.

2 Related Work

  • Full-instruction methods train or adapt VLAs on complete multi-stage demonstrations, a route suited to sequence distributions covered by composed data.The paper focuses instead on finite composed-trajectory coverage and unseen ordered compositions.Source: §2 · p. 3
  • Hierarchical methods switch among known subtasks, but prior benchmarks do not explicitly isolate sequencing, handoff timing, and downstream execution from handoff states.CALVIN and LIBERO make language-conditioned sequence execution and transfer measurable.Source: §2 · p. 3

3 RoboCombine Benchmark and HSE Diagnostics

ROBOCOMBINE tests whether familiar atomic skills can be recombined into ordered compositions without composed training trajectories. HSE separates subtask order, handoff timing, and atomic execution to make recombination failures diagnosable.

3.1 RoboCombine Benchmark

  • ROBOCOMBINE evaluates two-stage compositions of familiar atomic instructions without exposing models to complete evaluation sequences during training.Each target subtask starts from the source subtask’s terminal state.Source: §3.1 · p. 3, Figure 2 · p. 4
  • Ordered success measures completion of both subtasks in order, while transition conversion measures ordered success conditional on completing the first subtask.Source: §3.1 · p. 3
  • The benchmark uses 20 rollouts per composition, totaling 380 episodes per policy variant across 15 object-arrangement and four placement-to-interaction compositions.Source: §3.1 · p. 3

3.2 HSE Diagnostic Framework

  • HSE controls the subtask queue, handoff cue, and atomic policy separately, unlike full-instruction fine-tuning’s single composed imitation target.Source: §3.2 · p. 4, Figure 3 · p. 5
  • HSE latches a handoff cue during an action chunk and changes the active instruction before the next chunk, enabling matched modes to isolate queue, cue, and policy effects.Source: §3.2 · p. 5

4 Analysis and Pose Robust Repair

Full-instruction fine-tuning performs substantially better on seen compositions than on other evaluation sets, while oracle HSE exposes failures in downstream execution from handoff states. PRTI broadens atomic-skill start-state coverage using single-skill data.

4.1 Diagnostic Analysis

  • 19.7% aggregate ordered success with Oracle-HSE was Motus’s highest result across its three modes, compared with 17.4% under Ordered-HSE and 10.0% under Full-Inst.Source: Table 1 · p. 6, §4.1 · p. 7
  • Transition conversion varies by source-target pair, showing that successful first-stage completion does not reliably lead to downstream execution from the resulting handoff state.Source: Figure 4 · p. 6, §4.1 · p. 7
  • 40.0% ordered success on set A fell to 15.0% on set B and 4.4% on set C, averaging 7.7% across sets B and C under full-instruction fine-tuning.Source: §4.1 · pp. 5–6

4.2 Pose Robust Task Initialization

  • PRTI broadens each atomic skill’s start-state coverage using task-valid states drawn from late segments of successful atomic trajectories, without composed demonstrations.Source: §4.2 · p. 7
  • PRTI post-training mixes canonical and PRTI trajectories at a 1:1 ratio while conditioning the policy only on the target atomic instruction.Source: §4.2 · p. 7

5 Experiments

Experiments evaluate atomic-skill recombination under controlled HSE settings and on real-robot sequences. HSE performs better than full-instruction execution on real-robot compositions, while PRTI improves ordered success over initialization controls.

5 Experiments

  • The experiments ask whether PRTI improves downstream execution while preserving atomic-only training, and whether HSE and PRTI transfer to variable-length physical sequences.PRTI is the only added intervention in its matched real-robot comparison with ORACLE-HSE.Source: §5 · p. 7, §5.3 · p. 8

5.1 Experimental Setup

  • Simulations use RoboCombine with oracle HSE to remove subtask-ordering and boundary-timing uncertainty, while real-robot tests use Order HSE with online progress-predicted handoffs.The reported metrics include ordered success, first- and second-subtask success, and transition conversion.Source: §5.1 · p. 7

5.2 PRTI Broadens Atomic Start State Support

  • 18.16% ordered success with PRTI exceeds 4.21% with Random and 9.47% with Reset under oracle HSE.The comparison uses an oracle queue and handoff cue to remove ordering and boundary-timing uncertainty; PRTI executes from complete handoff states without an artificial reset.Source: §5.2 · p. 8, Table 2 · p. 8

5.3 Real-World Deployment

  • The real-robot suite contains 13 compositions of two to five steps, split into seen, partially overlapping held-out, and unseen compositions.The dual-arm platform uses three camera views; evaluation uses five rollouts per task under a shared ordered-milestone rubric.Source: §5.3 · p. 8, Table 3 · p. 9
  • ORDER-HSE achieves 0.67 average real-robot mean progress, versus 0.16 for FULL-INST and 0.35 for FULL-INST-FT, and scores 0.77 on unseen Set C tasks.With matched oracle switching, PRTI raises mean progress from 0.75 to 0.80; it improves 8 of 13 tasks, matches on four, and decreases on one.Source: §5.3 · pp. 9–10, Table 3 · p. 9

6 Conclusion

The conclusion presents RoboCombine and HSE for evaluating skill recombination without composed training trajectories. It reports that PRTI mitigates the start-state gap, while the LLM component serves only to select and order atomic skills.

6 Conclusion

  • RoboCombine benchmarks atomic-only recombination, and HSE diagnoses it using oracle- or progress-based handoffs; the conclusion identifies the start-state gap and reports that PRTI improves ordered execution without composed trajectories.The reported gap is between canonical atomic training starts and handoff states for downstream subtasks.Source: §6 · p. 10

AI Use Statement

  • The LLM decomposes instructions into ordered atomic-task labels but does not generate robot actions or replace the VLA policy.The authors state that they reviewed the LLM-assisted component and take responsibility for the methodology, experiments, and conclusions.Source: AI Use Statement · p. 10

A Limitations and future work

The paper limits its claims to recombining familiar atomic skills and outlines extensions to longer, more varied chains and jointly learned autonomous execution. The appendices specify the benchmark and HSE policy interface.

A Limitations and future work

  • The evidence concerns compositions of familiar atomic task labels, not unseen atomic tasks, new embodiments, or open-ended real-world environments.Future work includes longer and more diverse chains, repeated subtasks, and joint learning and evaluation of decomposition, progress prediction, recovery, and execution.Source: A Limitations and future work · p. 13

B RoboCombine Details

  • RoboCombine’s task appendix lists evaluated compositions with their corresponding atomic instructions; a separate table reports per-task RoboTwin atomic success rates for context.The main overview figure uses short labels, while Table 4 gives the complete source and target instructions.Source: B RoboCombine Details · p. 13, Table 4 · p. 13, Table 5 · p. 14

C HSE Details

  • HSE has task-decomposition, progress-prediction, and subtask-switching components, while the VLA remains an instruction-conditioned atomic executor.HSE does not modify the VLA policy itself.Source: C HSE Details · pp. 13, 15

C.1 VLA policy interface

  • At each invocation, the VLA receives an observation history and one active atomic instruction; the instruction can change only between action-chunk queries.The policy is never conditioned on multiple atomic instructions in a single invocation.Source: §C.1 · p. 15

C.2 Task decomposition

HSE decomposes commands into atomic-task queues, uses predicted progress to trigger handoffs, and changes instructions between policy queries. PRTI broadens atomic start-state coverage while keeping its training examples single-skill demonstrations.

C.2 Task decomposition

  • The decomposer maps a composed command to an ordered queue constrained to valid atomic-task labels; diagnostic settings can substitute the benchmark queue to isolate later failure sources.RoboCombine commands contain two stages, and extra LLM output is parsed away.Source: §C.2 · p. 15

C.3 Progress predictor for online handoffs

  • The progress predictor estimates completion on a 0-to-1 scale from frame-and-instruction features, triggering the next queued instruction at a switching threshold.It uses a Florence-based encoder and recurrent progress head; training uses cached features and a smooth tanh-shaped target.Source: §C.3 · p. 15

C.4 Subtask switching

  • HSE monitors and latches the progress predicate during an action chunk, then updates the active subtask index before the next VLA query without resetting the environment.The downstream policy therefore starts from the physical state produced by the preceding subtask.Source: §C.4 · p. 16

D PRTI Details

  • PRTI post-trains the atomic policy using single-skill demonstrations and changes the distribution of atomic initial states without using complete composed trajectories.The described mixture contains about 10K episodes, around 2M frames, and several hundred atomic task prompts.Source: D PRTI Details · p. 16

D.1 Reset and initialization controls

  • 69/380 episodes (18.16%) succeed with PRTI, compared with 16/380 (4.21%) with Random and 36/380 (9.47%) with Reset across 19 compositions under oracle-timed HSE.Random uses unstructured initial-state perturbations; Reset restores only the robot pose, leaving object and contact states unchanged.Source: §D.1 · pp. 16–17

E.1 Atomic tasks

The real-robot study instantiates 16 atomic manipulation tasks on a dual-arm AgileX platform, evaluating each independently as a building block for compositional experiments.

E.1 Atomic tasks

  • The study instantiates 16 atomic manipulation tasks on a dual-arm AgileX robot platform with three camera views.Source: §E.1 · p. 17
  • The tasks cover drawer interaction, button pressing, object pick-and-place, and object transfer between drawer and tabletop regions.Objects include an orange persimmon and a cube; the workspace includes a drawer, plates, boards, and start/stop buttons.Source: §E.1 · p. 17
  • Each task has a natural-language prompt and is evaluated independently, with per-task success rates reported in Table 7.These atomic skills serve as building blocks for the compositional real-robot experiments.Source: §E.1 · p. 17

E.2 Mean-progress metric

Real-robot mean progress scores each method by the furthest ordered milestone reached across five physical rollouts, using a shared task-length-specific rubric.

E.2 Mean-progress metric

  • Each method and compositional task receives an average mean-progress score over five physical rollouts.Source: §E.2 · p. 17
  • Each rollout is scored by the furthest ordered milestone reached under a fixed rubric tailored to task length.Source: §E.2 · p. 17
  • The same rubric applies to all methods, and its milestones capture atomic-subtask completion and successful handoff to the next subtask.Table 8 gives the complete scoring protocol.Source: §E.2 · p. 17

E.3 Visualization

Real-robot visualizations show progress-predictor scores tracking subtask completion and switching across compositional executions. The accompanying tables document task prompts, atomic-task success rates, and the ordered-milestone scoring rubric.

E.3 Visualization

  • The two-step task 1-2 trace shows the progress score rising during a stage, crossing the switching threshold near its end, then dropping after the switch.Figure 6 marks stage boundaries and switch moments alongside rollout frames.Source: §E.3 · p. 17, Figure 6 · p. 18
  • Across four compositional tasks, score curves generally form stage-wise peaks aligned with subtask completion and switching events.Figure 7 shows sampled rollout frames by task, with robot observations beside progress-predictor scores.Source: §E.3 · p. 17, Figure 7 · p. 20
  • The visualizations illustrate online monitoring of subtask progress to coordinate sequential atomic-skill execution on the real robot.Source: §E.3 · p. 17
  • The accompanying tables cover task prompts and episode counts, π0.5 atomic-task success rates, and the ordered-milestone rubric.For five-stage tasks, the rubric gives scores for initiating and completing each subtask.Source: Table 6 · p. 19, Table 7 · p. 19, Table 8 · p. 20
Loading 2610.15425v1…