PaperlayerView exact revision on arXiv

Source-linked AI summary

LiLo-VLA: Compositional Long-Horizon Manipulation via Linked Object-Centric Policies

Yue Yang, Shuo Cheng, Yu Fang, Homanga Bharadhwaj, Mingyu Ding, Gedas Bertasius, Daniel Szafir

arXiv:2602.21531v1cs.ROcs.AIcs.CVcs.LGcs.SYeess.SY

TL;DR

Current VLA approaches struggle to recombine learned atomic skills into novel long-horizon sequences, while environmental sensitivity can make execution brittle and failures cascade. LiLo-VLA combines a motion-planning Reaching Module for global transport with an object-centric VLA Interaction Module for local manipulation and skill reuse. LiLo-VLA reports a 69% average success rate, versus 28% for Pi0.5 and 2% for OpenVLA-OFT [22]; real-world evaluation achieves 85% average success across 8 tasks.

  • Problem

    Current VLA approaches struggle to recombine learned atomic skills into novel long-horizon sequences, while environmental sensitivity can make execution brittle and failures cascade.

    Source: §I · p. 1

  • Method

    LiLo-VLA combines a motion-planning Reaching Module for global transport with an object-centric VLA Interaction Module for local manipulation and skill reuse.

    Source: §I · p. 2

  • Results

    LiLo-VLA reports a 69% average success rate, versus 28% for Pi0.5 and 2% for OpenVLA-OFT [22]; real-world evaluation achieves 85% average success across 8 tasks.

    Source: §I · p. 2, B. Main Results: Zero-Shot… · p. 6

  • Takeaways & Limitations

    The evaluations cover randomized sequencing, visual clutter, and horizons up to 16 skills in simulation, as well as real-world long-horizon tasks.

    Source: §I · p. 2

    Transparent or severely occluded objects remain challenging, and overall system performance is bounded by the underlying VLA backbone’s atomic proficiency.

    Source: §VI · p. 9

Abstract

from arXiv · show

General-purpose robots must master long-horizon manipulation, defined as tasks involving multiple kinematic structure changes (e.g., attaching or detaching objects) in unstructured environments. While Vision-Language-Action (VLA) models offer the potential to master diverse atomic skills, they struggle with the combinatorial complexity of sequencing them and are prone to cascading failures due to environmental sensitivity. To address these challenges, we propose LiLo-VLA (Linked Local VLA), a modular framework capable of zero-shot generalization to novel long-horizon tasks without ever being trained on them. Our approach decouples transport from interaction: a Reaching Module handles global motion, while an Interaction Module employs an object-centric VLA to process isolated objects of interest, ensuring robustness against irrelevant visual features and invariance to spatial configurations. Crucially, this modularity facilitates robust failure recovery through dynamic replanning and skill reuse, effectively mitigating the cascading errors common in end-to-end approaches. We introduce a 21-task simulation benchmark consisting of two challenging suites: LIBERO-Long++ and Ultra-Long. In these simulations, LiLo-VLA achieves a 69% average success rate, outperforming Pi0.5 by 41% and OpenVLA-OFT by 67%. Furthermore, real-world evaluations across 8 long-horizon tasks demonstrate an average success rate of 85%. Project page: https://yy-gx.github.io/LiLo-VLA/.

I. INTRODUCTION

The introduction frames long-horizon manipulation as difficult because novel skill sequences and environmental variation challenge current VLA approaches. It presents LiLo-VLA as a modular approach and evaluates it on new task sequences in simulation and the real world.

A. Vision-Language-Action Models

  • Current VLA methods show promise on short-horizon skills, but struggle to recombine skills into unseen sequences and can fail across long tasks when visual features or spatial configurations vary.A single failure can jeopardize the task sequence.Source: §I · pp. 1–2, A. Vision-Language-Action Models · p. 2

B. Long-Horizon Manipulation and Skill Chaining

  • Prior modular approaches face skill-chaining and integration challenges, while unified approaches couple transport with interaction; recovery methods also assume robust underlying skills.These limits motivate separating transport from interaction and making recovery operate at the skill level.Source: §I · p. 2, B. Long-Horizon Manipulation and Skill… · pp. 2–3

I. INTRODUCTION

  • LiLo-VLA combines classical motion planning for global reaching with an object-centric VLA for local manipulation and reuses atomic skills to generalize zero-shot to novel tasks.Its object-focused observations are intended to reduce sensitivity to global layouts and background clutter.Source: §I · p. 2
  • The 21-task benchmark spans LIBERO-Long++ and Ultra-Long, both evaluating compositional generalization, with the former focused on visual robustness and the latter on extreme temporal scalability up to 16 sequential skills.Source: §I · p. 2
  • Extensive experiments under the simulation-performance heading show LiLo-VLA achieves a 69% average success rate, significantly outperforming Pi0.5 (28%) and OpenVLA-OFT (2%).Source: §I · p. 2
  • 85% average success rate is reported across 8 real-world long-horizon tasks, with sequences of up to 8 skills.The tasks include complex backgrounds and varied sequences.Source: §I · p. 2

III. METHODOLOGY

LiLo-VLA treats a long-horizon task as a sequence of symbolic actions and executes each through global reaching followed by local interaction. The Reaching Module plans collision-free transport to an object-relative approach pose, while pose perturbations during demonstrations help the downstream policy handle deployment deviations.

A. Overview of LiLo-VLA

  • LiLo-VLA executes each symbolic action through a two-phase process: motion planning brings the end-effector to an object-relative approach pose, then a learned policy performs contact-rich manipulation.The task plan is a sequence of parameterized symbolic actions, each tied to a reference object and auxiliary parameters.Source: A. Overview of LiLo-VLA · p. 3

B. The Reaching Module: Global Transport

  • The Reaching Module bridges atomic skills by planning collision-avoiding motion from the previous skill’s end state to a precise approach pose for the next interaction.The pose also provides a deterministic target for the planner at deployment.Source: B. The Reaching Module: Global Transport · p. 3

1) Relative Goal Generation with Perturbation:

  • The approach pose is defined relative to the reference object, with a fixed face-down orientation and a skill-specific translation such as pick clearance.The transformation applies an offset selected according to the skill.Source: 1) Relative Goal Generation with… · p. 3
  • Randomizing demonstration start poses around the canonical approach pose exposes the VLA to translation and rotation perturbations, helping it compensate for pose-estimation and planning errors at deployment.The perturbation is sampled in SE(3) with zero mean and covariance Σ.Source: 1) Relative Goal Generation with… · p. 3

2) Collision-Free Motion Planning:

  • MPLib uses the environmental point cloud and robot kinematics to generate a feasible, collision-free path to the target pose without learning the global transport motion.At deployment, the target is the canonical approach pose; demonstrations use the perturbed start pose.Source: 2) Collision-Free Motion Planning · p. 3

C. The Interaction Module: Object-Centric VLA

The Interaction Module uses an object-centric visuomotor policy for local manipulation, restricting its observations to wrist-camera views and masking non-target objects. Training augmentation simulates the black-rectangle masks used at deployment to keep masked observations closer to the policy’s training distribution.

C. The Interaction Module: Object-Centric VLA

  • The Interaction Module learns an end-to-end visuomotor policy for reactive end-effector actions, with observations centered on the object involved in the current skill.The Reaching Module first brings the end-effector into the object’s vicinity.Source: C. The Interaction Module:… · p. 3

1) Object-Centric Observation Space:

  • Wrist-camera views provide a consistent perspective relative to the workspace, whereas fixed global views can change with robot position and contribute to observation-space shift.The paper reports that wrist-only policies outperform multi-view configurations in atomic skill success rates.Source: 1) Object-Centric Observation Space · pp. 3–4

2) Visual Clutter Augmentation:

  • Deployment masks non-target objects with black rectangles; training simulates these occlusions by randomly erasing background regions while preserving the gripper, reference object, and grasped object.This augmentation is intended to keep masked inference observations within the policy’s learned distribution.Source: 2) Visual Clutter Augmentation · p. 4

D. Compositional Execution and Failure Recovery

For each planned skill, the system estimates the target pose, reaches an approach pose, executes the interaction, and verifies the result. Recovery depends on whether the failed skill involves retaining an object; the benchmark tests skill reordering, clutter robustness, and extended task horizons.

D. Compositional Execution and Failure Recovery

  • Each skill runs through target-pose estimation, collision-free reaching to an approach pose, VLA interaction, and geometric verification of the resulting object configuration.The verification checks whether the skill’s intended effect has been satisfied.Source: 1) Sequential Execution Pipeline · p. 5

2) Closed-Loop Recovery Mechanism:

  • A failed skill without object retention is retried after re-estimating the scene and resetting to the approach pose; a failed object-retaining skill backtracks to the most recent Pick.The system assumes an object may have been dropped or lost during a failed transport, making a local retry risky.Source: 2) Closed-Loop Recovery Mechanism · p. 5

1) Benchmarks:

  • 21 tasks span six cluttered, reorderable LIBERO-Long++ scenarios and three Ultra-Long tasks of 9, 10, and 16 steps, with multiple permutations per scenario.The benchmark is derived from nine core scenarios in the LIBERO environment.Source: 1) Benchmarks · p. 5

2) Baselines:

The section specifies how the benchmarks, baselines, metrics, and implementation probe long-horizon compositionality. It reports LiLo-VLA’s overall simulation success-rate advantage over two generalist policies.

2) Baselines:

  • Pi 0.5 and OpenVLA-OFT are the generalist-policy baselines; Suite 1 uses LoRA fine-tuning for standard sequences and reordered language prompts for variants.For Suite 2, baselines train on constituent atomic skills and receive sequential subtask prompts at inference.Source: 2) Baselines · p. 5

3) Evaluation Metrics:

  • Success Rate requires completing the entire skill sequence in order, while Average Progress counts only the continuous prefix of correctly ordered skills.Each configuration is evaluated over 10 trials; simulation uses oracle poses and segmentation masks to isolate manipulation and planning failures from perception errors.Source: 3) Evaluation Metrics · p. 5, 4) Implementation Details · p. 5

4) Implementation Details:

  • Training segments LIBERO-90 demonstrations into atomic skills and augments their initial states; the policy uses OpenVLA-OFT, wrist-camera input, and background random erasing.At inference, a motion planner reaches an approach pose, then the Interaction Module executes the instructed skill and geometric heuristics can trigger recovery.Source: 4) Implementation Details · p. 5
  • LIBERO-Long++ tests visual robustness with distractors, while Ultra-Long tests temporal scalability with sequences up to 16 steps; both vary skill order.The benchmark contains 21 tasks across the two suites.Source: Figure 3 · p. 6

B. Main Results: Zero-Shot Compositionality and Scalability

  • LiLo-VLA achieves 69% average success, compared with 28% for Pi0.5 and 2% for OpenVLA-OFT.On LIBERO-Long++ variants, LiLo-VLA reaches 85% success while Pi0.5 falls to 0%; on Ultra-Long, LiLo-VLA reaches 44% while both baselines score 0%.Source: B. Main Results: Zero-Shot… · p. 6

1) Necessity and Robustness of the Reaching Module:

The section tests the reaching module, interaction policy, and recovery mechanism through ablations and robustness evaluations. It also describes the real-world hardware and perception stack used for deployment.

1) Necessity and Robustness of the Reaching Module:

  • Removing the Reaching Module yields 0% success despite access to oracle object poses.The ablation tests whether the policy can handle transport without the module that decouples it from interaction.Source: 1) Necessity and Robustness of the… · p. 6
  • With initial-state perturbation during training, the policy shows no statistically significant success-rate drop under noisy execution poses; the canonical-endpoint baseline degrades.The evaluation aggregates 36 atomic skills from both suites.Source: 1) Necessity and Robustness of the… · p. 6, Figure 4 · p. 6

2) Visual Robustness via Object-Centric Design:

  • Wrist Only reaches 0.88 success on original tasks and a 15.9% Ratio Performance Delta under OSS, versus 18.8% for Both Views and 29.8% for 3rd Person.The evaluation uses the BOSS-C1 benchmark.Source: 2) Visual Robustness via… · p. 7
  • Removing random erasing lowers overall success from 69% to 48%, indicating that wrist-camera input alone does not eliminate distractor features.The authors use masking during training to reduce overfitting to irrelevant visual features.Source: 2) Visual Robustness via… · p. 7

3) Efficacy of Closed-Loop Recovery:

A. Real-World System Implementation

  • The real-world system uses a Franka Panda, global and wrist cameras, YOLOE and FoundationPose for object poses, and a Pi0.5 backbone instead of simulation’s OpenVLA-OFT.Training demonstrations cover atomic skills only, and success verification uses a human-in-the-loop protocol.Source: A. Real-World System Implementation · pp. 7–8

B. Evaluation Tasks: Stress-Testing Generalization

The real-world evaluation separates resilience to layout and visual changes from skill-order generalization, and reports strong results across both. The conclusion identifies perception and backbone proficiency as limitations.

B. Evaluation Tasks: Stress-Testing Generalization

  • The three physical tasks comprise two four-skill sequences and an eight-skill Task 3; altered layouts with distractors and permuted skill orders test distinct generalization axes.The order changes test behaviors not seen during data collection.Source: B. Evaluation Tasks: Stress-Testing… · p. 8

C. Results Analysis

  • 100% success across all three Standard tasks; Diverse Layout yielded 100% on Task 1 and 80% on Task 2, while 8-step Task 3 reached 60% success and 85% average progress.Each configuration used five evaluation trials.Source: C. Results Analysis · p. 8

VI. CONCLUSION AND LIMITATIONS

  • LiLo-VLA combines global transport with local manipulation and closed-loop recovery to address observation-space shifts in simulation and real-world deployment.Source: §VI · p. 8
  • External perception makes accurate detection of transparent or severely occluded objects challenging, and system performance is bounded by the underlying VLA’s atomic proficiency.Source: §VI · pp. 8–9

Appendix for LiLo-VLA

The appendix documents the task and skill inventories, real-world rollouts, and visual masking and recovery mechanisms. It also describes the broader simulation skill library.

Appendix for LiLo-VLA

  • The supplementary material covers real-world visualizations, task sequences, relative-pose methodology, data-efficiency analysis, training details, and hardware and perception specifications.Source: OVERVIEW · p. 11

A. Real-World Task Specifications

B. Real-World Rollout Visualizations

  • Rollout visualizations mark the active target in green and distractors in red; supplementary figures show random masking and pose re-estimation followed by a retry after grasp failure.Masking directs attention to the target object, while the recovery sequence resets the arm to the approach pose.Source: B. Real-World Rollout Visualizations · p. 11

A. Atomic Skill Library

B. Detailed Task Specifications

The detailed specifications define reordered and extended task sequences, while the methodology and complexity analysis describe spatially invariant inputs and the scaling advantage of modular skill learning.

B. Detailed Task Specifications

  • Table S4 lists task sequences; LIBERO-Long++ has six core tasks with order-permuted variants, while Ultra-Long has three scenarios, including a 16-step Table Organization stress test.Source: B. Detailed Task Specifications · pp. 11–12

IX. EXTENDED METHODOLOGY DETAILS

  • The simulation policy uses end-effector pose relative to the target object, a representation that makes interaction invariant to global workspace shifts.The real-world Pi0.5 policy operates in joint space and does not use this Cartesian transformation.Source: §IX · p. 12
  • With random object-position perturbations up to 20 cm, success across 22 atomic skills changed only from 0.80 to 0.75, with no statistically significant degradation (p > 0.05).Source: §IX · p. 12

X. DATA EFFICIENCY AND COMBINATORIAL COMPLEXITY

  • End-to-end policies must cover valid skill orderings; for the 16-step, eight-object tasks, the combinatorial trajectory space is intractable to cover through demonstrations alone.Source: §X · pp. 12–13
  • LiLo-VLA’s object-centric interaction learning scales with the number of unique atomic skills, O(|U|), rather than horizon length or task-structure complexity.For the 16-step Table Organization task, the analysis contrasts millions of potential trajectory variations with nine unique primitives to master.Source: §X · p. 13

XI. TRAINING AND IMPLEMENTATION DETAILS

Simulation training uses OpenVLA-OFT with a parallel action-chunk regression objective, while real-world training uses Pi0.5 with Conditional Flow Matching. The supplied materials also identify the simulation task-specification table and real-world hardware setup.

A. Training Objectives

  • OpenVLA-OFT simulation training minimizes L1 loss between predicted and ground-truth action chunks, predicting each chunk in parallel from the observation and instruction.H is the action chunk size.Source: A. Training Objectives · p. 13
  • Pi0.5 real-world training uses Conditional Flow Matching, minimizing mean squared error between a predicted vector field and target flow along a linear interpolation path.The interpolation is ϕτ(x0, x1) = (1 − τ)x0 + τx1.Source: A. Training Objectives · p. 13
  • Table S4 provides detailed task specifications for the simulation benchmarks.Source: Table S4 · p. 14
  • The real-world setup uses a Franka Panda, a wrist-mounted ZED Mini for fine interaction, and a fixed ZED 2i camera for global reaching.Perception and control run locally; VLA inference runs on a remote cluster.Source: Figure S4 · p. 14

B. Training Hyperparameters

The implementation trains one multitask policy per domain, uses a state-based success checker in simulation, and combines specified hardware and perception components for real-world experiments.

B. Training Hyperparameters

C. Success Checker Heuristic Functions

  • Simulation verification uses ground-truth state and standard LIBERO predicates, with Pick success requiring the target object to rise at least 3 cm.Other atomic skills retain default LIBERO success conditions.Source: C. Success Checker Heuristic Functions · p. 14

A. Hardware Specifications

  • The real-world setup uses a 7-DoF Franka Panda with a 15 Hz joint impedance controller and separate wrist-mounted and fixed stereo cameras.Remote GPUs run VLA inference while local hardware handles perception and real-time control.Source: A. Hardware Specifications · p. 14

B. Perception Models

  • FoundationPose estimates 6D object pose from RGB, segmentation masks, and CAD models, while YOLOE detects and segments targets using visual prompts.The authors report visual prompting is more robust to background clutter than open-vocabulary text prompts.Source: B. Perception Models · p. 15

Related papers

Loading 2602.21531v1…