PaperlayerView exact revision on arXiv

Source-linked AI summary

Look Before You Act: Bridging Active Perception and Vision-Language-Action Models via Observation-Aligned Data Adaptation

Jiawei Geng, Zhenyang Liu, Yuzheng Wu, Yongchong Gu, Yanwei Fu

arXiv:2610.23378v1cs.RO

TL;DR

Active perception can expose an occluded target, but the resulting viewpoint and robot state may differ from those represented in a VLA's training demonstrations. ActiveAlign adapts VLA training to successful continuations from active-view endpoints; across four suites, it reports higher success than standard training.

  • Problem

    Active perception can expose a target, yet a VLA must continue from an observation and robot-state handoff that differs from its conventional training demonstrations.

    Source: §I · p. 1

  • Method

    ActiveAlign connects active-view endpoints to demonstration prefixes with closed-loop alignment, replays the manipulation suffix, and retains successful continuations for VLA fine-tuning.

    Source: §I · pp. 1–2

  • Results

    ActiveAlign reaches 70.7% and 72.7% success, gains of 40.0 and 26.4 percentage points (pp) over Standard.

    Source: B. Simulation Results and Analysis · p. 5

  • Takeaways & Limitations

    Viewpoint selection and manipulation supervision should be designed together, even when the two policies are trained separately.

    Source: §VI · p. 8

    ActiveAlign data construction requires a restorable simulator or reconstructable physical workcell and replayable demonstration prefixes; successful replay can underrepresent difficult deployment endpoints.

    Source: §V · p. 7

Abstract

from arXiv · show

Robots manipulating objects in clutter must decide not only how to act, but also where to look. Although active camera motion can expose an occluded target, it may move the robot into an initial observation-state distribution that differs from the manipulation policy's training distribution. We identify this problem as first-frame distribution sensitivity: under perturbed starts in LIBERO, naively prepending camera repositioning to pi_0.5 reduces success from 46.3% to 11.5%. We introduce ActiveAlign, a data-adaptation framework that trains a Vision-Language-Action (VLA) policy from the states at which active perception terminates. A learned viewpoint module first generates a camera trajectory. Offline, ActiveAlign uses closed-loop alignment to connect the trajectory endpoint to an existing demonstration, replays the remaining manipulation trajectory, and retains successful continuations for fine-tuning. At deployment, the same viewpoint module precedes the adapted VLA without modifying its architecture. Across 40 LIBERO tasks, ActiveAlign achieves 70.7% success with pi_0 and 72.7% with pi_0.5, outperforming their standard-training baselines by 40.0% and 26.4% respectively. Matched-state and data-matched controls support gains beyond prefix shortening and generic viewpoint augmentation; OpenVLA and GR00T N1 also show the same trend. Success rate under physical occlusion in real world increases from 35/90 to 83/90. These results show that active perception benefits manipulation when policy training is aligned with the observation and robot-state distribution induced by viewpoint selection.

I. INTRODUCTION

Active camera motion can improve a view while leaving a VLA unable to use it, because the manipulation policy must continue from the viewpoint-induced observation and robot state. ActiveAlign addresses this handoff by training on successful continuations from active-view endpoints.

I. INTRODUCTION

  • Under a shared perturbed-start LIBERO protocol, camera repositioning reduces π0.5 success from 46.3% to 11.5% and π0 from 30.7% to 7.0%.A useful view and a view the policy can act from are distinct requirements.Source: §I · p. 1
  • Wrist-camera motion changes visual cues and arm configuration together, so the manipulation policy must continue from a different combination of visual and proprioceptive cues.The paper cautions that a difference in first-frame image distributions alone does not establish the cause of every failure.Source: §I · p. 1
  • Replay feasibility constrains data construction: usable supervision requires resettable simulators or reconstructable physical workcells.Deployment itself requires neither scene reset nor expert replay.Source: §I · p. 2
  • Across four suites, ActiveAlign improves π0 and π0.5 by 40.0 and 26.4 percentage points, with controls supporting gains beyond shorter suffixes and generic viewpoint coverage.The evaluation also includes matched-state and data-matched controls.Source: §I · p. 2
  • ActiveAlign connects active-view endpoints to existing demonstrations through prefix matching, closed-loop alignment, and replay filtering, retaining successful continuations for VLA training.The learned viewpoint trajectory is executed in a restored environment before the recorded manipulation suffix is replayed.Source: §I · pp. 1–2

II. RELATED WORK

Prior work studies VLA policies, active perception, and demonstration adaptation through different mechanisms. ActiveAlign focuses on physically moving an existing wrist camera and adapting demonstrations to the resulting handoff states.

II. RELATED WORK

  • VLA action formulations vary, but each policy depends on the observation–state distribution represented in its training demonstrations.ActiveAlign preserves the policy architecture while adapting its fine-tuning distribution to states induced by a deployment-time viewpoint module.Source: §II · p. 2
  • Unlike multi-view methods that aggregate or render views, ActiveAlign physically repositions a wrist camera, changing both the visual input and the robot configuration for manipulation.Its focus is compatibility between the selected viewpoint and the downstream policy's manipulation distribution.Source: §II · p. 3
  • ActiveAlign adapts demonstrations to endpoints from a separate viewpoint policy, using closed-loop alignment and successful replay because geometric endpoint matching alone cannot validate a continuation.This differs from paired sensing-and-manipulation systems requiring specialized active-vision embodiments or continuous gaze control.Source: §II · p. 3

III. METHOD

The formulation treats active perception as a change to the initial observation-and-robot-state distribution from which a VLA must manipulate. ActiveAlign therefore constructs adapted demonstrations that provide executable supervision from states induced by the viewpoint module.

A. Problem Formulation

  • A wrist-camera viewpoint can change both visual input and robot state, so its induced initial distribution may differ from that of demonstration starts.An image-marginal mismatch alone does not prove that inputs lie outside full training support or establish the cause of policy failure.Source: A. Problem Formulation · p. 3
  • The objective is to improve manipulation from the states induced by active perception by constructing adapted demonstrations and fine-tuning the VLA.The backbone’s supervised-learning objective is used for fine-tuning.Source: A. Problem Formulation · p. 3
  • The fine-tuning loss predicts an action or action chunk, while task success is evaluated through rollouts rather than optimized directly by that loss.Fine-tuning data depend on the condition; data-matched controls use the same mixing ratio.Source: A. Problem Formulation · p. 3
  • Useful adapted supervision must pair each active-view observation with an executable action target from its corresponding robot and scene state.Prefix matching finds a nearby continuation, alignment connects to it, and replay checks whether the episode remains successful.Source: A. Problem Formulation · p. 3

B. Active Perception Module

The active-perception module uses scene RGBD and language to predict a wrist-camera trajectory. Its output is a candidate sensing motion, with endpoints retained for adaptation when alignment and successful replay are feasible.

B. Active Perception Module

  • The module combines RGB, depth, and language features to predict a sequence of camera poses conditioned on the task and scene.Separate CLIP branches encode RGB and depth, and a conditional autoregressive decoder generates quantized pose tokens.Source: B. Active Perception Module · p. 3
  • The trajectory module maps a scene RGBD observation and instruction to camera poses; the default sequence contains 30 poses and can be interpolated to 120 for smooth execution.The poses are represented in SE(3) and transformed into the calibrated robot/world frame.Source: B. Active Perception Module · p. 3
  • Predicted paths are candidate sensing motions, not paths certified as optimal or collision-free.Simulation uses IK repositioning, whereas the physical robot moves continuously through its controller.Source: B. Active Perception Module · p. 3, Figure 2 · p. 4
  • Only endpoints permitting alignment and successful manipulation replay are retained for adaptation.The pipeline uses prefix matching, alignment, and successful replay to produce adapted demonstrations.Source: B. Active Perception Module · p. 4, Figure 2 · p. 4

C. Observation-Aligned Data Adaptation

Observation-aligned adaptation connects active-view endpoints to recorded manipulation continuations and retains only successful replays. This feasibility filtering determines which candidate views provide usable VLA supervision.

C. Observation-Aligned Data Adaptation

  • The adaptation procedure restores each demonstration’s starting scene and robot state, executes a candidate camera trajectory, matches an endpoint within a demonstration prefix, then aligns and replays the suffix.Only successful attempts contribute the alignment segment and suffix to the adapted data; the active approach itself is not a VLA training target.Source: C. Observation-Aligned Data Adaptation · p. 4
  • Successful replay retained 1,736 of 2,000 attempted simulation trajectories (86.8%) and 86 of 100 physical attempts (86.0%).Source: C. Observation-Aligned Data Adaptation · p. 4
  • Replay filtering and retries can make the adapted-data start distribution differ from the active-perception endpoint distribution, without guaranteeing distribution equality or optimal task success.Source: C. Observation-Aligned Data Adaptation · p. 4

D. Deployment and Applicability

Deployment runs viewpoint selection once before object contact, then the adapted VLA manipulates using fresh observations. Offline adaptation has reset or workcell-reconstruction requirements that deployment does not.

D. Deployment and Applicability

  • The viewpoint module is queried once before contact, after which the VLA uses fresh images and state inputs to predict actions.For chunked π0 and π0.5, the policy predicts H = 10 actions and executes the first R = 5 before replanning.Source: D. Deployment and Applicability · p. 5
  • Offline adaptation requires simulation reset or a reconstructable physical workcell, whereas deployment requires neither reset nor demonstration replay.The wrist camera continues moving with the arm during manipulation, but there is no further viewpoint-selection phase.Source: D. Deployment and Applicability · p. 5

IV. EXPERIMENTS

The experiments evaluate four LIBERO suites under perturbed starts, with additional tests for initial occlusion, backbone transfer, and physical manipulation.

A. Experimental Setup

  • LIBERO provides four suites of 10 tasks, with 50 original demonstrations per task; LIBERO-Occ uses the same suites with targets occluded in the default initial view.RLBench is used only to visualize zero-shot perception-module transfer.Source: A. Experimental Setup · p. 5
  • Each LIBERO result uses 50 rollouts per task, with suite-specific horizons and initial end-effector pose perturbations of standard deviation 0.1.The perturbed-start protocol is fixed across method conditions and differs from benchmark-prescribed initial states.Source: A. Experimental Setup · p. 5
  • The evaluation compares Standard, Naive Active, and ActiveAlign, and includes data-matched controls; physical tests use two arm settings and 100 original demonstrations.LIBERO uses the environment completion checker, while physical results report binary counts and Wilson 95% confidence intervals.Source: A. Experimental Setup · p. 5

B. Simulation Results and Analysis

In perturbed-start LIBERO, ActiveAlign improves success across suites and tested VLA backbones, while controls and occlusion tests probe when the gains arise.

Perturbed-start results

Backbone transfer

Beyond recovering naive-composition losses

  • On Object, ActiveAlign exceeds Standard by 46.0 pp for π0 and 39.0 pp for π0.5, while Naive Active reaches 0.0% and 2.2%, respectively.The two adapted backbones finish within 2.0 pp of each other on average.Source: B. Simulation Results and Analysis · p. 5

Start-state shift

Viewpoint and data controls

  • On LIBERO Spatial, ActiveAlign scores 78.2%, 20.2 and 16.8 percentage points above random-view adaptation (58.0%) and broad viewpoint-jitter adaptation (61.4%), respectively, under equal data and optimization budgets.These results support deployment-matched viewpoint adaptation beyond additional data, shorter continuations, and untargeted viewpoint diversity.Source: B. Simulation Results and Analysis · p. 6

Initial occlusion

  • On initially occluded LIBERO-Occ tasks, ActiveAlign exceeds Standard by 28.6, 24.6, 22.4, and 34.6 pp on Spatial, Object, Goal, and Long.Naive Active remains below Standard; the Long result does not test occluders that move during execution.Source: Table V · p. 6, B. Simulation Results and Analysis · p. 6

Perception transfer

Sensitivity analysis

  • Panel (c) shows consistent performance gains as trajectory budget grows from 5 to 30, indicating that additional viewpoint-trajectory capacity benefits performance.Source: B. Simulation Results and Analysis · p. 6

C. Physical Evaluation

Physical evaluations show gains under occlusion and across diverse tasks, with the longer manipulation task remaining more challenging. Data collection uses reset and replay offline, separately from deployment evaluation.

Physical Evaluation

  • Offline collection retained 86 of 100 attempts after fixture-based resets, endpoint matching, closed-loop correction, and suffix replay.Operators could abort near collision or force limits but could not correct motions or finish tasks; reset and replay were used only for offline data construction.Source: C. Physical Evaluation · p. 6
  • ActiveAlign succeeded in 17/20 cup-grasping, 15/20 container-retrieval, and 13/20 observe–open–grasp–place trials.These task-diversity results are separate from the occlusion aggregate; the 20-trial intervals are wide, and the longer task had lower success.Source: Table VII · p. 7, C. Physical Evaluation · p. 7
  • 83/90 trials (92.2%) under physical occlusion succeeded with ActiveAlign, versus 35/90 (38.9%) for Standard and 17/90 (18.9%) for Naive Active.Relative to Standard, ActiveAlign gained 43.3, 56.7, and 60.0 percentage points under partial, medium, and severe occlusion.Source: Table VIII · p. 7, C. Physical Evaluation · p. 7

V. LIMITATIONS

ActiveAlign’s offline construction depends on replayable demonstrations and a resettable or reconstructable environment. Its current deployment uses one pre-manipulation viewpoint-selection phase.

V. LIMITATIONS

  • Aligned continuations require a restorable simulator or reconstructable workcell and demonstration prefixes replayable from generated endpoints.Training retains only successful alignment-and-replay trajectories, so difficult endpoints may be underrepresented relative to deployment.Source: §V · p. 7
  • The system selects a viewpoint once before manipulation and does not reselect views in response to moving occluders.RLBench evidence is perception-only; end-to-end manipulation is evaluated in LIBERO and on the physical robot.Source: §V · p. 7
  • Extending the construction to non-resettable environments, continuous viewpoint selection, and broader physical deployments remains future work.Source: §V · p. 7

VI. CONCLUSION

ActiveAlign connects viewpoint selection with manipulation by constructing successful continuations from active-view endpoints. Results and controls support gains beyond prefix shortening and generic viewpoint coverage.

VI. CONCLUSION

  • ActiveAlign connects the decision of where to observe with the behavior needed to act from that observation.Source: §VI · p. 8
  • LIBERO and physical results show consistent gains, while matched-state and data-matched controls support benefits beyond prefix shortening and generic viewpoint coverage.Source: §VI · p. 8
  • The paper concludes that viewpoint selection and manipulation supervision should be designed together, even when the policies are trained separately.Extending the construction beyond replayable prefixes and interleaving observation selection with manipulation are identified as next steps.Source: §VI · p. 8
Loading 2610.23378v1…