PaperlayerView exact revision on arXiv

Source-linked AI summary

ActiveScale: Scaling Active Perception for Robots across Model, Data, and Hardware

Shuai Zhou, Kaisheng Pang, Wenxuan Song, Wenjie Zhang, Xinhu Zheng, Haoang Li

arXiv:2609.18514v2cs.ROcs.LG

TL;DR

Fixed viewpoints can leave task-relevant information occluded, and existing VLAs lack camera-pose inference and subsequent camera-motion prediction. ActiveScale combines pose-grounded temporal modeling, human–robot mid-training, and AMP; across five real-world task families, it improves mean task success from 30.0% to 70.0% over directly post-trained π0.5.

  • Problem

    Fixed-view observations can be incomplete, while existing VLAs lack camera-pose inference and subsequent camera-motion prediction needed for active perception.

    Source: §I · pp. 1–2

  • Method

    ActiveScale augments a VLA with historical frames and pose-grounded camera tokens, mid-trains on egocentric human and robot data, and introduces AMP for coordinated viewpoint, manipulation, and mobile-base control.

    Source: §I · p. 2

  • Results

    Across five real-world active-perception task families, mean task success improves from 30.0% to 70.0% over directly post-trained π0.5.

    Source: §V · p. 7

  • Takeaways & Limitations

    ActiveScale provides an integrated framework for active-perception robot learning across model, data, and hardware.

    Source: §V · p. 7

    The authors leave scaling AMP data collection and systematic evaluation on mobile active-perception tasks to future work.

    Source: §V · p. 7

Abstract

from arXiv · show

Active perception is essential for robotic manipulation when fixed viewpoints leave task-relevant information occluded or unobserved. However, enabling vision-language-action (VLA) models to reason across changing viewpoints and actively acquire informative observations remains challenging. We present ActiveScale, a framework that advances active perception through coordinated model, data, and hardware designs. Our model augments a VLA with historical video observations and explicit camera-pose supervision, using per-frame pose tokens and a lightweight prediction head to associate observations across viewpoints and support a coherent understanding of the scene. To learn from the camera motion naturally present in human activity, we introduce a scalable human--robot mid-training recipe using 1000 hours of egocentric and robotic data, adapting the model to temporal inputs and pose supervision. We further introduce Active-perception Mobile-manipulation Platform (AMP), a robotic platform that supports active perception and mobile manipulation through single-operator teleoperation, enabling scalable collection of demonstrations that coordinate viewpoint changes and manipulation. Experiments demonstrate improved success rates on active-perception tasks, while ablation studies validate the contributions of camera-pose-aware modeling and egocentric mid-training. Together, these components provide an integrated foundation for studying and developing active perception in robotic manipulation.

I. INTRODUCTION

ActiveScale targets gaps in VLA camera-pose reasoning, use of egocentric camera motion, and mobile active-perception research. It combines pose-grounded temporal modeling, human–robot mid-training, and AMP; experiments report state-of-the-art performance across five real-world active-perception tasks.

I. INTRODUCTION

  • Existing VLAs lack camera-pose inference and subsequent camera-motion prediction, while prior approaches underuse egocentric motion and largely study static environments.The introduction also identifies a lack of unified infrastructure for studying active perception across model, data, and hardware.Source: §I · p. 2
  • The model uses historical frames and per-frame camera tokens with an auxiliary head predicting translation, rotation, and field of view to associate observations across viewpoints.This supports perception of a persistent scene from consecutive video frames.Source: §I · p. 2
  • ActiveScale achieves state-of-the-art performance across five real-world active-perception tasks.Ablations also validate the model design and mid-training procedure.Source: §I · p. 2
  • The human–robot mid-training recipe scales active-perception training to 1000 hours of large-scale egocentric human and robot data, more than 5× larger than prior efforts.It provides a scalable path for transferring human manipulation and active-view priors to robot policies.Source: §I · p. 2
  • AMP integrates bimanual manipulation, an independently actuated camera, and mobile-base control through a unified interface that lets one operator coordinate them.The platform is intended to support scalable collection of coordinated mobile-manipulation demonstrations.Source: §I · p. 2

A. Active Perception in Robot Manipulation

Prior work couples viewpoint control with manipulation, uses egocentric demonstrations, and adds geometric inputs or supervision to VLA models. ActiveScale instead grounds temporal camera motion in its VLA representation and formulates manipulation as joint control of actions and viewpoint motion.

A. Active Perception in Robot Manipulation

  • Learning-based methods increasingly couple viewpoint control with manipulation, and ActiveScale follows this direction by grounding temporal camera motion in the VLA representation.Prior approaches include learned task-driven viewpoint control, coordinated human demonstrations, and VLAs that model camera control within visuomotor policies.Source: A. Active Perception in Robot… · p. 2

B. Learning from Egocentric Human Videos

  • Egocentric head-camera trajectories encode task-driven visual intent, including repositioning viewpoints to search for objects, resolve occlusions, and support manipulation.ActiveScale uses egocentric human data to transfer manipulation priors and task-driven viewpoint behaviors to robots.Source: B. Learning from Egocentric Human Videos · p. 2

C. Spatial Intelligence for VLA

  • Prior VLA spatial-understanding methods use point clouds, stereo, depth-derived information, geometric distillation, or depth reconstruction from RGB.The passage contrasts these approaches with active perception's concern for camera motion in a persistent scene rather than 3D geometry reconstruction.Source: C. Spatial Intelligence for VLA · p. 2, Figure 2 · p. 3

III. METHODOLOGY

  • The methodology proceeds from problem formulation to AMP and teleoperation, then to model architecture and training recipe.An overview is shown in Fig. 1.Source: §III · p. 3

A. Problem Formulation

  • Active-perception manipulation is formulated as a language-conditioned policy that predicts action chunks jointly specifying both end-effector poses, active-camera pose, and gripper commands.The observation includes visual inputs and non-visual robot state context, and actions are expressed in a shared robot-centric frame.Source: A. Problem Formulation · p. 3

B. AMP Setup

AMP combines a mobile base, two manipulation arms, and a third arm carrying an independently actuated camera. Its Meta Quest 2 interface maps one operator’s headset and controller motions to camera, arm, gripper, and base control.

B. AMP Setup

  • AMP adds a third 6-DoF arm carrying a camera to a mobile base with two 6-DoF Piper manipulation arms, preserving bimanual manipulation and mobility.Each manipulation arm also carries a camera near its end effector.Source: B. AMP Setup · p. 3

1) Teleoperation System:

  • The headset controls the active-camera arm, while the two controllers operate the corresponding manipulation arms; button inputs control grippers and mobile-base motion.Reengaging teleoperation re-anchors the references, allowing headset and controller repositioning without discontinuities.Source: 1) Teleoperation System · p. 3

2) Data Collection:

The data collection records synchronized robot demonstrations and represents camera and manipulator motion in a robot-centric frame. These data support policies that reason across viewpoints while jointly producing manipulation and camera actions.

2) Data Collection:

  • Demonstrations are recorded synchronously at 30 Hz, with camera and robot information represented relative to a frame that moves with the mobile platform.The frame origin is shifted 0.25 m downward along the middle-arm z axis.Source: 2) Data Collection · p. 3
  • The 23-D robot-centric state combines the poses of both end effectors and the active camera with the two gripper positions.The pose representation uses xyzw-ordered unit quaternions.Source: 2) Data Collection · p. 3
  • The collected action targets include realized Cartesian trajectories and gripper commands, while mobile-base motion is recorded as forward linear and yaw angular velocities.Source: 2) Data Collection · p. 3
  • The policy treats camera motion as part of the action space alongside manipulation, supporting action generation across changing viewpoints.Source: 2) Data Collection · p. 4, Figure 3 · p. 4

1) π0.5 Backbone:

The π0.5 backbone combines a pretrained vision-language model with an action expert. It uses flow matching for continuous action chunks alongside autoregressive subtask and action-token objectives.

1) π0.5 Backbone:

  • π0.5 combines a pretrained PaliGemma vision-language model with a lightweight transformer-based action expert.The VLM encodes visual observations, language instructions, and tokenized proprioceptive states as action-generation context.Source: 1) π0.5 Backbone · p. 4
  • The action expert uses flow matching to predict continuous action chunks conditioned on VLM context.Training interpolates clean action chunks with Gaussian noise and supervises the expert to predict the target velocity.Source: 1) π0.5 Backbone · p. 4
  • The VLM predicts subtasks autoregressively, while FAST action tokens are trained with next-token cross-entropy and also condition the continuous action expert.Source: 1) π0.5 Backbone · p. 4

2) Camera Tokens:

Camera tokens associate historical front-camera observations with pose and field-of-view supervision, allowing the model to relate views across time. Training first combines human and robot demonstrations, then adapts the policy to the target embodiment and tasks.

2) Camera Tokens:

  • The input includes the current front-camera view and three earlier observations sampled 16 frames apart, each associated with a learnable camera token.Source: 2) Camera Tokens · p. 4
  • Blockcausal attention lets each history block attend to itself and preceding history, while wrist, language, and action tokens can attend to all valid camera-history blocks.Source: 2) Camera Tokens · p. 4
  • A camera head predicts each token’s camera translation, quaternion orientation, and fields of view; its auxiliary geometric supervision grounds the representation and the head is removed at inference.The resulting prefix conditions the action expert, which generates manipulation and head-camera motions.Source: 2) Camera Tokens · p. 4

D. Training Recipe

  • Training starts with human–robot mid-training for transferable manipulation and active-view priors, then uses task-specific post-training on teleoperation data for the target embodiment and tasks.Source: D. Training Recipe · p. 4

1) Human-Robot Co-training:

Human–robot co-training uses egocentric human demonstrations for transferable semantic, manipulation, and active-view priors while retaining robot-aligned action learning. The training objectives apply only when their corresponding supervision is available.

1) Human-Robot Co-training:

  • Human and robot demonstrations are sampled at a balanced 1:1 ratio, with human head-camera poses supervising active views and wrist trajectories supervising manipulation.Human trajectories are expressed relative to the earliest valid observation in each temporal window, while robot trajectories remain in fixed robot reference frames.Source: 1) Human-Robot Co-training · p. 4
  • States and action trajectories are normalized separately by data source, and unavailable action dimensions and camera-pose targets are masked.Source: 1) Human-Robot Co-training · p. 5
  • The human data come from curated EgoLive and EgoVerse indoor household-manipulation clips with valid camera trajectories and task/action supervision; robot data combine demonstrations from AgiBot World, RoboCOIN, and AMP.Source: 1) Human-Robot Co-training · p. 5

FAST + LR

  • High-level human samples supervise subtask prediction, while low-level human and robot samples supervise FAST and flow objectives; the combined training objective includes camera-pose loss when supervision is available.The stated weights are λH = λC = 0.2.Source: FAST + LR · p. 5

2) Post-training with Teleoperation Data:

The experiments evaluate ActiveScale after teleoperation-data post-training across active-perception performance, human–robot transfer, model design, and execution efficiency. They also demonstrate AMP on a single-operator mobile-manipulation task.

2) Post-training with Teleoperation Data:

  • Post-training specializes the co-trained policy to AMP and downstream tasks using AMP teleoperation demonstrations, with the architecture and action representation unchanged.The post-training objective combines flow and camera losses; the camera-loss weight is 0.2.Source: 2) Post-training with Teleoperation Data · p. 5

IV. EXPERIMENTS

  • The evaluation asks whether the policy handles active viewpoint control, whether mid-training helps, how history and camera tokens contribute, and whether inference meets real-time requirements.Models are trained on H100 GPUs and deployed on AMP.Source: §IV · p. 5
  • For reproducible comparison, ActiveScale is evaluated against π0.5 because recent active-perception VLAs lack publicly released checkpoints and complete training recipes for direct evaluation.The experiments also analyze execution efficiency and demonstrate a single-operator task combining active viewpoint control, bimanual manipulation, and base motion.Source: §IV · p. 5

A. Task Suite and Evaluation Protocol

The task suite tests active perception under occlusion and across larger search spaces, then compares ActiveScale with directly post-trained π0.5. ActiveScale improves mean success and task progress across all five task families.

A. Task Suite and Evaluation Protocol

  • Each task family uses 150 post-training demonstrations and 20 matched real-world rollouts; success requires all stages, while TP credits completed stages.The task-progress table specifies stage credits and marks inapplicable stages with dashes.Source: A. Task Suite and Evaluation Protocol · p. 5, Table I · p. 6
  • The five task families divide into occlusion-driven Bag, Drawer, and Table tasks and Pot and Box tasks requiring active search and localization.The suite covers initially hidden or out-of-view information as well as search across workspace regions.Source: A. Task Suite and Evaluation Protocol · p. 5, Figure 4 · p. 6

B. Main Experiments (Q1)

  • Mean SR rose from 30.0% to 70.0% and mean TP from 41.6% to 78.4% for ActiveScale versus directly post-trained π0.5 across five task families.ActiveScale has higher SR and TP in every family; the comparison uses the same downstream post-training and evaluation protocol.Source: B. Main Experiments (Q1) · pp. 5–6, Figure 5 · p. 6
  • The gains are most pronounced when targets begin hidden or outside the default view, including backpack inspection and below-table target localization.On Bag, ActiveScale inspects the backpack while opening it; on Table, it redirects its viewpoint to locate the target.Source: B. Main Experiments (Q1) · p. 6

C. Ablation Studies

The ablation studies isolate the effects of human–robot mid-training and model design while holding task-specific post-training data and evaluation protocol constant.

C. Ablation Studies

  • The ablations isolate training-recipe and model-design contributions, corresponding to questions about human–robot transfer and camera-aware modeling.The variants use the same task-specific post-training data and real-world evaluation protocol unless otherwise specified.Source: C. Ablation Studies · p. 6

1) Human–Robot Mid-Training (Q2):

  • The mid-training comparison uses variants with the same architecture, task-specific demonstrations, and post-training protocol; the no-mid-training variant starts from pretrained π0.5.The supplied passage identifies the comparison setup but does not report its outcome.Source: 1) Human–Robot Mid-Training (Q2) · p. 6
  • Figure 6 presents an ablation study of the effect of mid-training.The caption identifies the ablation’s focus without giving comparative results.Source: Figure 7

2) Temporal History and Camera Tokens (Q3):

ActiveScale’s ablations find that pose-grounded camera tokens make temporal history more consistently useful across tasks. The paper also reports inference throughput, demonstrates AMP teleoperation, and summarizes its real-world results.

2) Temporal History and Camera Tokens (Q3):

  • History alone has task-dependent effects, while pose-grounded camera tokens improve both SR and TP across all five task families relative to the history-only variant.The tokens recover performance drops on Bag and Pot, supporting explicit grounding of observations acquired from different viewpoints.Source: 2) Temporal History and Camera Tokens… · p. 7

D. Execution Efficiency (Q4)

  • With RTC enabled, ActiveScale generates 50-action chunks at 264.5 Hz on one RTX 4090; asynchronous execution overlaps inference with ongoing actions.The paper uses action-generation throughput to quantify inference efficiency.Source: D. Execution Efficiency (Q4) · p. 7

E. AMP Teleoperation Demonstration

  • A single operator teleoperates AMP through viewpoint changes, drawer retrieval, mobile-base navigation, and placing a vegetable in a pot before covering it.The demonstration combines active viewpoint control, bimanual manipulation, and mobile-base motion in one interface.Source: E. AMP Teleoperation Demonstration · p. 7

V. CONCLUSIONS

  • Across five real-world active-perception task families, ActiveScale improves mean task success from 30.0% to 70.0% over directly post-trained π0.5.The conclusion also states that ablations validate human–robot mid-training and pose-grounded temporal modeling.Source: §V · p. 7
Loading 2609.18514v2…