PaperlayerView exact revision on arXiv

Source-linked AI summary

ActiveVLA: Injecting Active Perception into Vision-Language-Action Models for Precise 3D Robotic Manipulation

Zhenyang Liu, Yongchong Gu, Yikai Wang, Xiangyang Xue, Yanwei Fu

arXiv:2601.08325v1cs.RO

TL;DR

Existing VLA models often rely on fixed, end-effector-centric cameras and therefore lack adaptive viewpoint and resolution selection for long-horizon and fine-grained manipulation. ActiveVLA uses a coarse-to-fine framework that localizes critical 3D regions, selects informative views, and applies 3D zoom-in. It achieves strong results across three simulation benchmarks and real-world evaluations, while assuming access to expert demonstrations with fixed end-effector-centric viewpoints.

  • Problem

    Existing VLA models commonly rely on static or wrist-mounted cameras, limiting adaptive viewpoint and resolution selection for long-horizon and fine-grained manipulation.

  • Method

    ActiveVLA combines multi-view 3D-to-2D critical-region localization with active viewpoint selection and active 3D zoom-in in a coarse-to-fine architecture.

  • Results

    ActiveVLA achieves 91.8% average success on RLBench, 78.3% on COLOSSEUM, outperforms baselines on GemBench, and generalizes to real-world configurations.

  • Takeaways & Limitations

    Active perception integrated with VLA learning supports precise 3D manipulation across diverse simulated and real-world tasks.

  • Takeaways & Limitations

    The formulation assumes expert demonstrations containing observations from one or multiple fixed, end-effector-centric viewpoints.

Abstract

from arXiv · show

Recent advances in robot manipulation have leveraged pre-trained vision-language models (VLMs) and explored integrating 3D spatial signals into these models for effective action prediction, giving rise to the promising vision-language-action (VLA) paradigm. However, most existing approaches overlook the importance of active perception: they typically rely on static, wrist-mounted cameras that provide an end-effector-centric viewpoint. As a result, these models are unable to adaptively select optimal viewpoints or resolutions during task execution, which significantly limits their performance in long-horizon tasks and fine-grained manipulation scenarios. To address these limitations, we propose ActiveVLA, a novel vision-language-action framework that empowers robots with active perception capabilities for high-precision, fine-grained manipulation. ActiveVLA adopts a coarse-to-fine paradigm, dividing the process into two stages: (1) Critical region localization. ActiveVLA projects 3D inputs onto multi-view 2D projections, identifies critical 3D regions, and supports dynamic spatial awareness. (2) Active perception optimization. Drawing on the localized critical regions, ActiveVLA uses an active view selection strategy to choose optimal viewpoints. These viewpoints aim to maximize amodal relevance and diversity while minimizing occlusions. Additionally, ActiveVLA applies a 3D zoom-in to improve resolution in key areas. Together, these steps enable finer-grained active perception for precise manipulation. Extensive experiments demonstrate that ActiveVLA achieves precise 3D manipulation and outperforms state-of-the-art baselines on three simulation benchmarks. Moreover, ActiveVLA transfers seamlessly to real-world scenarios, enabling robots to learn high-precision tasks in complex environments.

1. Introduction

ActiveVLA addresses the limited perceptual flexibility of existing VLA systems by combining 3D-aware critical-region localization with active viewpoint and resolution adjustment for precise manipulation. Experiments across simulation benchmarks and real-world settings demonstrate strong performance and generalization.

  • Existing VLA approaches often use static or wrist-mounted cameras, limiting adaptive viewpoint and resolution selection during long-horizon and fine-grained manipulation.This end-effector-centric setup constrains contextual information acquisition during task execution.
  • ActiveVLA integrates active perception through adaptive viewpoint selection and active 3D zoom-in, refining observations according to task context.The framework seeks informative, task-relevant, minimally occluded views and selectively enhances task-critical regions.
  • Its coarse-to-fine pipeline projects 3D point clouds into multi-view 2D representations, localizes salient 3D regions, and selects camera poses centered on key regions.The method aligns these projections with a pretrained VLM backbone before refining perceptual focus.
  • 91.8% average success on RLBench, 78.3% on COLOSSEUM, and consistent superiority on GemBench demonstrate strong benchmark performance.Some RLBench tasks reach 100% success, and real-world evaluations show robustness on previously unseen configurations.
  • The paper reports state-of-the-art performance across three simulation benchmarks and strong real-world generalization for long-horizon and precision-critical tasks.These evaluations support the practical relevance of active perception for fine-grained manipulation.

2. Related Work

VLA models extend pretrained vision-language models to robotic control, benefiting from large-scale data, pretrained representations, and generalization to varied commands and environments. ActiveVLA further incorporates 3D structure and active perception to address the fixed viewpoints and resolutions common in prior methods.

  • VLA models extend pretrained VLMs to robotic control and leverage large-scale pretrained representations for learning and generalization.The cited advantages include interpreting diverse language commands and adapting to novel objects and environments.
  • ActiveVLA combines large-scale VLM backbones with 3D spatial structure and active perception for action prediction.Its 3D information is merged with visual-language features in a downstream action predictor.
  • Unlike previous methods, ActiveVLA uses active camera selection to address the limited perceptual flexibility of fixed viewpoints and resolutions.Prior systems generally cannot dynamically adjust viewpoint or camera resolution during manipulation.

3. Our Proposed ActiveVLA

ActiveVLA is a coarse-to-fine 3D vision-language-action framework that localizes task-relevant regions before actively refining viewpoint and resolution for action prediction. It combines multi-view 3D reasoning, heatmap-based localization, active view selection, and 3D zoom-in to support precise manipulation.

  • VLA Setup: ActiveVLA maps observations and language instructions to 6-DoF end-effector poses, gripper states, and collision flags for robot manipulation.The action representation includes a next-key-frame pose T ∈ SE(3), gripper state, and collision flag.
  • 3D Crucial Area Perception: The coarse stage reconstructs a point cloud and renders top, front, and right orthographic views with RGB, depth, and world-coordinate channels.World coordinates establish correspondence between pixels in different views that represent the same 3D point.
  • 3D Crucial Area Perception: Heatmap prediction rearranges patch tokens into a feature grid, applies learned convex upsampling, and back-projects multi-view heatmaps to identify a crucial 3D region.The learned upsampling module recovers finer spatial detail than fixed interpolation, while cross-entropy training supervises heatmap prediction.
  • 3D Active Perception: Active viewpoint selection scores candidate cameras using visibility, viewing distance, and angular diversity, then selects the top-K views around the localized region.The combined score uses normalized criteria with weights satisfying wvis + wdis + wdiv = 1.
  • 3D Active Perception: Active 3D zoom-in re-renders the selected viewpoint with a narrowed field of view, magnifying key interaction regions while preserving pixel resolution.The virtual renderer synthesizes high-resolution close-ups from local 3D structure without geometric loss, supporting precise gripper pose prediction.
  • 3D Action Prediction: The selected and zoomed views produce attention heatmaps that are back-projected into a 3D score volume, whose maximum determines translation while hierarchical fusion predicts rotation and gripper actions.Orientation uses discretized Euler angles, and the model also predicts gripper state and a binary collision flag.

4. Experiments

ActiveVLA is evaluated on three simulation benchmarks and through real-world manipulation scenarios, including occlusion-rich and fine-grained tasks. It achieves strong benchmark performance, while ablations show complementary gains from active view selection and 3D zoom-in.

  • Simulation Benchmarks: The evaluation covers RLBench, COLOSSEUM, and GemBench, targeting long-horizon, fine-grained manipulation, robustness, compositionality, and generalization.RLBench contains 18 tasks, COLOSSEUM tests 12 perturbation types, and GemBench evaluates hierarchical generalization across training and testing tasks.
  • RLBench: 91.8% average success rate on RLBench establishes ActiveVLA as the new state of the art, with an average rank of 1.22.It remains robust under occlusions, including 65.6% success on Place Cups.
  • COLOSSEUM: 65.9% average success rate on COLOSSEUM surpasses previous methods, with strongest reported results on Table Color (78.3%) and Camera Pose (76.3%).The benchmark evaluates generalization under object, scene, and camera variations.
  • Real-World and Qualitative Evaluation: Real-world evaluations show ActiveVLA actively perceiving and precisely completing manipulation tasks despite severe occlusions and complex spatial structures.Figure 3 presents the coarse-to-fine stages: projection and heatmap localization, followed by active view selection and active 3D zoom-in.
  • GemBench: ActiveVLA achieves the best GemBench performance across core levels L1–L3, with success rates of 92.4%, 66.3%, and 45.1%.Its overall average is 51.3%, improving on the previous state of the art by 1.3 percentage points, while L4 performance is 1.2%.
  • Ablation and Hyperparameters: Adding Active View Selection raises success from 87.6% to 89.4%, and adding Active 3D Zoom-in further raises it to 91.8%.Inference time increases from 0.26 s to 0.45 s and then 0.53 s per trial; three views and a zoom-in factor of four are selected as a balance.

5. Conclusion

ActiveVLA integrates active perception into a coarse-to-fine VLA architecture for precise manipulation. Its multi-view 3D perception, active view selection, and active 3D zoom-in support spatial reasoning and fine-grained control across simulated and real-world tasks.

  • Framework: ActiveVLA combines Active View Selection and Active 3D Zoom-in within a coarse-to-fine architecture.The framework dynamically adapts observation strategies during manipulation.
  • Conclusion: Multi-view 3D perception aligned with VLMs balances global context and fine-grained local detail during manipulation.Experiments in simulated and real-world environments demonstrate superior performance across diverse tasks.

1. Video

The video presents four manipulation scenarios designed around occlusion, spatial relationships, and fine-grained geometric constraints. ActiveVLA actively selects viewpoints and zoom regions before acting, supporting precision manipulation in cluttered environments.

  • Overall Capability: The video emphasizes ActiveVLA’s ability to handle dense, irregular occlusion structures and perform precision manipulation in cluttered environments.Its active viewpoint selection and 3D zoom-in acquire higher-quality visual information than static sensing.
  • Video Overview: The four scenarios require the robot to decide where to look and which region to zoom before determining how to act.They involve complex occlusion patterns, challenging spatial relationships, and fine-grained geometric constraints.
  • Retrieving a Towel from Layered Drawers: Active viewpoint probing and local 3D refinement expose partially hidden towel regions inside layered drawers for accurate picking.The scenario demonstrates resolution of internal structural occlusion.
  • Picking and Placing Blocks: Occlusion-aware viewpoint planning reveals both the red block and green placement target, supporting extended-horizon manipulation.The task combines clutter occlusion with relational placement constraints.
  • Grasping an Occluded Banana: Viewpoint sweeps and active 3D zoom-in uncover a grasp region on a banana embedded among irregularly shaped fruits.The scene presents dense, heterogeneous, non-convex, view-dependent occlusion.
  • Retrieving an Occluded Purple Cup: ActiveVLA selects exploratory viewpoints and zooms into the handle region to retrieve a purple cup from a hanging rack without disturbing neighboring objects.The task involves relational occlusion among articulated or suspended objects and strict collision constraints.

2. Additional Quantitative Results

ActiveVLA improves manipulation success across occlusion-heavy real-world tasks, outperforming TriVLA and achieving high overall performance. The results support active perception as useful for reducing uncertainty and enabling more precise actions in partially observed environments.

  • Real-world quantitative results: ActiveVLA improves success rates over TriVLA by 24% on Retrieving a Towel, 41% on Red to Green Block, 29% on Occluded Banana, and 17% on Occluded Purple Cup.These tasks involve layered occlusion, cluttered alignment, partially obstructed objects, and fine-grained spatial reasoning.
  • Active perception effects: Active perception autonomously selects informative viewpoints, adjusts camera zoom, and focuses on regions of interest to support more precise manipulation.The paper links this proactive sensing to reduced uncertainty and disambiguation of occluded objects.
  • Real-world quantitative results: 96.3% overall success rate was achieved across the evaluated real-world manipulation tasks.
  • Implications: The results indicate that combining active perception with high-level policy reasoning is important for manipulation in cluttered and occluded environments.The authors present strategic observation adaptation as a generalizable framework for partially observable and dynamically structured scenarios.

3. Additional Implementation Details

ActiveVLA uses sequential training across simulation benchmarks and real-world data while freezing key language and vision representations. Its perturbation evaluation reports mean and standard deviation across trials to assess robustness.

  • Training stages: ActiveVLA is trained sequentially on RLBench, COLOSSEUM, GemBench, and finally real-world robot data.All stages share an identical optimization pipeline.
  • Representation stability: The SigLIP vision encoder and language token embeddings remain frozen throughout training to avoid representation drift under changing viewpoints.
  • Evaluation reporting: COLOSSEUM reports mean and standard deviation over multiple trials for tasks with diverse visual and spatial perturbations.The table evaluates performance under environmental variations and highlights the role of active perception.

4. Details of Simulation Tasks

The simulation evaluation spans structured manipulation, perturbation-based robustness, and hierarchical skill composition. These settings cover grasping, tool use, spatial interaction, occlusion, clutter, and long-horizon recombination.

  • Simulation suite: The simulation suite covers manipulation difficulty, task compositionality, and robustness through three complementary benchmark settings.The benchmarks include structured multi-step manipulation, perturbation-heavy robustness evaluation, and hierarchical skill composition.
  • COLOSSEUM: COLOSSEUM introduces 12 systematic variations in object geometry, material, scene layout, kinematics, lighting, and camera configuration.It preserves task semantics while changing environmental statistics to stress-test generalization, viewpoint invariance, and policy stability.
  • Task coverage: The evaluated tasks include object grasping, tool use, complex spatial interactions, and direct visuomotor challenges such as cloth manipulation.
  • GemBench: GemBench composes seven action primitives into 16 training tasks and 44 held-out evaluation tasks with novel combinations.The benchmark tests long-horizon reasoning, unfamiliar task graphs, skill recomposition, and cross-task transfer.
  • Occlusion-heavy tasks: Occlusion-heavy tasks involve layered drawer retrieval, partially occluded stacking, cluttered banana grasping, and narrow-grasp extraction from a hanging rack.These scenarios require active viewpoint selection, 3D zoom-in, and fine-grained 6D pose reasoning.

5. Details of Real-Robot Tasks

Real-robot evaluation covers basic handling, articulated interaction, non-prehensile control, and complex occlusion-heavy manipulation. The tasks use a Franka Panda with an eye-in-hand RGB-D sensor and two auxiliary static cameras.

  • Task categories: Real-robot tasks span basic object handling, articulated-object interaction, non-prehensile control, and complex occlusion-heavy scenarios.
  • Robot and sensing setup: The robot uses an eye-in-hand RGB-D sensor and two auxiliary static cameras to provide multi-view observations under natural occlusion.
  • Direct manipulation tasks: Direct manipulation includes wiping a table with a deformable towel, placing an orange into a basket, and placing a mouse inside an opened drawer.These tasks involve cloth deformation, stable grasp synthesis, precise placement, and articulated-object interaction.
  • Occlusion-heavy tasks: Occlusion-heavy tasks require viewpoint selection and 3D zoom-in for severe occlusion, multi-layer geometry, and fine-grained 6D pose reasoning.Retrieving a towel from layered drawers is one example of the compounded visibility and precision demands.
Loading 2601.08325v1…