Source-linked AI summary
SaPaVe: Towards Active Perception and Manipulation in Vision-Language-Action Models for Robotics
Mengzhen Liu, Enshen Zhou, Cheng Chi, Yi Han, Shanyu Rong, Liming Chen, Pengwei Wang, Zhongyuan Wang, Shanghang Zhang
TL;DR
Robots need to perceive complex scenes actively and act from changing viewpoints, but existing methods struggle to combine semantic perception with robust execution. SaPaVe decouples camera and manipulation actions and learns them in stages, supported by a semantic camera-movement dataset and geometry-aware execution. It reports improvements over recent models in simulation and real-world tasks.
Problem
Method
SaPaVe first learns semantic camera movement from large-scale data, then trains manipulation with hybrid data using decoupled action spaces and spatial knowledge injection.
Source: §1 · p. 2, §3.3 · p. 5Results
SaPaVe surpasses Gemini 2.5 Pro by 16% on ActiveViewPose-200K, fixed-view GR00T-N1 by an absolute 58% success rate in simulation, and π0 and GR00T-N1 by 40% and 31.25% in real-world tasks.
Source: §1 · p. 2Takeaways & Limitations
The authors conclude that SaPaVe combines semantic active perception with active-view execution and has potential for human-like robots.
Source: §5 · p. 8With a fixed robot base, manipulation is confined to local exploration: objects beyond the arm’s maximum reach cannot be acted upon.
Source: G. Limitations and Future Work · pp. 22–23
Abstract
from arXiv · showhide
Active perception and manipulation are crucial for robots to interact with complex scenes. Existing methods struggle to unify semantic-driven active perception with robust, viewpoint-invariant execution. We propose SaPaVe, an end-to-end framework that jointly learns these capabilities in a data-efficient manner. Our approach decouples camera and manipulation actions rather than placing them in a shared action space, and follows a bottom-up training strategy: we first train semantic camera control on a large-scale dataset, then jointly optimize both action types using hybrid data. To support this framework, we introduce ActiveViewPose-200K, a dataset of 200k image-language-camera movement pairs for semantic camera movement learning, and a 3D geometry-aware module that improves execution robustness under dynamic viewpoints. We also present ActiveManip-Bench, the first benchmark for evaluating active manipulation beyond fixed-view settings. Extensive experiments in both simulation and real-world environments show that SaPaVe outperforms recent vision-language-action models such as GR00T N1 and \(\pi_0\), achieving up to 31.25\% higher success rates in real-world tasks. These results show that tightly coupled perception and execution, when trained with decoupled yet coordinated strategies, enable efficient and generalizable active manipulation. Project page: https://lmzpai.github.io/SaPaVe
1. Introduction
SaPaVe addresses active manipulation by coordinating semantic viewpoint selection with execution from dynamic views. It uses staged training and decoupled action spaces, supported by a camera-movement dataset and a geometry-aware module.
1. Introduction
- Active manipulation requires both semantic active perception, which adjusts viewpoints to acquire useful information, and active-view execution, which grounds perceptual cues into actions.The two capabilities must inform one another in real time, including when execution begins from a suboptimal viewpoint.Source: §1 · p. 2
- Discrete candidate viewpoints limit fine-grained camera control, while fixed-view VLA training leaves models sensitive to viewpoint shifts; collecting joint camera-and-action data is expensive.The paper also notes that fine-tuning in a unified action space can cause conflicts and suboptimal performance.Source: §1 · p. 2
- SaPaVe decouples camera and manipulation actions, first learning camera movement from large-scale data and then jointly optimizing both action types with manipulation data.The authors state that decoupling minimizes cross-interference and lets each action type benefit from specific data.Source: §1 · p. 2
- ActiveViewPose-200K provides 200k image-language and camera-movement pairs; a camera adapter retains semantic camera knowledge, and spatial knowledge injection supports execution under dynamic viewpoints.The second training stage uses diverse hybrid data with the pretrained camera adapter.Source: §1 · p. 2
- SaPaVe surpasses Gemini 2.5 Pro by 16% on ActiveViewPose-200K, fixed-view GR00T-N1 by an absolute 58% success rate in simulation, and π0 and GR00T-N1 by 40% and 31.25% in real-world tasks.The framework is evaluated in simulation and real-world settings.Source: §1 · p. 2
2. Related Work
Prior work addresses active perception through next-best-view selection, VQA over discrete camera motions, and end-to-end VLA policies, but each leaves gaps in semantic flexibility, data efficiency, or evaluation. SaPaVe’s work is positioned against these limitations and the lack of simulation benchmarks for active manipulation.
Active Perception and Manipulation
- Next-best-view methods often lack semantic input and end-to-end pipelines, while VQA-based viewpoint selection cannot operate in continuous camera-motion space.The paper describes these limits as restricting adaptation to semantically rich tasks and camera flexibility.Source: §2 · pp. 2–3
VLA for Robotics
- VLA models are commonly trained on fixed views, while adding continuous camera control raises data demands and 3D geometric awareness remains limited under viewpoint changes.The paper identifies scarce active-perception demonstrations and sensitivity to camera perturbations as related challenges.Source: §2 · p. 3
Benchmarks for Active Perception and Manipulation
- Real-world evaluations are difficult to reproduce, and most simulation benchmarks use fixed viewpoints; the paper introduces the first simulation benchmark specifically for active manipulation.The benchmark addresses the stated absence of established simulation evaluation for active perception and manipulation.Source: §2 · p. 3
3. Method
SaPaVe separates camera control from manipulation and uses a bottom-up strategy to learn semantic active perception before geometry-informed active-view execution.
3.1. Problem Formulation
- The policy maps RGB observations, optional 3D geometry, and language instructions to chunked camera and manipulation action trajectories.Camera actions represent relative head pitch and yaw; manipulation actions specify joint-position changes.Source: §3.1 · p. 3
3.2. Architecture
- Separate camera and manipulation decoders preserve semantic camera priors while spatial tokens guide action decoding under changing viewpoints.The Camera Adapter uses LoRA, while the Universal Spatial Encoder supports varied 3D inputs without retraining or architectural changes.Source: §3.2 · p. 4, Figure 2 · p. 4
3.3. Two-Stage Training Strategy
- Stage 1 trains camera movement on ActiveViewPose-200K; Stage 2 freezes the Camera Adapter and trains the action head on mixed data.The first stage minimizes mean squared error between predicted and ground-truth camera movement.Source: §3.3 · p. 5
3.4. ActiveViewPose-200K and ActiveManip-Bench
- ActiveViewPose-200K provides 200k image-language and camera-movement pairs, while ActiveManip-Bench evaluates active manipulation across 12 tasks, 100 objects, and 20 scenes.The benchmark addresses fixed-view evaluation and is designed to accommodate additional scenes and objects.Source: §3.4 · pp. 5–6, Figure 4 · p. 6
4. Experiments
The experiments evaluate semantic camera control, active manipulation across camera configurations, real-world VLA comparisons, generalization, and component ablations. Across these evaluations, SaPaVe benefits from specialized camera-perception training and active viewpoints, while its components address distinct weaknesses in active manipulation.
4.1. Experimental Setup
- The evaluation covers camera-pose prediction on ActiveViewPose-200K and active-manipulation success rates across simulated camera configurations and real-world tasks.ActiveManip-Bench includes six task types; subsequent experiments use a teleoperated real-robot dataset spanning occluded and out-of-view pick-and-place and articulated manipulation.Source: §4.1 · pp. 6–7, Table 2 · p. 7
4.4. Comparison with existing VLA models
- SaPaVe achieves the best reported real-world active-manipulation performance, while GR00T-N1 and π0 underperform it in nearly all tasks.The authors attribute the gap to insufficient active-perception priors and the lack of dedicated active-view manipulation modules in existing VLAs.Source: Table 3 · p. 7, §4.4 · pp. 7–8
4.2. Semantic Active Perception Evaluation
- An average score 11.6% higher than Gemini-2.5-Pro [12] shows SaPaVe’s advantage on ActiveViewPose-200K, especially on the instruction-inference-heavy Test2 split.The authors argue that generalist VLMs struggle to translate abstract instructions into precise camera poses, whereas specialized training supports a more accurate policy.Source: §4.2 · p. 7
4.3. Fixed and Dynamic Cameras Evaluation.
- Fixed viewpoints substantially reduce success across task types, with Out-of-View tasks dropping by more than 60%; adding a wrist camera still falls short of an active head camera.An additional wrist camera paired with an active head camera yields minimal gains and can degrade some metrics when dynamic perception and 3D understanding are already robust.Source: §4.3 · p. 7
4.5. Generalization Ability Evaluation.
- SaPaVe generalizes across unseen objects, varying lighting, and diverse scenarios in real-world active-manipulation evaluations.The authors associate this generalization with high-level semantic understanding and 3D information that supports spatial perception.Source: §4.5 · p. 8, Table 4 · p. 8
4.6. Ablation Studies
- Both training stages contribute to performance: omitting Stage 1 halves success on artifact-manipulation out-of-view tasks, while omitting Stage 2 lowers overall success.The authors interpret Stage 1 as providing semantic active-perception priors and Stage 2 as active-manipulation finetuning.Source: §4.6 · p. 8
- A unified action decoder weakens active manipulation, while camera-adapter training outperforms full finetuning across all four real-world experiments.The authors say decoupling avoids disrupting semantic camera priors; the adapter learns camera rotation while preserving high-level semantic information.Source: §4.6 · p. 8
Universal Spatial Knowledge Injection greatly enhances the model’s robustness for basic operations under active
- Omitting Universal Spatial Knowledge Injection causes a 15% performance drop even on relatively simple Occluded Pick-and-Place tasks.The authors describe consistent 3D spatial understanding as critical for active manipulation and say prior 3D knowledge can facilitate implicit 3D learning.Source: Universal Spatial Knowledge Injection… · p. 8
5. Conclusion
SaPaVe combines semantic active perception and active-view execution through decoupled action learning and a camera adapter. The paper also introduces a camera-learning dataset and an active-manipulation benchmark.
5. Conclusion
- SaPaVe learns semantic camera control as prior knowledge in a decoupled action space, then uses Universal Spatial Knowledge Injection to handle geometric inputs during active-view execution.The paper introduces ActiveViewPose-200K for semantic camera learning and ActiveManip-Bench for evaluating active manipulation.Source: §5 · p. 8
Supplementary Material
The supplement discusses real-world evaluation and wrist-camera limitations, then details the construction of ActiveViewPose-200K and ActiveManip-Bench. It describes their data-generation process, task formulations, and simulation setup.
A. Discussion
- For real-world tasks with camera shifts exceeding 90°, only SaPaVe succeeds, with a 45% success rate.The supplement also reports that paired head–wrist data is limited to 20k samples and that this scarcity constrains learning to integrate both views.Source: A. Discussion · p. 12
- Wrist cameras help when the head view is fixed but can add occlusion noise in active settings; only 20k paired head–wrist samples are available for Stage 2.ActiveViewPose-200K contains head-camera movements only, and the authors identify the scarcity of paired data as limiting effective integration of the views.Source: A. Discussion · p. 12
B. ActiveViewPose-200K
- ActiveViewPose-200K is built through a four-stage pipeline, with assets filtered for indoor relevance, single-object integrity, stable geometry, visual consistency, and clean topology.The assets come from Objaverse and use LVIS semantic categories.Source: B. ActiveViewPose-200K · p. 12, §B.1 · pp. 12–13
B.2. Procedural Scene Generation
- Infinigen scenes combine curated assets across kitchens, living rooms, dining rooms, and bathrooms, with room-type probabilities of 32%, 18%, 29%, and 21%, respectively.The generated scenes vary in lighting, furniture layouts, and clutter distributions.Source: §B.2 · p. 13, Figure 7 · p. 15
B.3. Task Formulation
- Dataset tasks cover pick, reorient, open, close, pour, and place, with prompts for visual centering, explicit spatial directions, or commonsense location inference.Objects are positioned to satisfy task templates, and perturbed initial views are paired with computed target views; scenarios include conditional reasoning and container interaction.Source: §B.3 · p. 13
B.4. Instruction Augmentation via LLM
- GPT-4 rewrites template instructions for linguistic variety while preserving object names, target locations, imperative form, and task difficulty.The prompts permit varied verbs and sentence structure but prohibit changing referents or adding unrelated task steps.Source: §B.4 · pp. 13–14
C. ActiveManip-Bench
- ActiveManip-Bench organizes 12 semantically annotated tasks along task-objective and visual-complexity dimensions.Its simulation uses 100 objects and 20 scenes, with articulated digital twins, precise collision meshes, and dynamic joint parameters.Source: §C.1 · p. 15
C.1. Simulation Environment Details
- The simulation uses a G1 robot with a 2-DoF active head and supports active ego, wrist, and fixed third-person camera viewpoints.The environment is built in Isaac Sim and provides photorealistic rendering and physically accurate simulation.Source: §C.1 · pp. 14–15
C.2. Active Manipulation Task Suite.
ActiveManip-Bench organizes its task suite across task objectives and visual complexity.
C.2. Active Manipulation Task Suite.
- The 12 semantically annotated tasks span atomic skills to long-horizon reasoning and fully visible scenes to those requiring active semantic exploration.These are the benchmark’s two orthogonal task-suite dimensions.Source: §C.2 · p. 15
C.2.1. Hierarchical Task Objectives
The benchmark organizes active manipulation into tasks of increasing horizon and complexity, then varies how visible the target is at initialization. Its data pipeline combines human demonstrations, simulation-based augmentation, and semantically enriched instructions.
C.2.1. Hierarchical Task Objectives
- The 12 tasks span atomic manipulation, short-horizon composites, and long-horizon sequences, with success defined by geometric or physical thresholds held through a stabilization period.Long-horizon examples include opening a container, retrieving and placing an object, then closing it; pouring tests transfer of liquid into a target container.Source: §C.2.1 · p. 17, Table 6 · p. 18
C.2.2. Visual Complexity and Initialization
- Three initialization levels test visibility robustness: fully visible, partially observable through truncation or occlusion, and completely out of view.Out-of-view tasks require using language cues to search before manipulation; physical occlusion may require moving an obstacle.Source: §C.2.2 · p. 18
C.3. Data Collection and Generation Pipeline
- The data-generation pipeline combines human demonstrations, simulation-based augmentation, and semantic instruction enrichment.The teleoperation system captures head pose and hand movement, then retargets wrist and finger states to robot actions through analytical inverse kinematics.Source: §C.3 · p. 18, §C.3.1 · p. 18
C.3.2. Scalable Data Augmentation.
- Visual and asset randomization broaden the training scenes, while residual reinforcement learning refines human-demonstration policies into additional successful trajectories.The flywheel explores local adjustments to failures in the original demonstrations.Source: §C.3.2 · p. 19
C.3.3. Semi-Automatic Semantic Enrichment.
- Structured task templates are expanded by an LLM into linguistically varied, spatially specific instructions, then checked by humans for grounding errors.Verification targets hallucinations and spatial mismatches.Source: §C.3.3 · p. 19
D. Real-World Experiments
The physical-robot experiments evaluate manipulation and active perception, using a humanoid platform whose camera can move independently to explore the scene.
D. Real-World Experiments
- The experimental design covers hardware setup, task protocols, and generalization testing.Figure 10 identifies the robot, hand, camera, and servos used.Source: D. Real-World Experiments · p. 19, Figure 10 · p. 19
D.1. Hardware Setup
- The real-world platform is a Unitree G1 humanoid equipped with Inspire 3 dexterous hands and a custom active head with independent pitch and yaw.A RealSense D455 RGB-D camera on the head provides the egocentric sensor view.Source: §D.1 · p. 19, Figure 10 · p. 19
D.2. Real-World Task Protocols
The real-world protocols test active manipulation under partial visibility, targets outside the initial view, and articulated-container tasks, then probe generalization to unseen conditions.
D.2. Real-World Task Protocols
- Four task categories evaluate manipulation and active perception under varying visibility constraints.The assets and scenarios are illustrated in Figure 11.Source: §D.2 · p. 19
1. Occluded Pick-and-Place.
- For partially truncated pick-and-place, the target or placement goal begins partly outside the camera field of view, requiring spatial inference and viewpoint adjustment.The example transfers food from a left-side basket to a right-side plate when both are initially only partly visible.Source: §1 · p. 19
2. Out-of-View Pick-and-Place.
- In out-of-view pick-and-place, the target or goal is absent from the initial view, so semantic cues guide camera search before manipulation.The example uses “upper shelf” to find a cup and place it into a basket.Source: §2 · pp. 19–20
3. Occluded Articulated Manipulation.
- Occluded articulated manipulation requires finding a container that starts completely out of sight, then opening it, retrieving an object, closing it, and locating the placement goal.The example uses the cue “under the counter” to guide a downward gaze shift toward a cabinet.Source: §4 · p. 20, §3 · p. 20
D.3. Generalization Evaluation
- Generalization is tested on unseen objects and scenes, as well as lighting conditions absent from the training data.Lighting examples include dim evening light and strong directional sunlight.Source: §D.3 · p. 20
E. SaPaVe Architecture and Implementation
SaPaVe combines a decoupled diffusion policy, a semantic camera adapter, and spatial geometry features. Its staged training first aligns language with camera movement, then jointly trains camera and manipulation actions; demonstrations show active search in simulation and recovery on a physical robot.
E.0.1. Decoupled Action Heads via Diffusion Transformer
- The diffusion-transformer policy uses specialized decoders for camera movement and manipulation rather than a single action output.The camera decoder predicts 2-DoF head kinematics, while the manipulation decoder predicts 26-DoF joint positions.Source: §E.0.1 · pp. 20–21
E.0.2. Camera Adapter with Eagle-2 and LoRA
- A LoRA-based Camera Adapter adapts Eagle-2 semantic features to active camera control while freezing the pretrained VLM weights.The adapter uses trainable low-rank matrices in attention layers and less than 2% of trainable parameters.Source: §E.0.2 · p. 21
E.0.3. Universal Spatial Knowledge Injection
- Universal Spatial Knowledge Injection combines semantic tokens with spatial features so action decoding is conditioned on both language intent and 3D geometry.The fused representation guides each denoising step through the action heads’ cross-attention layers.Source: §E.0.3 · p. 21
E.1. Detailed Training Protocols
- Training proceeds in two stages to establish perception before coordinating camera and manipulation actions.The stated goal is to avoid manipulation before the robot has established a valid visual observation.Source: §E.1 · p. 21, §E.1.2 · p. 22
E.1.1. Stage 1: Semantic Active Perception Alignment
- Stage 1 trains semantic camera control on ActiveViewPose-200K examples that pair images and instructions with camera actions.The data targets transitions from suboptimal or out-of-view starting states to task-oriented viewpoints; the manipulation decoder and spatial encoder are frozen.Source: §E.1.1 · pp. 21–22
E.1.2. Stage 2: Active Manipulation Fine-tuning
- Stage 2 jointly trains camera and manipulation decoders using a mixture of camera-pose data, 12-task manipulation data, and real-world data.The combined objective weights head-movement and body-manipulation losses.Source: §E.1.2 · p. 22
F.1. Simulation Demonstrations
- Simulation demonstrations show semantic camera search before arm movement in out-of-view tasks and head repositioning to inspect opened containers before grasping.The sequences cover 12 active manipulation tasks.Source: §F.1 · p. 22
F.2. Real-World Robot Deployment
- On the physical robot, SaPaVe re-initiates active search when a target is moved out of the current frame during execution.The examples include out-of-view pick-and-place and occluded articulated manipulation.Source: §F.2 · p. 22
G. Limitations and Future Work
SaPaVe’s current fixed-base setup expands perception with an active head, but manipulation remains bounded by arm reach. Future work proposes mobile active manipulation that controls both the head and base.
G. Limitations and Future Work
- The current robot has a fixed base, so its manipulation workspace remains bounded by the stationary arms’ reach.The active head expands the effective perceptual field of view, but does not expand that workspace.Source: G. Limitations and Future Work · pp. 22–23
- Objects beyond the arms’ maximum span cannot be reached, confining current active manipulation to local exploration rather than global search.Source: G. Limitations and Future Work · p. 23
- Future work proposes mobile manipulation that controls both head movement and base navigation to find objects and position the robot for manipulation.The proposed framework includes moving to locate objects and positioning the base to improve manipulability in confined spaces.Source: G. Limitations and Future Work · p. 23