Source-linked AI summary
Vision in Action: Learning Active Perception from Human Demonstrations
Haoyu Xiong, Xiaomeng Xu, Jimmy Wu, Yifan Hou, Jeannette Bohg, Shuran Song
TL;DR
Robotic imitation systems often lack active perception, and mismatched human and robot viewpoints can limit the perceptual strategies learned from demonstrations. ViA combines a flexible active neck, shared-observation VR teleoperation, and an intermediate 3D scene to learn those strategies; it outperforms baseline camera setups on three occlusion-heavy tasks.
Problem
Many robotic imitation systems omit active perception, relying on camera views that can be occluded or mismatched with the human demonstrator’s gaze.
Source: §1 · p. 2Method
ViA learns active perceptual strategies and bimanual manipulation from human demonstrations using a flexible neck and shared-observation teleoperation with an intermediate 3D scene.
Source: §1 · p. 2Results
ViA outperforms wrist-camera and fixed-chest-camera baselines by 45% in success rate on three multi-stage bimanual tasks with significant visual occlusions.
Source: §1 · p. 2Takeaways & Limitations
The results show that active perception is important for the evaluated occlusion-heavy bimanual manipulation tasks.
Source: §1 · p. 2, §5 · p. 8The current tabletop system may not transfer directly to mobile manipulation, where active perception is more challenging.
Source: §6 · p. 9
Abstract
from arXiv · showhide
We present Vision in Action (ViA), an active perception system for bimanual robot manipulation. ViA learns task-relevant active perceptual strategies (e.g., searching, tracking, and focusing) directly from human demonstrations. On the hardware side, ViA employs a simple yet effective 6-DoF robotic neck to enable flexible, human-like head movements. To capture human active perception strategies, we design a VR-based teleoperation interface that creates a shared observation space between the robot and the human operator. To mitigate VR motion sickness caused by latency in the robot's physical movements, the interface uses an intermediate 3D scene representation, enabling real-time view rendering on the operator side while asynchronously updating the scene with the robot's latest observations. Together, these design elements enable the learning of robust visuomotor policies for three complex, multi-stage bimanual manipulation tasks involving visual occlusions, significantly outperforming baseline systems.
1 Introduction
Active perception uses deliberate viewpoint changes to search, reduce occlusion, and focus on action-critical regions. ViA learns these strategies from human demonstrations through a shared-observation system and outperforms baseline camera setups on three occlusion-heavy tasks.
Motivation
- Retrieving a banana from a bag illustrates why viewpoint changes matter: scanning locates the bag, peeking identifies the banana, and focusing supports grasp selection.These changes increase visual coverage, reduce obstacle occlusion, and direct attention to action-critical regions.Source: §1 · p. 1, Figure 1 · p. 1
- Robotic imitation systems often use wrist or fixed third-person cameras, which can miss task-relevant information under occlusion and fail to capture human searching, tracking, and focusing.Human gaze shifts during demonstrations, while the robot often observes from a fixed or mismatched viewpoint.Source: §1 · pp. 1–2
Approach
- ViA learns active perception directly from human demonstrations, using a flexible 6-DoF robot neck, an intermediate 3D scene representation, and shared observations rather than hand-designed gaze strategies.The shared observation space captures human perceptual strategies across task stages and scenarios.Source: §1 · p. 2
Results
- ViA outperforms wrist-camera and fixed-chest-camera baselines by 45% in success rate across three multi-stage bimanual tasks with significant visual occlusions.The tasks involve retrieving objects with interactive perception, rearranging cups in clutter, and precisely aligning objects with coordinated bimanual actions.Source: §1 · p. 2
2 Related Work
Prior work explores active vision through robot necks, heuristics, uncertainty reduction, reinforcement learning, and VR teleoperation. ViA distinguishes itself by learning manipulation and active perceptual behavior from real-world demonstrations while using physical camera control and an intermediate 3D scene to address latency.
Active Perception and Robot Necks
- Active-vision systems include humanoid necks with varying degrees of freedom; ViA instead uses an off-the-shelf 6-DoF arm as a directly teleoperated active neck.The idea of using an arm as a neck has prior precedents; the paper distinguishes its integrated teleoperation interface and demonstration-based learning.Source: §2 · pp. 2–3
- Prior active-vision systems often use hand-designed saliency heuristics or uncertainty reduction, while reinforcement-learning approaches remain challenging to apply to real-world systems.The cited approaches typically define objectives around perception rather than manipulation.Source: §2 · p. 3
Teleoperation Systems
- Existing teleoperation often uses wrist or fixed third-person cameras, while direct VR camera streaming can induce motion sickness from motion-to-photon latency.ViA renders views from an intermediate 3D scene and allows purposeful physical camera control, unlike a related radiance-field system that lacks physical camera control.Source: §2 · p. 3
3 The Vision in Action System
The Vision in Action system combines a flexible active neck, VR teleoperation through an intermediate 3D scene, and a visuomotor policy that learns neck and bimanual actions from demonstrations.
3.1 Hardware Design
- ViA uses an off-the-shelf 6-DoF robot arm as a neck because a static torso with a 2-DoF neck cannot reproduce the full range of human head motion.The active head camera provides RGB, depth, and synchronized camera-pose data; two additional 6-DoF arms support bimanual manipulation.Source: §3.1 · pp. 3–4
3.2 Teleoperation Interface
- The teleoperation interface balances user visual latency below 7 ms with smooth robot action execution below 10 Hz control frequency, enabling effective, practical data collection for complex manipulation tasks.Stereo RGB views are rendered from the point cloud using the user’s latest head pose. The robot head pose is updated using aggregated user head pose, while the point cloud is asynchronously updated with new RGB-D observations at 10 Hz.Source: §3.2 · pp. 4–5
3.3 Learning Active Perception for Bimanual Manipulation
- ViA’s policy learns active perception alongside bimanual manipulation, predicting arm actions and neck movements from visual and proprioceptive observations.The policy is based on Diffusion Policy and uses the active head camera to learn from human demonstrations.Source: §3.3 · p. 5
- The system coordinates neck and arm end-effector poses in a shared world frame, using RGB images and proprioceptive state to condition policy actions.The proprioceptive state includes neck and arm end-effector poses and gripper widths; the policy predicts future action sequences and executes only the first part.Source: §3.3 · p. 5
4 Evaluation
Evaluation tests ViA on three multi-stage bimanual tasks and compares camera setups, visual representations, and teleoperation interfaces. ViA outperforms the chest-and-wrist camera baseline, while the interface reduces reported motion sickness at the cost of slightly longer demonstrations.
4 Evaluation
- Evaluation spans bag retrieval, occluded cup transfer and placement, and lime-and-pot alignment, with cumulative stage-wise success requiring all earlier stages.The tasks involve interactive perception, active viewpoint switching, and coordinated bimanual actions, respectively.Source: §4 · p. 5
4.1 Policy Learning Camera Setup Comparison
- An average 18.33% performance decrease followed adding wrist-camera observations to ViA’s active-head-camera input.The authors hypothesize that added views can be redundant or noisy, particularly under frequent occlusion and limited training data.Source: §4.1 · pp. 6–7
- 45% higher average task performance with ViA than with Chest & Wrist Cameras, which lack sufficient task-relevant visual information.ViA consistently outperforms both alternative camera setups across all three tasks.Source: §4.1 · pp. 6–7, Figure 5 · p. 7
4.2 Policy Learning Visual Representation Comparison
- ViA’s pretrained DINOv2 representation achieves the highest final-stage success rate across all three tasks compared with ResNet-DP and DP3.The authors describe DP3 errors including directing the arm toward an empty shelf section in the cup task and failing the bag task’s opening stage.Source: Figure 6 · p. 8, §4.2 · p. 8
4.3 Teleoperation Interface Comparison
- Point-cloud rendering slightly lengthens data collection versus stereo RGB streaming, but reduces motion sickness; 6 of 8 participants preferred it.Source: Figure 7 · p. 8, §4.3 · p. 8
5 Conclusion
ViA combines a robotic neck for human-like head movements with a real-time teleoperation interface that asynchronously updates its scene. On three occlusion-heavy bimanual tasks, ViA significantly outperforms baseline setups.
5 Conclusion
- The simple robotic neck lets the robot mimic human-like head movements.Source: §5 · p. 8
- The interface renders views from the user’s latest head pose while asynchronously updating the scene through control of the robot’s active head camera.Source: §5 · p. 8
- Across three challenging, multi-stage tasks with significant visual occlusion, ViA significantly outperforms baseline setups.Source: §5 · p. 8
6 Limitations
The paper frames ViA as an initial step toward learning active perception from human demonstrations and identifies interface, hardware, and policy-learning limitations that motivate further work.
6 Limitations
- ViA is an initial step toward the broader goal of learning active perception from human demonstrations.The authors identify several limitations and areas for future improvement.Source: §6 · p. 9
Teleoperation Interface Design
- Noisy depth sensing and incomplete scene reconstruction can make the 3D scene visualization lower-fidelity than RGB video, challenging fine-grained manipulation.Users often require practice to adapt; dynamic scene fusion and rendering remain future directions.Source: §6 · p. 9
Hardware Design
- The off-the-shelf 6-DoF arm used as a neck may not replicate the complexity of human whole-body movement.The authors also identify mobile manipulation as a future setting where active perception is more challenging and better reflects real-world scenarios.Source: §6 · p. 9
Policy Learning Design
- The policy framework concatenates encoded camera features rather than fusing observations into a shared space, and it lacks language conditioning and memory.Memory is relevant to search tasks such as Lime & Pot, where the robot should remember searched areas and avoid repetitive behavior.Source: §6 · p. 9