Source-linked AI summary
An Empirical Study on What Matters for Viewpoint-Generalizable Policies in Visual Imitation Learning
Mino Nakura, Sriram Krishna, Yufei Wang, Shubham Tulsiani, Zackory Erickson, David Held
TL;DR
Visual imitation policies can be brittle when deployment viewpoints differ from training views, particularly when the robot is out of frame. This paper studies design choices for viewpoint generalization, combining dense visual tokens with geometric reasoning in the policy. The resulting policy remains performant across randomized viewpoints and transfers zero-shot from simulation to a real robot under random cameras.
Problem
Pixel-to-action imitation policies can overfit their training camera configurations and degrade sharply under viewpoint shifts, especially when the robot is out of frame.
Source: §I · p. 1Method
The study combines dense visual tokens, a cross-view-finetuned DINOv2 encoder, and robot-frame 4D rotary positional encodings for vision, proprioception, and action tokens.
Source: §I · p. 1, B. Design Axes · p. 4Results
GP outperforms 2D and 3D alternatives under randomized camera placement while approaching fixed-camera performance, and transfers zero-shot to real cameras.
Source: §I · p. 1, §VI · p. 7Takeaways & Limitations
The study identifies retaining dense scene representations and grounding visual, proprioception, and action tokens in scene geometry as factors associated with viewpoint generalization.
Source: §VI · p. 7GP requires calibrated RGB-D input at training and deployment, and the sim-to-real proof of concept covers a single object category.
Source: §VII · p. 8
Abstract
from arXiv · showhide
Visual imitation learning is a promising approach to training robot manipulation policies capable of completing a wide variety of tasks. However, policies today remain brittle to viewpoint perturbations, making deployment in diverse environments a challenge. We present a controlled empirical study of which design choices allow visuomotor policies to generalize across viewpoints. We find that viewpoint generalization improves when dense visual tokens are retained and the action head participates in geometric reasoning. On a suite of simulated tasks that span a wide range of camera poses, we show that these design choices yield a policy that remains performant across viewpoints. As a practical consequence, a policy trained with these design choices also transfers zero-shot from simulation to the real world under random camera configurations.
I. INTRODUCTION
Viewpoint shifts expose a weakness in pixel-to-action imitation policies, particularly when the robot leaves the image. The study identifies design choices for viewpoint robustness and reports that GP performs well across randomized views and transfers zero-shot to real cameras.
I. INTRODUCTION
- End-to-end pixel-to-action policies can overfit their training camera configurations and degrade sharply when the viewpoint shifts, especially when the robot is out of frame.Without explicit structure relating the robot to observations, the policy must learn localization implicitly from demonstrations.Source: §I · p. 1
- The study identifies dense visual features, a cross-view visual encoder, and robot-frame rotary positional encodings as ingredients for viewpoint generalization.Together, these choices retain pretrained image-backbone priors while adding geometric structure associated with 3D approaches.Source: §I · p. 1
- GP outperforms 2D and 3D alternatives under randomized camera placement while approaching fixed-camera performance.The study also reports zero-shot transfer to a real robot under random cameras.Source: §I · p. 1
- GP is a controlled empirical study of design choices, not a new architecture; its components appear in prior work.The study holds the dataset and action head common while isolating the pieces responsible for viewpoint robustness.Source: §I · p. 1
- The paper demonstrates the practical use of its design recipe by training on diverse simulated viewpoints and transferring zero-shot to the real world under random cameras.The supplied contribution passage states this transfer as the practical consequence of the study.Source: §I · p. 1
II. RELATED WORK
Prior work addresses viewpoint robustness through data, representations, camera conditioning, or geometry-aware policies, with differing trade-offs and limited isolation of design effects. The problem setting trains one policy across independently sampled camera poses, assuming calibrated RGB-D observations.
II. RELATED WORK
- Camera-pose conditioning supplies camera location, whereas this work argues that policies need pixel locations relative to the robot; prior sim-to-real studies often use narrower camera randomization or different architectural questions.The passage contrasts this study's focus on architectural choices with large-scale VLA training in related work.Source: §II · p. 2
- Point-cloud and voxel policies offer scene-centric viewpoint robustness, but sacrifice image-scale pretraining and often downsample observations to remain tractable.Reprojection methods can introduce holes that make inputs out-of-distribution for pretrained image backbones.Source: §II · p. 2
- Geometry-aware image features alone may not ensure viewpoint robustness, and related 3D actor studies do not isolate the effects of their design choices on viewpoint generalization.VGGT-DP degrades under modest camera rotations; 3DFA combines several relevant choices without isolating them experimentally.Source: §II · p. 2
III. PROBLEM STATEMENT
- The policy-learning setting uses expert demonstrations as sequences of visual observations, proprioceptive states, and actions, with observations comprising calibrated RGB-D views.The proprioceptive state includes end-effector pose and gripper width; actions are delta end-effector commands.Source: §III · p. 2
- Training and deployment camera poses are independently sampled from the same distribution, and the goal is one policy with consistent performance across that distribution.The setting assumes camera calibration and depth maps, supplied by a sensor or an RGB-based estimator.Source: §III · p. 3
IV. STUDY DESIGN
The study fixes the base policy, data, and evaluation protocol, then varies one design choice at a time across three axes: visual-token compression, visual encoder properties, and positional encoding.
IV. STUDY DESIGN
- The controlled study fixes the base policy, training data, and evaluation protocol, varying one design choice at a time.The choices are organized along three axes for processing observations from changing viewpoints.Source: §IV · p. 3
- One design axis tests whether visual features should be compressed before reaching the action head.The study treats token compression as a distinct choice in how observations are processed.Source: §IV · p. 3
- The other axes vary visual-encoder properties and how token positions are encoded.Together with compression, these form the three design dimensions in the study.Source: §IV · p. 3
A. Base Policy
The base policy uses a transformer action head that attends to visual tokens, and the design axes test how visual information and geometry are represented. GP combines dense visual tokens, a cross-view finetuned DINOv2 encoder, and robot-frame 4D rotary positional encodings.
A. Base Policy
- The action head uses a Diffusion Transformer that cross-attends to observation tokens and self-attends over proprioception and action tokens.An MLP encodes proprioception and noised actions; the policy takes an observation history and predicts an action sequence.Source: A. Base Policy · p. 3
- Flow matching trains the action head to predict the velocity from a noise-interpolated action chunk toward the ground-truth chunk.At inference, Euler integration converts the predicted velocity into a clean action chunk.Source: A. Base Policy · p. 3
B. Design Axes
- The visual-token comparison tests retaining dense patch tokens against compressing observations with spatial softmax or max pooling.The retained-token option lets the action head attend to a set of visual tokens rather than a single global feature vector.Source: B. Design Axes · p. 4
- The encoder comparison includes ResNet-18, frozen and finetuned DINOv2, and DINOv2 with cross-view attention.The cross-view variant freezes the first within-view layers and finetunes later layers, which alternate within-view and cross-view attention.Source: B. Design Axes · p. 4
- The positional-encoding comparison contrasts image-space patch encodings with 4D RoPE based on each token’s robot-frame position and timestep.Image-token positions come from lifting pixels with depth and camera pose; proprioception-token positions use the gripper’s 3D position.Source: B. Design Axes · p. 4
- Action-token positions are estimated by adding predicted translational displacements to the current gripper position.As the action chunk denoises, these positions converge toward the future gripper trajectory.Source: B. Design Axes · p. 4
- GP combines dense tokens, a cross-view finetuned DINOv2 encoder, and robot-frame 4D RoPE; its ablations change one design axis at a time.The study also reports Diffusion Policy, ACT, and DP3 as external reference points that differ from the base policy in multiple components.Source: B. Design Axes · p. 4
V. EXPERIMENTS AND ANALYSIS
The experiments compare GP with viewpoint-generalization methods on simulated manipulation tasks using randomized cameras, then test transfer to a physical robot. The cabinet task uses over-the-shoulder and wrist-mounted views, with evaluation across randomly sampled camera poses.
Experimental Setup
- The study evaluates simulated tasks under randomized cameras and demonstrates zero-shot transfer on a physical robot with camera poses unknown during simulation data generation.The real-world camera poses are within the distribution of randomized simulation camera poses.Source: §V · p. 4
- The comparison evaluates GP against viewpoint-generalization methods on tasks from Jiang et al. and an articulated-object task, reporting each method’s best-checkpoint success rate averaged over three seeds.The table covers cabinet opening and the “Do You Know Where Your Camera Is?” benchmark; its average is over Cabinet, Pick Place Can, and Square Assembly.Source: §V · p. 4, Table I · p. 5
- The cabinet policy imitates an expert opening a small-knob cabinet from two randomly sampled over-the-shoulder views, with a wrist camera keeping the object visible.The dataset has 989 demonstrations and over 100,000 observation-action pairs; evaluation uses 100 test trajectories with random camera poses and normalized opening performance.Source: §V · pp. 4–5
A. Camera information
The camera-information comparison holds the visual encoder and action-head architecture as similar as possible, comparing positional encodings, Plücker raymaps, and canonical views. The reported results show that 4D RoPE outperforms or matches the alternatives across tasks.
A. Camera information
- The comparison uses the same visual encoder and action-head architecture across camera-information approaches, while GP uses 4D RoPE and the others use additive sinusoidal embeddings.This controls the comparison as much as possible across approaches.Source: A. Camera information · p. 5
- Plücker conditioning fuses raymap tokens with corresponding DINOv2 image tokens, while canonical views reproject lifted colored point clouds into virtual camera views.Plücker tokens condition the DiT action head via cross-attention; canonical views use fixed virtual cameras for the cabinet task.Source: A. Camera information · p. 5
- 4D RoPE outperforms or performs competitively with the alternatives across all tasks in the camera-information comparison.Table I covers articulated cabinet opening and the “Do You Know Where Your Camera Is?” benchmark, with randomized cameras unless marked fixed.Source: A. Camera information · p. 5
B. Design choice analysis
The ablations vary visual-token compression, encoder, and positional encoding relative to GP. Dense tokens and robot-frame positional encodings show the clearest reported benefits under randomized cameras, while cross-view attention adds gains that are less significant than those changes.
B. Design choice analysis
- Replacing dense tokens with global compression causes a large performance drop under randomized cameras, although compression performs reasonably with fixed cameras.The authors attribute the drop to loss of spatial information needed by the action head to reason across viewpoints.Source: B. Design choice analysis · p. 5
- Cross-view attention yields additional gains by helping the encoder fuse information across cameras, but encoder changes matter less than removing token compression or changing positional encoding..Source: B. Design choice analysis · p. 5
- Robot-frame 4D RoPE improves Cabinet normalized opening from 39.5 to 64.4 under randomized cameras, where identical image-patch indices can map to different 3D locations as the camera moves.Table II compares positional-encoding ablations across one fixed-camera task and three randomized-camera tasks.Source: B. Design choice analysis · p. 5, Table II · p. 6
- The extrinsic-noise evaluation tests Cabinet normalized opening with test-time calibration perturbations even though the policy was trained without calibration noise.The figure marks expected real-world calibration error at approximately 1 cm translation and 1° rotation.Source: Figure 4 · p. 6
C. Sim-to-Real Transfer
The study tests whether the viewpoint-generalization recipe transfers to real-world microwave opening. GP outperforms Canonical Views and Plücker Raymaps on both reported real-world measures.
C. Sim-to-Real Transfer
- The microwave policy is trained from simulated demonstrations of five objects, totaling 2,358 demonstrations and over 100,000 state-action pairs.The objects are selected from the 322-object set used in.Source: C. Sim-to-Real Transfer · p. 6
- Visual domain randomization varies scene appearance and camera poses, and adds depth-sensor noise; all approaches use the same randomized dataset.Trajectories are rendered from two over-the-shoulder views and one wrist camera view.Source: C. Sim-to-Real Transfer · p. 6
- GP outperforms Canonical Views and Plücker Raymaps on both grasp success rate and opening performance in the real-world microwave task.The evaluation averages 30 trials across three camera configurations.Source: C. Sim-to-Real Transfer · p. 7, Figure 6 · p. 7
D. Generalization to out-of-distribution viewpoints
Under unseen elevated and oblique camera views, GP retains an advantage over other camera-conditioned approaches. It also achieves a comparably small fixed-to-randomized-camera gap to DP3 while exceeding DP3 in performance.
D. Generalization to out-of-distribution viewpoints
- GP retains its advantage over other camera-conditioned approaches under unseen viewpoints, although all policies drop versus in-distribution randomized cameras.The unseen views include elevated and oblique angles.Source: D. Generalization to… · p. 7
E. Comparison to 3D Policies
- The DP3 comparison uses an object point cloud fused from two randomly sampled views and a wrist view, then downsampled to 4,500 points.The object is segmented using a ground-truth mask, and the gripper point cloud supplies proprioceptive input.Source: E. Comparison to 3D Policies · p. 7
- GP achieves a comparably small fixed-to-randomized-camera gap to DP3 and exceeds DP3 in performance.DP3's small gap is consistent with the viewpoint robustness of a point-cloud policy.Source: E. Comparison to 3D Policies · p. 7, Table IV · p. 7
F. Robustness to Calibration and Depth Noise
GP tolerates reasonable calibration error but degrades as depth noise increases. The study concludes that dense scene representations and robot-frame geometry are key factors, while calibrated RGB-D input and a single-category real-world demonstration bound the current scope.
F. Robustness to Calibration and Depth Noise
- GP shows robustness to reasonable calibration error but tends to degrade as test-time depth noise increases.The policy is trained without depth or calibration error; training with such noise may improve robustness.Source: F. Robustness to Calibration and Depth… · p. 7
VI. CONCLUSION
- The study identifies retaining dense scene representations and grounding visual, proprioception, and action tokens in scene geometry as factors for viewpoint generalization.GP instantiates these recommendations and is reported to perform across camera variations.Source: §VI · p. 7
- The paper demonstrates zero-shot sim-to-real transfer under random cameras as a practical use case for its recommendations.Source: §VI · p. 7
VII. LIMITATIONS AND FUTURE WORK
- GP requires calibrated RGB-D input at training and deployment, and the real-world transfer demonstration covers a single object category.The paper identifies scaling to large, heterogeneous real-world datasets as an open question.Source: §VII · p. 8
APPENDIX I RANDOMIZATION DETAILS
The appendix describes per-trajectory task and camera randomization, alongside visual, lighting, camera, and depth variations used for sim-to-real transfer.
A. Demonstration Generation
- Initial object pose, scale, and robot/object configurations are sampled per trajectory, with camera poses resampled if the handle is not visible initially.Initial arm joints place the end-effector within 0.8 m of the object.Source: A. Demonstration Generation · p. 9, Table VI · p. 12
B. Visual Domain Randomization
- For sim-to-real rendering, appearance, lighting, and camera intrinsics and extrinsics are randomized across the object, robot, table, and background.The table dimensions are randomized around measurements from the real robot workstation.Source: B. Visual Domain Randomization · p. 9, Table V · p. 12
- Camera parameters receive small focal-length, orientation, and translation jitter, while depth edge artifacts are modeled with correlated noise from a shifted depth map.The noise is produced using bilinear interpolation.Source: B. Visual Domain Randomization · p. 9
- The appendix visualizes randomized-camera trajectories for the cabinet and microwave datasets.Source: Figure 7 · p. 10, Figure 8 · p. 11