Source-linked AI summary
RVT: Robotic View Transformer for 3D Object Manipulation
Ankit Goyal, Jie Xu, Yijie Guo, Valts Blukis, Yu-Wei Chao, Dieter Fox
TL;DR
Explicit 3D representations improve 3D manipulation but impose computational costs that limit scalability. RVT uses multi-view transformer attention and virtual-view re-rendering, outperforming PerAct across RLBench tasks while training faster and also working on real-world tasks with few demonstrations.
Problem
View-based methods have limited success on 3D manipulation, while explicit voxel representations are computationally expensive and less scalable.
Method
RVT re-renders RGB-D camera input into virtual views around the robot workspace and uses multi-view transformer attention to predict 3D manipulation actions.
Results
RVT achieves 26% higher success than PerAct across 18 RLBench tasks, trains 36X faster for the same performance, and runs at 2.3X its inference speed.
Takeaways & Limitations
RVT performs multi-task manipulation in simulation and the real world, including 5 tasks and 13 variations with 51 demonstrations.
Takeaways & Limitations
RVT requires calibration of camera extrinsics to the robot base, motivating extensions that remove this constraint.
Abstract
from arXiv · showhide
For 3D object manipulation, methods that build an explicit 3D representation perform better than those relying only on camera images. But using explicit 3D representations like voxels comes at large computing cost, adversely affecting scalability. In this work, we propose RVT, a multi-view transformer for 3D manipulation that is both scalable and accurate. Some key features of RVT are an attention mechanism to aggregate information across views and re-rendering of the camera input from virtual views around the robot workspace. In simulations, we find that a single RVT model works well across 18 RLBench tasks with 249 task variations, achieving 26% higher relative success than the existing state-of-the-art method (PerAct). It also trains 36X faster than PerAct for achieving the same performance and achieves 2.3X the inference speed of PerAct. Further, RVT can perform a variety of manipulation tasks in the real world with just a few ($\sim$10) demonstrations per task. Visual results, code, and trained model are provided at https://robotic-view-transformer.github.io/.
1 Introduction
RVT addresses the tension between strong 3D manipulation performance and the scalability costs of explicit voxel representations. It combines multi-view attention with virtual-view re-rendering to achieve accurate, scalable manipulation.
- View-based methods struggle on 3D manipulation tasks, achieving less than 2% success on RLBench.
- Voxel-based methods improve 3D reasoning but scale cubically with resolution, making them more computationally costly and less scalable than image-based methods.Training PerAct on 18 RLBench tasks took 16 days using 8 V100 GPUs.
- RVT achieves 26% higher success than PerAct across 18 RLBench tasks and reaches PerAct’s peak performance in 36X less training time.It also achieves 2.3X PerAct’s inference speed.
- RVT jointly attends across multiple scene views, aggregates view information, and predicts end-effector pose from view-wise heatmaps and features.
- Virtual-view re-rendering decouples transformer inputs from physical camera viewpoints, enabling task-useful views and operation with a single sensor camera.
- The paper contributes a scalable multi-view transformer, studies its architectural choices, and evaluates multi-task manipulation in simulation and the real world.
2 Related Work
Related work spans vision-based manipulation, multi-view representations, and transformer policies. RVT differs by combining re-rendered multi-view inputs with transformer processing for 3D action prediction.
- Vision-based Object Manipulation: Vision-based policies use high-dimensional visual inputs to provide more generalizable observations across tasks and accessible real-world perception.
- Multi-View Networks in Computer Vision: RVT constructs virtual images from scene point clouds and uses a multi-view transformer to predict 3D manipulation actions.
- Vision-based Object Manipulation: Unlike NeRF-based novel-view methods, RVT avoids optimizing a scene NeRF at test time and can operate from a single-view RGB image.
- Multi-Task Learning in Robotics: Multi-task robotics methods use representations such as object point clouds, segmentation, optical flow, and object-centric features, but limited expressiveness can restrict generalization.
- Transformers for Object Manipulation: Transformer-based manipulation models use attention to extract features from sensory inputs, while RVT learns from few demonstrations and handles multiple views.
- Multi-View Networks in Computer Vision: Prior computer-vision work studies multi-view representations for point-cloud recognition, 3D grounding, and view synthesis, whereas RVT focuses on 3D object manipulation.
3 Method
RVT learns manipulation actions from language, RGB-D observations, and gripper state using re-rendered virtual views and a joint transformer. Its multi-view outputs are converted into end-effector actions through heatmap-based translation prediction, rotation and binary-state prediction, and back-projected 3D pose estimation.
- Inputs and outputs: RVT takes task language, RGB-D observations, and gripper state, and predicts the next key-frame end-effector pose and gripper action.Key-frames represent bottleneck steps such as prepick, grasp, or place poses.
- Rendering: RVT reconstructs a scene point cloud, re-renders it from virtual viewpoints around the robot base, and provides each view with RGB, depth, and world-coordinate channels.World-coordinate channels establish pixel correspondences across views by identifying shared 3D points.
- Rendering: The re-rendering pipeline decouples sensor-camera images from transformer inputs, enabling useful arbitrary viewpoints and multi-view reasoning from a single sensor camera.It also supports orthographic views, point-cloud augmentations, and additional correspondence channels.
- Joint Transformer: A joint transformer processes virtual-image patches, language tokens, and gripper state, first attending within images and then propagating information across images and text.The architecture uses eight self-attention layers, with the first four restricted within images and the last four enabling cross-image and text attention.
- Action Prediction: The model predicts translation heatmaps for each view and combines them with image features to form global features for rotation, gripper state, and collision-indicator prediction.Its action contains a 6-DoF pose, binary gripper state, and binary collision-planner indicator; training uses cross-entropy and binary classification losses.
- Evaluation: 35 RVT is 2.3X faster on execution speed than PerAct and outperforms it on 16/18 RLBench tasks.The comparison uses the same GPU model for PerAct and RVT training-time and inference-speed measurements.
4 Experiments
RVT is evaluated across simulation and real-world manipulation settings, including baseline comparisons, ablations, rendering choices, and multi-task performance. It achieves strong multi-task simulation results with substantially lower training and inference costs, while real-world performance varies by task and sensing quality.
- Simulation Setup: RVT is evaluated on the same 18 RLBench tasks as PerAct, covering picking, placing, tool use, drawer opening, and peg insertions.The training dataset uses 100 expert demonstrations per task.
- Multi-Task Performance: RVT outperforms PerAct by 13 percentage points (26% relative improvement) averaged across 18 tasks and wins on 16/18 tasks.It also outperforms C2F-ARM by 42 percentage points (213% relative improvement).
- Efficiency: 36X faster training and 2.3X faster inference make RVT more computationally scalable than PerAct at comparable performance.The training comparison uses the same GPU type and number of GPUs for both models.
- Ablation Study: Ablations show improvements from higher rendered-image resolution, view correspondence, depth, separated initial processing, orthographic projection, rotation augmentation, and re-rendered cube views.The resolution comparison specifically reports that virtual image resolution 220 outperforms 100.
- View Design: Five re-rendered views perform best, followed by three aligned views, while a single-view model performs substantially worse for 3D manipulation.Rotating camera locations by 15° relative to the table and robot decreases performance.
- Rendering Analysis: Re-rendered images outperform sensor-camera images because they support useful viewpoints, orthographic projection, 3D augmentation, and correspondence channels.Matching sensor cameras to re-rendering views would require multiple cameras and would forgo several benefits of re-rendering.
- Real-World Experiments: In real-world experiments, one RVT model performs five tasks with 13 variations using about 10 demonstrations per task, but struggles with marker-related tasks.Reported success rates include 100% for stacking blocks, 80% for pressing sanitizer, 50% on longer-horizon drawer and shelf tasks, and 82.5% on non-marker tasks.
5 Conclusions and Limitations
RVT outperforms prior state-of-the-art methods while improving scalability and speed, and it works on real-world manipulation tasks with few demonstrations. The authors identify view specification and camera-to-robot calibration as limitations for future work.
- RVT outperforms PerAct and C2F-ARM on varied 3D manipulation tasks while being more scalable and faster.The conclusion summarizes RVT's performance and scalability advantages across simulation and real-world settings.
- RVT performs real-world manipulation tasks with only a few demonstrations.The paper reports this capability without specifying the exact demonstration count in the cited passage.
- RVT's view specification is not optimized or learned from data, motivating future research.The authors explored several view options and identified one that works well across tasks, but leave learned view specification as future work.
- RVT requires calibration of camera extrinsics to the robot base, unlike prior view-based methods.The authors identify removing this calibration constraint as an extension for future research.
6 Appendix
The appendix documents the RLBench evaluation scope and summarizes RVT's transformer architecture and ablation reporting. The model processes language and virtual scene views before inferring a 3D action.
- 6.1 RLBench Tasks: The evaluation covers 18 RLBench tasks with 249 task variations.Table 3 summarizes these tasks, which are the same tasks used in PerAct.
- 6.2 RVT Overview: RVT converts language and virtual scene images into tokens, processes within-image and joint attention layers, then infers a 3D action.Language uses pretrained CLIP embeddings, while virtual images use patchification and projection before transformer processing.
- 6. Appendix: The appendix reports ablations together with per-task means and standard deviations.Table 4 presents ablation results for RVT on RLBench with metrics for each task.