Source-linked AI summary
RVT-2: Learning Precise Manipulation from Few Demonstrations
Ankit Goyal, Valts Blukis, Jie Xu, Yijie Guo, Yu-Wei Chao, Dieter Fox
TL;DR
Few-shot 3D manipulation systems must handle multiple tasks precisely, but prior approaches struggle with high-precision tasks. RVT-2 combines architectural and system-level improvements to increase speed and precision, achieving stronger RLBench performance and real-world peg and plug insertion with 10 demonstrations.
Problem
Few-shot robotic manipulation needs to support multiple tasks and high-precision execution from only a few demonstrations.
Method
RVT-2 combines architectural and system-level improvements, including multi-stage inference, convex upsampling, and location-conditioned features for rotation prediction.
Results
RVT-2 improves RVT's training speed by 6X, inference speed by 2X, and RLBench task success by 15 points, while solving real-world peg and plug insertion with 10 demonstrations.
Takeaways & Limitations
A single RVT-2 model can perform multiple real-world, millimeter-level precision manipulation tasks using one third-person camera and few demonstrations.
Takeaways & Limitations
The individual techniques used in RVT-2 are not novel; the contribution is their effective combination.
Abstract
from arXiv · showhide
In this work, we study how to build a robotic system that can solve multiple 3D manipulation tasks given language instructions. To be useful in industrial and household domains, such a system should be capable of learning new tasks with few demonstrations and solving them precisely. Prior works, like PerAct and RVT, have studied this problem, however, they often struggle with tasks requiring high precision. We study how to make them more effective, precise, and fast. Using a combination of architectural and system-level improvements, we propose RVT-2, a multitask 3D manipulation model that is 6X faster in training and 2X faster in inference than its predecessor RVT. RVT-2 achieves a new state-of-the-art on RLBench, improving the success rate from 65% to 82%. RVT-2 is also effective in the real world, where it can learn tasks requiring high precision, like picking up and inserting plugs, with just 10 demonstrations. Visual results, code, and trained model are provided at: https://robotic-view-transformer-2.github.io/.
I. INTRODUCTION
RVT-2 targets precise, multitask 3D manipulation from few demonstrations and improves RVT through architectural and system-level changes. It reports faster training and inference, higher RLBench success, and real-world millimeter-level manipulation with 10 demonstrations.
- Motivation: RVT-2 addresses multitask, precise 3D manipulation from only a few demonstrations.The target system should handle multiple tasks, require few demonstrations, and solve tasks with high precision.
- Motivation: Prior methods such as PerAct and RVT made progress but RVT still struggled with high-precision tasks like screwing bulbs and peg insertion.PerAct used voxel-based scene representations, while RVT introduced multi-view representations with faster training and inference.
- Contributions: 6X faster training, 2X faster inference, and a 15-point RLBench success-rate gain from 62.9 to 77.6 are reported for RVT-2 over RVT.Training increased from 2.4M to 16M samples per day, while inference increased from 11.6 fps to 20.6 fps.
- Real-world results: A single RVT-2 model solves multiple real-world tasks requiring millimeter-level precision with as few as 10 demonstrations and one third-person camera.Demonstrated tasks include peg insertion and plug insertion; the paper describes this as the first such few-example vision-policy test for these high-precision tasks.
- Contributions: RVT-2 combines a multi-stage inference pipeline, convex upsampling, location-conditioned rotation features, accelerated rendering, optimized optimizers, and mixed-precision training.The multi-stage pipeline zooms into the area of interest for more precise end-effector pose prediction.
II. RELATED WORK
The related work frames 3D manipulation as more difficult than planar manipulation and reviews visual, sensory, and representation-based approaches. RVT-2 builds on RVT’s multi-view, key-frame formulation for language-conditioned multitask manipulation.
- Robotic Manipulation in 3D: Full 3D manipulation is harder than 2D top-down manipulation because of higher action-space degrees of freedom and complex spatial reasoning.Prior vision-based systems use camera images, while depth information is commonly required for more effective 3D spatial reasoning.
- Robotic Manipulation in 3D: High-precision manipulation has used proprioception, visual residual reinforcement learning, and reinforcement learning, often with data-expensive algorithms.Examples include peg-in-hole imitation learning and visual industrial insertion methods.
- RVT-2: RVT-2 trains one model to solve multiple 3D tasks from language instructions and few demonstrations while improving RVT’s performance, precision, and speed.It retains RVT’s key-frame-based manipulation paradigm and groups changes into architectural and system-related improvements.
- Background: Key-frame methods learn to predict the next bottleneck pose from a language goal and the current scene point cloud.A trajectory can be represented by poses such as pre-grasp, grasp, and pull for drawer opening.
- RVT: RVT reconstructs a scene point cloud, renders five orthogonal virtual views, and uses a multi-view transformer to predict gripper location, rotation, and state.Heatmaps are back-projected into 3D to select the highest-scoring point for gripper location.
B. Architectural Changes: RVT →RVT-2
RVT-2 changes RVT’s architecture to improve precision and efficiency, especially when objects are small or pose accuracy is critical. Its main design uses coarse localization followed by zoomed-in pose prediction, alongside revised upsampling, parameters, rotation features, and view selection.
- Multi-Stage Design: RVT-2 first predicts an area of interest from fixed views, then re-renders a 4× zoomed region for precise gripper-pose prediction.The zoomed-in cameras cover one quarter of the coarse cameras’ region.
- Convex Upsampling: Convex upsampling replaces feature upsampling and predicts high-resolution heatmaps through learned convex combinations of coarse-grid features.The method saves memory without sacrificing performance and produces sharper higher-resolution predictions.
- Rotation Prediction: Location-conditioned local features enable rotation prediction that depends on the selected end-effector location rather than only global visual features.This addresses scenes where similar objects have different orientations and either location is valid.
- Fewer Virtual Views: RVT-2 uses three virtual views—front, top, and right—without sacrificing performance in its multi-stage model.Fewer views reduce rendered images and tokens because final prediction uses zoomed-in views.
C. System-Related Changes: RVT →RVT-2
The system-level changes accelerate virtual point-cloud rendering by replacing a general differentiable renderer with a custom CUDA projection renderer. The renderer projects points, resolves visibility by depth, and splats points in screen space.
- Point-Renderer: A custom projection-based point-cloud renderer in CUDA replaces PyTorch3D to reduce rendering time and memory overhead.PyTorch3D is fully featured but incurs significant overhead for point-cloud rendering.
- Projection: Projection maps each 3D point to camera-dependent depth and pixel coordinates, then computes a linear image-pixel index.The operation can be accelerated with GPU matrix multiplications.
- Z-ordering: Z-ordering keeps the nearest point at each pixel and assigns its RGB value and depth to the rendered images.The renderer accelerates this step by packing depth and point index into a 64-bit integer and using atomicMin CUDA operations.
- Screen-space splatting: Screen-space splatting models each point as a finite camera-facing disc to reduce noise when point-cloud resolution is not higher than image resolution.This follows projection and Z-ordering in the rendering pipeline.
IV. EXPERIMENTS
RVT-2 is evaluated comprehensively in both simulation and the real world.
- The evaluation covers simulation experiments.
- The evaluation also covers real-world experiments.
- Together, these experiments assess RVT-2 across simulated and physical manipulation settings.
A. Simulation
In RLBench simulation, RVT-2 is evaluated across 18 language-specified manipulation tasks against established and recent baselines, with strong performance and substantial training-efficiency gains.
- Dataset and Setup: RVT-2 is evaluated on 18 RLBench tasks spanning non-prehensile, pick-and-place, and high-precision peg-in-hole manipulation.
- Baselines: RVT-2 is compared with behavioral-cloning, 3D-manipulation, and recent methods including PerAct, RVT, Act3D, PolarNet, and HiveFormer.
- Training time vs. Performance: 6X faster training and 19% absolute performance improvement are reported for RVT-2 versus RVT under the same compute budget.
- Multi-Task Performance: 81.4% average success rate is achieved by RVT-2, which outperforms Act3D by 16.4% absolute while using 6X less training compute.
- High-Precision Tasks: RVT-2 outperforms other methods on high-precision tasks including insert peg, stack cups, and screw bulb.
B. Real World
In the real world, a single RVT-2 model learns multiple tasks from roughly 10 demonstrations per task and improves over RVT, including on newly introduced high-precision tasks.
- Dataset and Setup: The real-world setup uses a statically mounted Franka Panda arm and a fixed third-person Azure Kinect RGB-D camera positioned closer for point-cloud quality.
- Dataset and Setup: Approximately 10 demonstrations per task are collected for the existing and new real-world tasks, whose object and socket locations are randomized.
- Experiment Results: Both RVT-2 and RVT use one model across eight tasks and 16 variations, with results reported in Table II.
- Experiment Results: RVT-2 improves over RVT by 8 absolute points on the five tasks inherited from RVT.
- Experiment Results: 53.3% average success rate is achieved by RVT-2 on three new high-precision tasks, versus 33.3% for RVT.
- Failure Modes: Minor position-prediction inaccuracies are the major failure reason for several insertion tasks, while incorrect object selection also causes failures.
C. Ablations
Ablations show that RVT-2’s architectural and system-level components jointly affect speed and performance, with multi-stage inference improving accuracy while some optimizations accelerate training.
- Multi-Stage Design: 17.5% success-rate improvement comes from the multi-stage design, although the added rendering and inference stage slows training.
- Parameter Rationalisation: GPU-friendly parameter rationalisation accelerates training without compromising network performance.
- Location Conditioned Rotation: 4.2% performance improvement results from predicting rotation with local, location-conditioned features.
- Fewer Virtual Views: Reducing virtual camera views from 5 to 3 maintains task performance while doubling training speed.
- Convex Upsampling: Removing convex upsampling increases training time by 20.7 hours and, with mixed precision, produces undefined gradients.
- Improved Training Pipeline: Removing automatic mixed precision increases training time by 300%, while removing 8-bit LAMB and fast attention adds a further 20%.
D. Generalization Case Study
RVT-2 is evaluated on generalization beyond training conditions, including unseen object configurations, workspace positions, lighting, backgrounds, appearances, and language formulations.
- RVT-2 is tested with different object configurations and object positions across a 2 feet X 2 feet workspace.
- The block stacking evaluation varies lighting conditions, background table appearance, and object appearance.
- The model is also evaluated on paraphrased language instructions for block stacking, beyond the training format “put x block on y block.”
E. Failure Recovery Case Study
RVT-2 operates in a closed loop at key-point frequency and can adjust execution when the scene changes or recover from failures during simulated manipulation.
- RVT-2 uses the current observation to predict the next pose, making the policy closed loop at the frequency of key-points.
- When the target block is moved midway through stacking, the policy adjusts its behavior accordingly.
- In simulation, the system repeatedly retries stacking after a block falls, indicating learned failure recovery behavior.
V. CONCLUSIONS AND LIMITATIONS
RVT-2 combines architectural and system-level improvements to advance speed, precision, and task performance in few-shot 3D manipulation. It outperforms prior methods on RLBench and performs high-precision insertion tasks in the real world, while retaining important limitations.
- Conclusions: RVT-2 is a fast and precise 3D object manipulation model built on the prior state-of-the-art RVT.
- Conclusions: Its gains come from combining architectural and system-level improvements rather than introducing individually novel techniques.
- Conclusions: RVT-2 significantly outperforms prior methods on RLBench while requiring much less compute.
- Conclusions: In the real world, RVT-2 solves high-precision peg and plug insertion using one third-person camera and 10 demonstrations.
- Limitations: RVT-2 is limited to trained object instances, can fail from minor insertion-position errors, and reaches 82% simulation and 72% real-world success rates.