RVT-2: Learning Precise Manipulation from Few Demonstrations

• RVT-2 raises RLBench task success rate by 15 points (62.9 to 77.6), trains 6X faster (2.4M to 16M samples per day), and infers 2X faster (11.6 to 20.6 fps), achieving state-of-the-art results.Source: §I · p. 2
• With as few as 10 demonstrations and a single third-person camera, one RVT-2 model performs real-world peg-in-hole and plug-in-socket tasks requiring millimeter-level precision.The authors describe this as the first test of a few-example-trained vision-based policy on such high-precision tasks.Source: §I · p. 2
• RVT-2 adds a multi-stage inference pipeline for more precise end-effector poses, convex upsampling for training memory and speed, and location-conditioned features for rotation prediction.Source: §I · p. 2
• A custom virtual image renderer replaces RVT’s PyTorch3D [35] renderer to improve speed and reduce memory use in training and inference; training also uses optimized optimizers and mixed precision.The authors say the contribution is successfully combining these practices into a precise 3D manipulation system.Source: §I · p. 2
• The paper reports careful ablations quantifying how different factors contribute to RVT-2’s improvements.Source: §I · p. 2

II. RELATED WORK

• Prior 3D manipulation methods use RGB-D images, point clouds, voxels, or multi-scale feature clouds; RVT [17] uses multi-view virtual images, while RVT-2 zooms into task-critical scene regions.Full 3D manipulation is challenging because it involves more action-space degrees of freedom and complex spatial reasoning; the zoom-in strategy follows coarse-to-fine inference [25] [13].Source: §II · p. 2
• RVT-2 learns high-precision tasks with 10 real-world robot demonstrations, unlike MimicPlay [47], which uses human videos for pretraining and 20–40 robot demonstrations for fine-tuning.MimicPlay focuses on long-horizon continuous control, whereas RVT-2 predicts key-points for tasks requiring millimeter-level precision.Source: §II · p. 2
• Earlier high-precision methods use proprioception, cameras, or tactile feedback, but often require reinforcement learning or hundreds of demonstrations and learn one model per task [18] [37] [45] [30] [33] [11].High-precision manipulation concerns tasks with low motion-error tolerance, including industrial tasks.Source: §II · pp. 2–3
• Unlike ACT [53], RVT-2 accepts language to handle task variations, predicts key-points from point clouds, and is not limited to one task variation at a time.ACT predicts continuous joint states from multiview images and does not take language input.Source: §II · p. 3
• Virtual views apply image-based CNN and transformer architectures to 3D scenes; prior work used them in vision tasks [43] [15] [20] [6] [23], while RVT [17] brought multi-view representations to robot action prediction.RVT-2 builds on RVT with architectural and system-level improvements for performance and efficiency.Source: §II · p. 3

III. METHOD

• RVT-2 uses one language-conditioned, key-frame model to solve multiple 3D manipulation tasks from few demonstrations, with architectural and system-level improvements over RVT [17].The method section groups its improvements into changes to the neural network and software optimizations.Source: §III · p. 3

A. Background

• Key-frame behavioral cloning predicts the next pose from a language goal and scene point cloud; RVT renders five virtual views and uses multi-view heatmaps to locate the gripper.RVT averages scores from 2D projections back onto 3D points to select the gripper location, and uses global features to predict rotation and gripper state.Source: A. Background · p. 3

B. Architectural Changes: RVT →RVT-2

• RVT-2 first predicts an area of interest from fixed views, then renders 4× zoomed-in views around it to predict the gripper pose.The close-up cameras cover one-quarter the region of the coarse cameras; adaptive rendering uses RVT’s virtual-rendering flexibility.Source: B. Architectural Changes: RVT →RVT-2 · pp. 3–4, Figure 2 · p. 4
• RVT-2 replaces memory-intensive feature upsampling with the convex upsampling layer from [46], which predicts high-resolution heatmaps from token-resolution features.The layer uses learned convex combinations of coarse-grid features; the authors report memory savings without sacrificing performance.Source: B. Architectural Changes: RVT →RVT-2 · p. 4
• RVT-2 changes the image and patch sizes to 224 and 14, respectively, reducing transformer tokens and improving speed without affecting performance.These GPU-friendlier values replace RVT’s image size of 220 and patch size of 11.Source: B. Architectural Changes: RVT →RVT-2 · p. 4
• Using only front, top, and right views preserves performance in RVT-2 while reducing rendered images and transformer tokens, improving training and inference speed.The authors suggest zoomed-in views for final prediction explain why three views suffice.Source: B. Architectural Changes: RVT →RVT-2 · p. 5
• RVT-2 predicts rotation from local features pooled at the end-effector location, enabling rotation predictions that depend on the selected location.This addresses scenes with multiple valid end-effector locations where the appropriate rotation varies by location.Source: B. Architectural Changes: RVT →RVT-2 · pp. 4–5

Point-Renderer

• A custom CUDA point-cloud renderer replaces PyTorch3D, whose fully featured differentiable rendering incurs significant time and memory overhead.It projects points using camera parameters, then assigns each pixel the color and depth of the nearest point.Source: C. System-Related Changes: RVT →RVT-2 · p. 5
• Packing depth and point index into a 64-bit integer enables two CUDA kernels and atomicMin to select the nearest point per pixel, extending Schütz et al. to arbitrary feature channels.The first kernel retains the minimum-depth point at each pixel; the second unpacks its index to retrieve the point feature.Source: C. System-Related Changes: RVT →RVT-2 · p. 5
• Screen-space splatting models each point as a camera-facing disc and replaces pixels with nearby closer samples within a radius-dependent threshold to reduce rendering noise.Applying splatting after projection and z-ordering reduces the computation required for those earlier steps.Source: C. System-Related Changes: RVT →RVT-2 · p. 5

Improved training pipeline

• Mixed precision, an 8-bit LAMB optimizer [10], and xFormers attention speed RVT-2 training without affecting performance.The pipeline adopts transformer-training techniques selected for improving speed without reducing performance.Source: C. System-Related Changes: RVT →RVT-2 · p. 5

• The evaluation spans simulated and real-world settings.Source: §IV · p. 5

A. Simulation

• The RLBench evaluation covers 18 tasks, using 100 training demonstrations and 25 unseen test demonstrations per task, with 128×128 RGB-D inputs from four cameras.A simulated Franka Panda robot with a parallel-jaw gripper performs the tasks. Compared methods include Image-BC, C2F-ARM-BC, PerAct, HiveFormer, RVT, PolarNet, and Act3D.Source: A. Simulation · pp. 5–6
• Training uses 8 V100 GPUs for about 80K steps with batch size 192, cosine learning-rate decay, and 2,000 warmup steps; evaluation averages four runs per task.Augmentation applies 12.5 cm translations along each axis and 45° rotation about the z axis; the four-run evaluation accounts for RLBench’s sampling-based motion planner.Source: A. Simulation · pp. 5–6
• 6X faster training than RVT brings a 19% absolute (29% relative) performance gain; RVT-2 also reaches 20 fps, around 2X RVT’s inference speed.RVT-2 outperforms RVT trained for 24 hours and PerAct trained for 16 days within 2 hours of training.Source: Figure 3 · p. 6, A. Simulation · p. 6
• RVT-2 achieves an 81.4% average success rate, beating prior-best Act3D by 16.4% absolute (25% relative) while using 6X less training compute.It achieves the best result on 13 of 18 tasks and an average rank of 1.5.Source: A. Simulation · p. 6
• RVT-2 achieves 88% versus 48% for the previous best on screw bulb, 69% versus 26.4% on stack cups, and 40% versus 27% on insert peg.These tasks test precise alignment: of the bulb base with its socket, cups during picking and placing, and a square peg with a cuboid stick.Source: A. Simulation · p. 6

Dataset and Setup

• The evaluation adds randomized peg and plug insertion tasks to the five RVT tasks, using a closer, fixed Azure Kinect view and about 10 demonstrations per task.The peg locations and plug tray and socket locations are randomized; the plug task requires precise 3D manipulation.Source: B. Real World · pp. 6–7

Training and Evaluation Details

• Both models train on the same dataset for 10 epochs, and the final model is evaluated; RVT uses batch size 24 because it cannot fit RVT-2’s larger batch in memory.RVT uses the official learning rate of 2.4×10^3 [17].Source: B. Real World · pp. 7–8

Experiment Results

• 53.3% average success on the three new high-precision tasks versus 33.3% for RVT; on the five RVT tasks, RVT-2 improves by 8 absolute points (12.5 in relative terms).Both models use a single model across all eight tasks and 16 variations.Source: B. Real World · p. 8, Table II · p. 8

Failure Modes

• Failures on the other tasks include incorrect block placement, missing the sanitizer top, selecting the wrong marker, and grasp failures; no failures occurred on the shelf task.For marker-in-mug/bowl failures, 80% involved picking the wrong marker and 20% not going to the goal; drawer failures were wrong-marker selection (40%), grasping (40%), and not going to the goal (20%).Source: B. Real World · p. 8
• Small placement or insertion errors are the main reported failure modes for the 16mm peg, 8mm peg, and plug tasks.For 16mm pegs, small placement errors account for 75% of failures; for plugs, small socket-insertion errors account for 100%. For 8mm pegs, small placement errors account for 40%, not reaching the goal 40%, and grasp failure 20%.Source: B. Real World · p. 8

a) Multi-Stage Design

• The multi-stage design improves success rate by 17.5% but slows training because it adds a rendering and inference stage.The zoom-in view provides more task-relevant detail about the region of interest.Source: C. Ablations · p. 8

b) Parameter Rationalisation

• GPU-friendly network parameters accelerate training without compromising network performance.Source: C. Ablations · p. 8

c) Location Conditioned Rotation

• Predicting rotation from local features improves performance by 4.2%.Source: C. Ablations · p. 8

d) Fewer Virtual Views

• Reducing camera views from 5 to 3 maintains task performance and doubles training speed; RVT-2’s multi-stage network performs well with fewer virtual views per stage.Source: C. Ablations · p. 8

e) Convex Upsampling

• Removing convex upsampling increases training time by 20.7 hours; with mixed precision retained, removing it causes undefined training gradients.The ablation compares rows 6 and 7, both without mixed precision.Source: C. Ablations · p. 8

f) Point-Renderer

• Replacing PyTorch3D with the customized Point-Renderer speeds up the forward pass and yields 3.6X faster training.Source: C. Ablations · p. 8

g) Improved Training Pipeline

• Removing automatic mixed precision increases training time by 300%, while also removing 8-bit LAMB and fast attention adds a further 20%.The authors note that LAMB and fast attention offer modest gains but may help when scaling up RVT-2.Source: C. Ablations · p. 8

D. Generalization Case Study

• RVT-2 generalizes to unseen object configurations, with positions varying across a 2 feet X 2 feet workspace.Training and testing use different object configurations.Source: D. Generalization Case Study · p. 9
• With few demonstrations, RVT-2 generalizes on block stacking to unseen lighting, background appearances, and modifications to object appearance.Source: D. Generalization Case Study · p. 9
• For block stacking, RVT-2 is robust to language instructions beyond “put x block on y block,” including paraphrases and reordered relations.Tested variants include “move x block such that it is on y block,” “move x block onto y block,” “stack x block on y block,” and “move blocks such that x is under y.”Source: D. Generalization Case Study · p. 9

E. Failure Recovery Case Study

• When the target block moves midway through stacking, RVT-2 adjusts its actions, demonstrating closed-loop control at key-point frequency.The network uses the current time-step observation to predict the pose for the next time step.Source: E. Failure Recovery Case Study · p. 9
• Failure recovery appears across several simulation tasks; in one “stack two black blocks” episode, the system repeatedly retries after a block falls.The paper also presents examples for “stack blocks,” “place cups,” and “slide block to the color target.”Source: E. Failure Recovery Case Study · p. 9

• RVT-2 combines architectural and system-level improvements to RVT for faster, more precise 3D object manipulation; its contribution is their effective combination, not novel individual techniques.Source: §V · p. 9
• On RLBench, RVT-2 significantly outperforms prior methods while requiring much less compute, and it solves real-world peg and plug insertion using one third-person camera and 10 demonstrations.Source: §V · p. 9
• A 17-point overall improvement still leaves RVT-2 at 82% success in simulation and 72% in the real world, so multi-task 3D manipulation remains far from solved.Source: §V · p. 9
• RVT-2, like RVT and PerAct, works only with trained object instances, and despite success with one RGB-D sensor, can fail from minor insertion-position errors; force-guided fine motion is a proposed direction.Source: §V · p. 9
• Multi-task optimization can worsen some tasks as training progresses, as seen on open drawer, motivating strategies to prevent this degradation.Source: §V · p. 9