Source-linked AI summary
RoboVIP: Multi-View Video Generation with Visual Identity Prompting Augments Robot Manipulation
Boyang Wang, Haoran Zhang, Shujie Zhang, Jinkun Hao, Mingda Jia, Qi Lv, Yucheng Mao, Zhaoyang Lyu, Jia Zeng, Xudong Xu, Jiangmiao Pang
TL;DR
Manipulation data is difficult to scale, while prior visual augmentation often lacks the temporally coherent multi-view observations used by current policies. RoboVIP augments masked multi-view videos using action-guided segmentation and visual identity prompting, and reports consistent success-rate gains across simulation and real-world evaluations.
Problem
Real-world manipulation data is difficult to scale, and prior image-based augmentation often lacks the temporal and multi-view observations used by modern policies.
Source: §1 · p. 1Method
RoboVIP uses action-guided segmentation and a multi-view video diffusion model conditioned on text and exemplar-image visual identities to inpaint manipulation videos.
Source: §1 · p. 2, §3.1 · p. 3Results
Takeaways & Limitations
The authors demonstrate the practicality of RoboVIP for large-scale VLA training and low-data policy learning.
Source: §1 · p. 2Current segmentation tools struggle with gripper localization, interactive-object identification, and consistent masks; SimplerEnv supports only single-view inputs, limiting simulation evaluation of multi-view consistency training.
Source: §5 · p. 9
Abstract
from arXiv · showhide
The diversity, quantity, and quality of manipulation data are critical for training effective robot policies. However, due to hardware and physical setup constraints, collecting large-scale real-world manipulation data remains difficult to scale across diverse environments. Recent work uses text-prompt conditioned image diffusion models to augment manipulation data by altering the backgrounds and tabletop objects in the visual observations. However, these approaches often overlook the practical need for multi-view and temporally coherent observations required by state-of-the-art policy models. Further, text prompts alone cannot reliably specify the scene setup. To provide the diffusion model with explicit visual guidance, we introduce visual identity prompting, which supplies exemplar images as conditioning inputs to guide the generation of the desired scene setup. To this end, we also build a scalable pipeline to curate a visual identity pool from large robotics datasets. Using our augmented manipulation data to train downstream vision-language-action and visuomotor policy models yields consistent performance gains in both simulation and real-robot settings.
1. Introduction
Real-world manipulation data is costly to scale, while prior visual augmentation often lacks the temporal and multi-view observations modern policies use. RoboVIP combines action-guided segmentation, multi-view video generation, and visual identity prompting, and reports success-rate gains across simulation and real-world evaluations.
1. Introduction
- Real-world manipulation data is difficult to scale across quantity and environmental diversity because episodes require precise setups, calibrated cameras, and synchronized sensing.Generative models are presented as a complementary way to synthesize additional data.Source: §1 · p. 1
- Prior text-guided image augmentation typically changes backgrounds and tabletop contents in single-frame, single-view settings, unlike policies that use longer histories and multi-view observations.The passage notes that a single frame can leave a policy unable to distinguish pre- and post-interaction states in a push-button task.Source: §1 · p. 1
- RoboVIP generates multi-view videos, including dynamically moving wrist-camera views, and uses action information to guide segmentation when interacted objects are difficult to localize.The 1D gripper state identifies the time window in which the arm interacts with the target object, narrowing the search space.Source: §1 · pp. 1–2
- Visual identity prompting conditions video diffusion on exemplar images to guide semantic and low-level scene content, while an automated curation pipeline builds a million-scale identity pool from robotics datasets.The method addresses limitations of text prompts, which may be simplistic, hallucinated, misaligned, or unable to specify low-level details.Source: §1 · p. 2
- RoboVIP reports consistent success-rate gains in simulation and real-world evaluations after augmenting 12K BridgeV2 trajectories for VLA training and a 100-trajectory real-world dataset for visuomotor training.The evaluated models include π0 and Octo for VLA training and Diffusion Policy for visuomotor policy learning.Source: §1 · p. 2
2. Related Works
Prior work explores conditioned video generation and robotics visual augmentation, but multi-view, temporally coherent training data remains scarce. RoboVIP is positioned against approaches with limited conditioning, manual setup, or single-view inputs.
2.1. Conditioned Video Generation
- Video generation supports conditioned sequence synthesis and editing, but robotics visual augmentation remains underexplored, with prior approaches relying mainly on image diffusion or single-view conditioning.Identity references have also emerged as a way to inject explicit visual attributes into generation [16] [32] [47].Source: §2.1 · pp. 2–3
2.2. Visual Augmentation on Robotics
- Robotics augmentation spans manual scene construction, pixel-aligned editing, and masked-region inpainting, with each approach trading generality or semantic-content flexibility against setup effort.GreenAug [43] uses green-screen capture; ReBot [15] and RoboSplat [50] use real-to-sim-to-real workflows; Cosmos-Transfer [2] [3] and RoboTransfer [33] use pixel-aligned conditions; Rosie [52] and RoboEngine [53] inpaint masked regions.Source: §2.2 · p. 3
2.3. Manipulation Models
- Robot policies increasingly use temporal conditioning and multi-view inputs, yet suitable frame-to-frame and view-to-view aligned data remains scarce because real-world collection is slow and costly.The passage describes multi-view inputs as supporting 3D reasoning and motivates augmentation that preserves temporal and cross-view consistency.Source: §2.3 · p. 3
3. Method
RoboVIP preserves action sequences while inpainting masked multi-view videos, using action cues for segmentation and a video diffusion model for generation. Its visual identity pool and packed exemplar conditioning provide additional guidance for scene content.
3.1. Problem Formulation
- The pipeline masks the robot arm and interacted objects, preserves end-effector pose and gripper-state actions, and trains diffusion to generate latent multi-view frames conditioned on masked videos, text, and visual identities.The inpainted videos become augmented observations, while the original action sequences are reused.Source: §3.1 · p. 3
3.2. Action-guided Segmentation of Robots and Interacted Objects
- Gripper-closure intervals identify wrist-view interaction frames, where a video-reasoning VLM infers the object label; that label guides masks in other views and video segmentation tracks the robot and object.Median blurring filters outlier pixels, and k-means sampling supplies points for video segmentation.Source: §3.2 · p. 4
3.3. Multi-View Inpainting Video Diffusion Model
- The video model starts from the 14-billion-parameter Wan2.1 image-to-video model, but uses LoRA because direct fine-tuning is computationally infeasible and risks overfitting collapse.The patchification encoder is also enabled for training because the objective shifts to masked video conditioning; adding it yields slight empirical improvements.Source: §3.3 · p. 4
- Masked frames from different views are stitched vertically at matching timestamps, and target sequences are processed the same way to preserve view alignment and encourage cross-view spatial consistency.The model input uses channel-wise concatenation of the full video sequence.Source: §3.3 · p. 4, Figure 3 · p. 4
3.4. Visual Identity Prompting
- Panoptic segmentation and filtering build a million-scale identity pool, while packed references are randomly resized and sampled from one view during multi-view training to reduce overhead and view ambiguity.Filters include image quality, sharpness, CLIP text-image scoring from the class label, and resolution; packed identities are concatenated with video latents and excluded from loss computation.Source: §3.4 · pp. 4–5
4. Experiment
RoboVIP’s experiments compare multi-view video generation and downstream policy performance, including simulation and real-robot trials. The reported results include gains over prior methods on most video metrics and stronger performance in cluttered real-robot settings.
4.1. Video Diffusion Model Implementation Details
- Training discards sequences under 25 frames, crops those over 550, re-captions videos with Qwen2.5-VL, and randomly samples two BridgeV2 views while always including Droid’s wrist view.Training uses at most 49 frames because current VLA models cannot condition on longer observation histories.Source: §4.1 · pp. 5–6
4.3. Simulation Results
- 18.5% average success for Octo RoboVIP (Text+ID) exceeds zero-shot 12.2%, Bridge SFT 12.8%, and text-only 13.0%; π0 Text-only reaches 29.0%, versus SFT 17.25% and RoboEngine 18.5%.π0 Text+ID performs similarly at 27.75%.Source: Table 1 · p. 7, §4.3 · pp. 7–8
4.2. Video Generation Results
- RoboVIP outperforms prior approaches on most quantitative metrics across 300 Droid test cases, but none of the methods achieves multi-view-consistent generation.The qualitative comparisons report temporally consistent, visually diverse RoboVIP results; the authors attribute diversity beyond edge- or depth-aligned conditions to their inpainting design.Source: Table 2 · p. 7, §4.2 · p. 7
4.3. Simulation Results
- At six-frame conditioning, RoboVIP preserves meaningful success rates while RoboEngine’s average success falls to zero; RoboVIP also stays higher across the tested history lengths.The comparison retrains and tests Octo with different numbers of history frames.Source: Figure 7 · p. 8, §4.3 · p. 8
- Octo RoboVIP (Text+ID) reaches 41.1% average Put success versus 23.0% for Octo SFT; π0 Text-only reaches 55.0% versus 43.1% for SFT and 39.8% for RoboEngine.Put is conditional success after a successful grasp; the reported gains span the post-grasp phase as well as task initiation.Source: §4.3 · p. 8
4.4. Real-World Robot Results
- RoboVIP achieves 9/10 success in cluttered real-robot trials, versus 0/10 for baseline Diffusion Policy; in open space the results are 10/10 and 7/10, respectively.The RoboVIP policy was trained on 100 real demonstrations plus 100 augmented trajectories; both policies were evaluated over 10 trials.Source: Figure 8 · p. 8, §4.4 · p. 8
5. Conclusion
RoboVIP augments manipulation observations with multi-view video generation and visual identity prompting. The paper reports effectiveness across VLA and visuomotor policies in simulation and real-world deployment, while noting tool-related limitations and single-view simulation evaluation.
5. Conclusion
- RoboVIP uses multi-view inpainting video diffusion with visual identity prompting to augment manipulation observations.Its contribution also includes pipelines for multi-view robot and object segmentation and visual-identity-pool curation.Source: §5 · p. 8
- The authors report effectiveness for VLA and visuomotor policies in both simulation and real-world deployment.Source: §5 · p. 9
- Current segmentation and VLM tools have localization, flickering, interactive-object identification, and mask-consistency limitations; SimplerEnv supports only single-view inputs.The authors say more extensive simulation studies are needed to evaluate multi-view consistency training.Source: §5 · p. 9
6. Overview
The supplementary material adds technical and implementation details, further evaluations, and visual examples to complement the main manuscript.
6. Overview
- Section 7 provides implementation details on video diffusion, dataset curation, captioning, segmentation, and visual-identity filtering.Source: §6 · p. 12
- The supplement describes the simulation setup for tabletop manipulation and the real-world robot experiments.Source: §6 · p. 12
- Sections 8 and 9 add real-world baseline comparisons, a visual-identity user study, and visualizations of rollouts and generated videos.Source: §6 · p. 12
7. More Experimental Details
The supplementary section details video-generation and identity-pool curation, simulation evaluation, and real-robot experiments. Its setup includes temporal and segmentation choices, task-specific evaluation conditions, and a multi-camera real-world pipeline.
7.1. More Video Generation Details
- Scene-setup and robot-action captions are generated separately, with long Droid videos chunked to avoid describing events outside a segment.The caption prompts request brief, high-confidence descriptions; the Droid action prompt also says not to predict future events.Source: §7.1 · p. 12
- Segmentation uses parallel robot-arm and interacted-object streams, with gripper-state thresholding and a five-frame buffer around interaction events.Robot masks are selected from five sampled frames; wrist views additionally use VLM reasoning, and videos are resized to 320×448.Source: §7.1 · p. 12
- The identity pool filters selected tabletop and background classes, removes crops with over 60% masked background, and applies quality, size, sharpness, and CLIP-alignment filters.The pool removes the lowest 50% by CLIP-IQA, the smallest and largest 10% by crop resolution, the lowest 30% by sharpness, and 60% by text–image similarity.Source: §7.1 · pp. 12–13
- Video diffusion training uses up to 33 Droid frames or 49 Bridge frames, at original frame rate, and at most one packed identity frame.For RoboEngine comparisons on Droid, the authors use their segmentation masks to compare generative capability; Bridge comparisons use dataset task descriptions.Source: §7.1 · p. 13
7.2. More Simulation Setup Details
- Simulation augmentation uses a keyword-filtered set of 12k episodes and policy-specific checkpoints, while evaluation uses a single static camera because SimplerEnv lacks wrist-camera views.Octo training enables goal relabeling and task-conditioning deletion; the rollout figures show four tasks with frames progressing from episode start to end.Source: Figure 9 · p. 14, Figure 10 · p. 14, §7.2 · p. 15
- SimplerEnv is chosen for quantitative evaluation because it resembles BridgeData V2 scenes; LIBERO and ManiSkill3 were excluded after segmentation performed poorly and their textures differed from training data.The four tasks use tabletop objects and receptacles, with eggplant-in-basket serving as a novel, semantically related goal without an exact BridgeData V2 instruction or demonstration set.Source: §7.2 · pp. 13, 15
7.3. Real-World Robot Setup
- Real-robot experiments use wrist and third-person cameras, 100 teleoperated demonstrations, and 100 synthesized demonstrations per augmentation method.Videos are chunked to at most 33 frames for synthesis, with shorter chunks padded to the model’s required 4N+1 length.Source: §7.3 · p. 15
- Diffusion Policy is trained with and without augmentation and evaluated by whether a blue cube remains stably stacked on a red cube, in open or four-distractor cluttered scenes.All models share an 8-step horizon, 2 observation steps, and 4 action steps; augmented settings use 5,000 training steps versus 2,500 without augmentation.Source: §7.3 · pp. 15–16
8. Additional Results and Analysis
In real-world experiments, RoboVIP augmentation achieves the best reported Diffusion Policy performance across seen tasks and unseen environments, including a 9/10 success rate in clutter. In a user study, raters favored visual-identity conditioning for identity fidelity and richer tabletop content.
8.1. Real-World Robot Baseline Comparisons
- 9/10 success rate in the cluttered setting; RoboVIP achieves the best reported Diffusion Policy performance across seen tasks and unseen environments.The comparison averages success rates over 10 trials; the cited results also describe baseline grasp-localization and stable-placement failures in clutter.Source: §8.1 · p. 16
8.2. User Study on Visual Identity Prompting
- 97.3% of pairwise comparisons preferred visual-identity-conditioned generation for identity fidelity, and 80.0% preferred it for richer tabletop content.Three raters compared videos generated by the visual-identity-conditioned model and its text-only variant.Source: §8.2 · p. 16
9. More Visualization
The visualizations show RoboVIP-generated real-world rollouts, including a successful grasp and placement under clutter where baseline methods fail. They also illustrate scene variation, zero-shot deployment, and how rollout frames are presented.
9.1. Rollout Visualization
- In the SimplerEnv rollout strips, each row represents one of four tasks and frames run left-to-right from episode start to finish.Frames are sampled uniformly and show the object moving from its initial pose toward the target configuration.Source: §9.1 · p. 16
9.2. Real-World Robot Augmented Videos
- Real-world rollouts are described as genuine zero-shot cases because the lab wrist-camera view is left-shifted, unlike Droid’s centered, symmetric gripper views.The generated videos are used to train Diffusion Policy.Source: §9.2 · p. 16
- Zero-shot generations show changing tabletop textures, different third-person backgrounds, and chunk-to-chunk table geometry, with diverse and realistic objects.The passage attributes the enriched tabletop scenes to visual identity prompting.Source: §9.2 · p. 16
- Under the cluttered setup, baseline methods [3, 12, 53] fail during grasping, while RoboVIP achieves a successful grasp and final placement.Figure 11 shows the representative real-world robot rollout.Source: Figure 11 · p. 17
- For long-horizon real-world videos, RoboVIP generates sequential 33-frame chunks with different visual identities and shows equally sampled frames across the trajectory.The chunking addresses the number of frames in real robot videos exceeding what current video diffusion models can process directly.Source: Figure 12 · p. 18