PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboCurate: Harnessing Diversity with Action-Verified Neural Trajectory for Robot Learning

Seungku Kim, Suhyeok Jang, Byungjun Yoon, Dongyoung Kim, John Won, Jinwoo Shin

arXiv:2602.18742v1cs.ROcs.AIcs.CV

TL;DR

Synthetic neural trajectories offer scalable robot-learning data but can contain physically implausible videos and incorrect action labels that existing VLM validation does not directly evaluate. RoboCurate verifies predicted actions through simulator-replay motion consistency and diversifies scenes and appearances with I2I and action-preserving V2V transfer, yielding gains across simulation and real-world benchmarks. The framework reports +70.1% on GR-1 Tabletop, +16.1% on DexMimicGen, and +179.9% on real-world ALLEX relative to real-data-only training.

  • Problem

    Neural trajectories can contain physically implausible videos and low-quality action annotations, while VLM-based validation does not directly evaluate generated actions.

  • Method

    RoboCurate verifies IDM-predicted actions by comparing generated-video motion with simulator replay and expands diversity through I2I editing and action-preserving V2V transfer.

  • Results

    +70.1% on GR-1 Tabletop (300 demos), +16.1% on DexMimicGen, and +179.9% relative improvement in success rate on real-world ALLEX versus real-data-only training.

  • Takeaways & Limitations

    RoboCurate's action-consistency measure supports filtering training data and Best-of-N sampling, boosting performance across simulation and real-world benchmarks.

  • Takeaways & Limitations

    Synthetic trajectories use zero-padded proprioceptive states because inverse dynamics models do not predict state information, and alignment with social values requires care.

Abstract

from arXiv · show

Synthetic data generated by video generative models has shown promise for robot learning as a scalable pipeline, but it often suffers from inconsistent action quality due to imperfectly generated videos. Recently, vision-language models (VLMs) have been leveraged to validate video quality, but they have limitations in distinguishing physically accurate videos and, even then, cannot directly evaluate the generated actions themselves. To tackle this issue, we introduce RoboCurate, a novel synthetic robot data generation framework that evaluates and filters the quality of annotated actions by comparing them with simulation replay. Specifically, RoboCurate replays the predicted actions in a simulator and assesses action quality by measuring the consistency of motion between the simulator rollout and the generated video. In addition, we unlock observation diversity beyond the available dataset via image-to-image editing and apply action-preserving video-to-video transfer to further augment appearance. We observe RoboCurate's generated data yield substantial relative improvements in success rates compared to using real data only, achieving +70.1% on GR-1 Tabletop (300 demos), +16.1% on DexMimicGen in the pre-training setup, and +179.9% in the challenging real-world ALLEX humanoid dexterous manipulation setting.

1. Introduction

RoboCurate addresses unreliable action annotations in neural robot trajectories by combining simulator-replay verification with controllable visual diversification. Its curated synthetic data improves policy performance across simulation and real-world evaluations.

  • Robotics datasets remain limited by costly collection, while simulation faces visual discrepancies, sim-to-real gaps, and substantial engineering requirements for diverse data.
  • Neural trajectory pipelines can produce instruction-inconsistent or physically implausible videos, and learned action labelers can generate low-quality annotations.
  • RoboCurate replays predicted actions in simulation and compares simulator-rollout motion with generated-video motion to evaluate visual quality and action-label correctness.
  • Image-to-image editing expands scene diversity, while action-preserving video-to-video transfer augments appearance while preserving motion dynamics.
  • +70.1% relative improvement on GR-1 Tabletop (300 demos), +16.1% on DexMimicGen, and +179.9% on real-world ALLEX compared with real-data-only training.The evaluations span pre-training benchmarks and co-finetuning on a real humanoid platform.
  • +162.3% relative improvement on novel object pick-and-place tasks and emergent success on novel action tasks from 0.0% to 25.0%.

2. Preliminaries

The preliminaries describe video and action-generation components used to construct neural trajectories and train policies. Video diffusion models generate videos in latent space, while inverse dynamics models infer action sequences from observations.

  • Video generative models encode pixel-space videos into latent representations and learn their distribution with flow matching.
  • The video diffusion model predicts the velocity field ϵ − x and is trained by minimizing mean squared error.
  • Inverse dynamics models predict intermediate actions between current and future observations over a fixed action horizon.
  • Applying inverse dynamics models to synthetic videos converts action-free video into paired videos and pseudo-labeled action sequences for policy training.
  • Policies are trained with imitation learning on the union of limited real-world data and neural trajectory data.
  • The policy receives observations, task instructions, and proprioceptive states, then outputs an action chunk of length H.
  • Synthetic trajectories use zero-padded proprioceptive states because inverse dynamics models do not predict state information.

3. Method

RoboCurate generates diverse neural trajectories and filters them by testing whether IDM-predicted actions produce simulator rollouts with motion consistent with generated videos.

  • Generating Plausible Manipulation Scenarios: RoboCurate expands neural-trajectory diversity through I2I scene editing, V2V appearance transfer, and VLM-generated task instructions.I2I varies initial scenes, while V2V preserves motion dynamics and reuses IDM action annotations.
  • Action-level Filtering of Neural Trajectory: The framework represents each sample as a generated video paired with IDM-predicted actions, then verifies action quality through simulator replay.Simulator rollouts provide proxy videos whose actions are guaranteed to correspond to the replayed motion.
  • Action-level Filtering of Neural Trajectory: Positive probe pairs match real demonstration clips with simulator rollouts, while negatives use temporal shifts or cross-episode pairings.The negative construction targets subtle motion mismatches without using potentially noisy neural-trajectory labels for probe supervision.
  • Action-level Filtering of Neural Trajectory: An attentive probe concatenates frozen video-encoder embeddings, predicts alignment with cross-attention, and is trained using binary cross-entropy.The probe uses a learnable query token and a linear head to output an alignment logit.
  • Improve Neural Trajectory via Best-of-N Sampling: At inference, RoboCurate retains a neural-trajectory sample only when its alignment probability exceeds threshold c.The same score can select the highest-scoring video-action pair among N candidates through Best-of-N sampling, improving quality without discarding samples.
  • Experimental Design: Experiments compare two regimes: pre-training on real data and neural trajectory followed by simulation fine-tuning, or co-finetuning on real data and neural trajectory.These regimes are summarized in the experimental design overview.

4. Experiments

RoboCurate is evaluated across pre-training, co-finetuning, multiple embodiments, and filtering and diversity analyses. Results assess synthetic-data gains, action-level verification, and visual and task diversity.

  • Experiments evaluate RoboCurate in two training regimes: pre-training followed by fine-tuning, and co-finetuning on real data and neural trajectory.
  • The benchmark suite covers GR-1 Tabletop, DexMimicGen with GR-1 humanoid and bimanual Panda arms, and real-robot ALLEX tasks.GR-1 Tabletop includes 24 tasks; DexMimicGen includes six tasks; ALLEX includes three tabletop manipulation tasks.
  • 4.1. Pre-training Experiments: Pre-training with RoboCurate’s synthetic data consistently outperforms baselines, with visual augmentation and action-level filtering each further improving downstream VLA performance.The framework uses I2I editing and V2V transfer for visual augmentation, while filtering can improve performance with less data.
  • 4.1. Pre-training Experiments: RoboCurate’s neural trajectory prior transfers across embodiments, including from GR-1 humanoid data to bimanual Panda arms with dexterous hands.
  • 4.2. Co-finetuning Experiments: On ALLEX, co-finetuning with neural trajectory enables OOD behavior without real-world data for those tasks, increasing pour-can success from 0% to 12.5%.Action-level filtering through Best-of-N sampling provides an additional ∼40% relative gain over unfiltered neural trajectory.
  • 4.3. Ablation Studies and Analyses: Action-level filtering outperforms video-only physical-plausibility filtering, while attentive-probe motion-consistency classification outperforms embedding-similarity and human-label baselines.The attentive probe detects subtle motion discrepancies between generated videos and simulator replays; task and visual diversity also improve VLA performance.

5. Related Work

Prior work uses video generative models and simulation to scale robot-policy data, but each approach has important limitations. Video models generate novel trajectories, while simulation faces realism and transfer challenges.

  • Video generative models provide novel scenes or motions for robot policy learning, sometimes augmented with simple tracking signals.
  • Text-to-video diffusion models can generate synthetic trajectories and use inverse dynamics models to infer pseudo-actions.
  • Simulation enables scalable data collection but suffers from the sim-to-real gap, inaccurate physical modeling, and difficulties with complex interactions.

6. Conclusion

RoboCurate improves synthetic robot data by verifying predicted actions through simulator-replay consistency and increasing visual diversity with image and video editing. Its results indicate that visual diversity and action-consistency filtering benefit downstream policy learning.

  • RoboCurate verifies IDM-predicted actions through simulator-replay consistency and expands observation diversity using I2I editing and action-preserving V2V transfer.
  • Increasing neural-trajectory visual diversity is identified as a key driver of downstream policy performance.
  • The I2I and V2V pipeline produces more beneficial synthetic data than naive I2V generation.
  • Action consistency supports both training-data filtering and Best-of-N video-generation criticism, consistently boosting performance across simulation and real-world benchmarks.

Impact Statement

RoboCurate uses synthetic robot data to expand robot-policy capabilities, while emphasizing the need to consider whether synthetic data and resulting policies align with social values.

  • RoboCurate is intended to expand robot-policy capabilities using synthetic robot data, but alignment with social values requires careful consideration.

A.1. Pre-training Experiments

The pre-training experiments use a WSD learning-rate schedule over 60K gradient steps and maintain a high learning rate before decay. Fine-tuning starts from setup-specific pretrained checkpoints and follows task-adjusted training configurations.

  • Pre-training runs for 60K gradient steps, with a constant high learning rate for 50K steps followed by sharp decay over the final 10K steps.
  • Fine-tuning initializes GR00T N1.5 from the pretrained checkpoint for each setup and tunes gradient steps by downstream task dataset.

A.2. Co-finetuning Experiments

The co-finetuning setup follows GR00T N1.5 while mixing real and neural data across three manipulation tasks.

  • The model is fine-tuned from a pretrained GR00T N1.5 checkpoint for 10K gradient steps.
  • The setup uses a global batch size of 128 while otherwise following the original GR00T N1.5 fine-tuning configuration.

A.3. Attentive Probe Training

An attentive probe is trained on a frozen pretrained V-JEPA2 large model, with its training configuration summarized in Table 7.

  • An attentive probe is attached to a pretrained 0.3B V-JEPA2 large model with the backbone frozen during training.
  • The probe follows the V-JEPA2 implementation and training strategy.
  • Table 7 reports the attentive probe training hyperparameters.

B. Neural Trajectory Generation

Neural trajectory generation combines video synthesis, visual augmentation, instruction and plausibility post-processing, and pseudo-action labeling.

  • Cosmos-Predict2-14B is fine-tuned as the base video generative model on ActionNet, GR00T-GR1-1001, and manually collected ALLEX data.
  • Approximately 10K tabletop manipulation videos are generated, while co-finetuning produces 48/50/50 videos for one ID task and two OOD tasks.
  • Image editing with FLUX.2-dev and video-to-video transfer with Cosmos-Transfer2.5-2B augment visual appearance, using an I2I:V2V ratio of 2:1.
  • Post-processing filters implausible task instructions, checks instruction following, and recaptions videos that do not match their instructions.
  • A diffusion-transformer IDM with a SigLIP-2 vision encoder provides pseudo-action labels for generated trajectories.

C. Prompt for VLM

The prompts generate diverse, actionable robot instructions, verify instruction following and physical plausibility, and control visual variation while preserving scene structure.

  • C.1. Task Instruction for Image-to-Video Generative Model: Task-instruction prompts require exactly five novel, realistic manipulation instructions grounded in visible objects, positions, colors, and spatial relationships.Each instruction represents an independent primitive task feasible within 10 seconds.
  • C.1. Task Instruction for Image-to-Video Generative Model: The prompts assign a specific hand to each instruction and enforce a fixed sentence format.
  • C.2. Post-processing for Neural Trajectory: Instruction-following prompts compare the video frames with the original instruction and return a binary YES or NO judgment.
  • C.2. Post-processing for Neural Trajectory: Physical-plausibility prompts evaluate approach, grasp contact, and transport using depth- and geometry-consistent hand motion.
  • C.2. Post-processing for Neural Trajectory: A 1–5 plausibility scale distinguishes fully consistent trajectories from those with minor, clear, or major physical violations.
  • C.3. System Prompts for Image-to-Image Editing: The image-editing pipeline uses scene descriptions and reference instructions to generate structured variations while preserving the underlying manipulation setup and scene structure.
  • C.3. System Prompts for Image-to-Image Editing: Variation prompts support original-object attribute changes and completely different object types, with varied materials, patterns, textures, and finishes.
Loading 2602.18742v1…