RynnVLA-001: Using Human Demonstrations to Improve Robot Manipulation

• Robot-manipulation data are scarce and costly to collect because large-scale datasets typically require human teleoperation on physical robots.Prior dataset-building efforts remain much smaller than datasets for language, vision-language, and generative models, while other work transfers knowledge from pretrained generative models or VLMs.Source: §1 · p. 1
• Ego-Centric Video Generative Pretraining learns future-frame prediction from 12M curated first-person human manipulation videos conditioned on an image and language instruction.The videos emphasize hand operations; the model predicts subsequent frames from a single image and instruction.Source: §1 · pp. 1–2
• Human-Centric Trajectory-Aware Video Modeling adds future human-keypoint trajectory prediction to frame prediction, connecting visual dynamics with low-level robot actions.The trajectories share similarities with robot actions and are intended to facilitate transfer to robot manipulation.Source: §1 · p. 2
• ActionVAE compresses action chunks into compact embeddings that the model predicts and decodes into executable robot-action sequences.The robot model is trained to predict action chunks conditioned on RGB observations and language instructions.Source: §1 · p. 2
• RynnVLA-001 achieves higher success rates than GR00T N1.5 and Pi0 when all are finetuned on the same robot manipulation data.The authors describe this as evidence that their pretraining framework provides a more effective initialization for VLA modeling.Source: §1 · p. 2

2.1 Vision-Language-Action Models

• VLA approaches discretize continuous actions into categorical bins compatible with standard LLM backbones, but often lose precision in action representations; Pi0 uses conditional flow matching for continuous control, and GR00T is an end-to-end trained framework.OpenVLA improved generalization through extensive pretraining. LCB pioneered a dual-system architecture combining high-level reasoning with a specialized policy head for continuous low-level actions.Source: §2.1 · p. 3

2.2 Future Prediction for Robot Learning

• Future prediction supports generated visual goals, joint future-image and action generation, or video-based representation pretraining; RynnVLA-001 advances the pretraining approach with a trajectory-aware stage.Its three-stage architecture progresses from future-frame prediction to trajectory prediction and robot-centric action embeddings.Source: §2.2 · p. 3, Figure 2 · p. 4

3 Methodology

• The three-stage methodology trains an ego-centric video model, adds human keypoint-trajectory prediction, then transfers pretrained weights to robot-centric VLA modeling.The final stage predicts action embeddings decoded by ActionVAE into executable actions.Source: §3 · p. 4

3.1 Ego-Centric Video Generative Pretraining

• Ego-Centric Video Generative Pretraining trains an autoregressive Image-to-Video model to predict future frames from an initial observation and language description.Training uses 12M curated ego-centric human manipulation videos, along with 244K filtered robotic manipulation videos.Source: §3.1 · pp. 4–5

3.2 Human-Centric Trajectory-Aware Video Modeling

• Human-Centric Trajectory-Aware Video Modeling jointly predicts future frames and wrist-trajectory chunks, with current wrist positions supplied as state embeddings.Wrist keypoints approximate end-effector positions; trajectory chunks are represented using a pretrained ActionVAE.Source: §3.2 · p. 5

3.3 ActionVAE: Action Representaton via VAE

• ActionVAE encodes action chunks into compact continuous embeddings and decodes them into action sequences, supporting smooth, coherent chunk-level prediction.Separate embodiment-specific VAEs are trained for human trajectories and robot actions.Source: §3.3 · pp. 5–6

3.4 Robot-Centric Vision-Language Action Modeling

• Robot-centric modeling predicts the next robot-action-chunk embedding from language, two camera views, and robot state, while retaining future-frame prediction as an auxiliary objective.The Stage 2 action head is replaced because human hand trajectories differ from robot-arm kinematics; the robot-specific ActionVAE decodes predicted embeddings.Source: §3.4 · p. 6

3.5 Inference

• At inference, the model repeatedly observes the robot and instruction, predicts an action embedding, and executes its decoded action chunk; future-vision generation is skipped for speed.The paper also reports that embeddings from ActionVAE supported downstream VLA training on tasks absent from its training data.Source: §3.5 · pp. 6–7

• For pretraining, the authors construct a large-scale dataset of ego-centric manipulation videos from web sources; because raw videos are noisy and highly diverse, a multi-stage pipeline filters and annotates them.Source: §4 · p. 7
• Keypoint-based filtering discards videos with facial landmarks and retains frames where wrist and hand keypoints are visible.The pipeline extracts facial, torso, and hand-related keypoints before filtering.Source: §4 · p. 7
• Qwen2-VL-7B generates concise descriptions for curated videos, providing text-video supervision for vision-action alignment.Descriptions are kept short to resemble robot-learning instructions.Source: §4 · p. 7

5.1 Experimental Setup

• The SO100 experiments use expert teleoperation across block placement, strawberry picking, and pen insertion, with shared-data finetuning and multiple evaluation settings.Data vary in distractors, arms, and lighting; evaluation includes single-target, multi-target, and distractor scenarios, with success defined by placing at least one target within the time limit.Source: §5.1 · pp. 7–8

5.2 Comparison with SoTA methods

• RynnVLA-001 outperforms GR00T N1.5 and Pi0 across all three tasks in overall success rates, while its SR@1 is comparable to Pi0.Table 1 reports task-specific and average success rates, with each number averaged across the three settings in Table 2.Source: §5.2 · p. 8, Table 1 · p. 8
• As scenes become more challenging, RynnVLA-001 remains stable, while GR00T N1.5 declines in multi-target and distractor settings and Pi0 drops with distractors.The evaluation distinguishes single-target, multi-target, and instruction-following with distractors.Source: Table 2 · p. 8, §5.2 · p. 9

5.3 Effectiveness of Pretraining

• Ego-centric video pretraining improves over Chameleon initialization, which reaches only 50.0% success, while the scratch model performs extremely poorly.RynnVLA-001-Video starts from Chameleon weights and is further pretrained on ego-centric videos; the passage attributes its improvement to priors learned from those videos.Source: §5.3 · p. 9
• Adding human trajectory prediction to the video-pretrained model yields RynnVLA-001’s best performance among the tested variants.The study presents this stage as an explicit bridge between visual prediction and action generation.Source: §5.3 · p. 9

5.4 Ablation Study on Model Designs

• Reducing resolution from 384×384 to 256×256 substantially degrades performance, which the authors attribute to poorer VQGAN reconstructions and imprecise visual tokens.They choose 384×384 as a balance between reconstruction fidelity and computational overhead relative to 512×512.Source: §5.4 · p. 10
• On Calvin ABC->D, VAE latent action prediction outperforms raw-action prediction, while a five-layer MLP head lowers the score from 4.019 to 3.323.The VAE compresses action sequences and its latent structure improves temporal consistency; the authors report a linear head is sufficient and the deeper head may introduce noise or overfitting.Source: §5.4 · p. 10

5.5 Further Analysis

• The pretrained I2V model generates plausible motion with consistent content, though frame-to-frame changes can be subtle.The authors consider this sufficient for its role as a backbone for subsequent VLA training.Source: Figure 4 · p. 11, §5.5 · p. 12

• RynnVLA-001 outperforms state-of-the-art models including GR00T N1.5 and Pi0.Source: §6 · p. 13
• Two pretraining stages learn manipulation dynamics through next-frame prediction and human keypoint trajectories, while ActionVAE embeds action chunks compactly.Source: §6 · p. 13
• Evaluation is limited to one robot embodiment, an environment closely mirroring training, and a fixed front-facing camera.Source: §6 · p. 13