RynnVLA-Latent: Scaling Robotic Manipulation Foundation Models with 3D-Grounded Latent Actions

• Robot demonstrations are costly and use incompatible controls, while human and web videos offer broader interaction coverage but lack corresponding robot commands.A shared supervision interface is needed to use their visual transitions before adapting policies to particular robots.Source: §1 · pp. 1–2
• RynnLAM combines residual future-feature prediction, camera-compensated 3D-flow supervision, and camera-motion separation to learn continuous transition representations without native robot-action labels.The objectives encourage control-relevant motion, while not guaranteeing exact viewpoint invariance or physical identifiability.Source: §1 · p. 2
• The RynnLAM-labeled corpus spans 25 human, robot, and simulation video sources, totaling approximately 30.1K hours and yielding 745.52M latent-action descriptors.RynnVLA-Latent predicts latent-action chunks from multi-view observations and language through flow matching, then reuses its pretrained backbone and action-expert trunk for embodiment-specific post-training.Source: §1 · p. 2
• 98.8% success on LIBERO, 84.7% on LIBERO-Plus, 51.9% average success across five VLABench tracks, and 88.3% average success under the RoboTwin 2.0 Native protocol are reported for the post-trained 4B policy.Evaluation also includes LARY frozen-representation probes and six real-world manipulation tasks.Source: §1 · p. 2
• The paper presents representation probes, simulation and real-world evaluations, and data-scaling and component studies as validation of latent-action supervision for policy learning.Its contributions include control predictability, manipulation performance across platforms, and improvements with increasing latent-action pretraining data.Source: §1 · p. 2

2.1 Latent Action Learning

• Prior latent-action work uses transition reconstruction, feature prediction, task-centric representations, and forward-dynamics objectives to support policy learning or ground representations in robot states and actions.Examples include LAPA, Moto, UniVLA, and villa-X.Source: §2.1 · pp. 2–3
• RynnLAM combines residual future-feature prediction, camera-compensated 3D-flow supervision, and camera-motion separation to learn continuous transition representations without robot-native action or proprioceptive supervision.The objectives encourage representations of physical scene changes across human, simulated, and real-robot videos.Source: §2.1 · p. 3

2.2 Vision-Language-Action Models

• Prior VLA work includes action-token prediction, continuous action generation, cross-embodiment training, embodiment-specific decoding, and additional supervision from human videos.Examples include OpenVLA (Kim et al., 2024), π0 (Black et al., 2024), UniAct, and VITRA (Li et al., 2025).Source: §2.2 · p. 3
• RynnVLA-Latent learns RynnLAM-generated latent-action chunks from human, robot, and simulation videos, introducing native actions and proprioception only during embodiment-specific post-training.This separates shared video-based pretraining from executable control alignment; the descriptors also condition cross-embodiment video generation in RynnWorld-Latent.Source: §2.2 · p. 3

3 RynnVLA-Latent

• RynnVLA-Latent uses shared latent-action targets from heterogeneous videos for policy pretraining, then retargets the same flow-matching objective to a target robot’s action space.The framework uses costly action-labeled demonstrations only for final embodiment alignment.Source: §3 · pp. 3–4, §3.3 · p. 8

3.1 Pretraining Data

• The corpus spans 25 video sources, including human interaction, real robots, and simulation, with approximately 30.1K hours, 16.63M episodes, and 3.057B frames.Offline labeling yields 745.52M latent actions; policy-pretraining samples use chunks of six targets.Source: Figure 1 · p. 4, §3.1 · pp. 4–5

3.2 RynnLAM

• RynnLAM encodes frame transitions as motion tokens, a global action latent, and a camera latent, using residual future-feature prediction, dynamic 3D-flow supervision, and camera-pose objectives.Camera-compensated flow makes static regions approximately zero-flow, while gradient reversal encourages ego-motion to be routed separately from manipulation-relevant cues.Source: §3.1 · p. 6, §3.2 · pp. 6–7
• Contrastive feature warping brings flow-warped source features closer to target features than unwarped source features, with motion-magnitude weighting emphasizing dynamic regions.The full 608-dimensional descriptor retains the camera latent and performed better in control-predictability probes than any individual component.Source: §3.2 · p. 7

3.3 RynnVLA-Latent

• The Mixture-of-Transformers policy pairs a pretrained vision-language backbone with a narrower action expert, jointly attending across streams while keeping their output projections and feed-forward pathways separate.Post-training adds proprioceptive state and retargets a thin output projection to native robot actions, reusing the pretrained backbone and action-expert trunk without a separate decoder.Source: §3.3 · pp. 8–9, Figure 4 · p. 8
• During pretraining, the policy predicts latent-action chunks from multi-view observations and language through flow matching, without robot-action or proprioceptive supervision.Missing camera views are masked from attention and loss; the interface does not guarantee invariance to unseen viewpoints.Source: §3.3 · pp. 8–9

4 Experiments

• The evaluation spans frozen-representation control probes, simulation benchmarks, RoboTwin 2.0 Native, and real-world deployment on dual-arm platforms.The stated dimensions include manipulation success, perturbation robustness, language grounding, and bimanual manipulation.Source: §4 · p. 9

4.1 Implementation Details

• The VLM and action expert are trained with flow matching on six-target latent-action chunks conditioned on multi-view observations and language, without proprioception.Post-training retargets the objective to native robot actions and introduces proprioceptive state.Source: §4.1 · p. 10

4.2 Real-Robot System Setup

• Real-world evaluation uses stationary Marvin-WUJI with dexterous hands and mobile Astribot-S1 with parallel grippers; policies execute native action chunks in closed loop.Inference speed is not an evaluation metric.Source: §4.2 · p. 10, Figure 5 · p. 10

4.3 Latent Action Representation Quality

• LARY’s low-level control regression track uses a lightweight MLP to decode end-effector trajectories from frozen representations.The relative setting predicts displacement from the first frame to better assess motion encoding than static pose; comparisons include semantic, pixel-generative, and latent-action encoders.Source: §4.3 · p. 11, Table 3 · p. 11

4.4 Simulation Benchmarks

• The simulation evaluation covers LIBERO, LIBERO-Plus, and VLABench; RoboTwin 2.0 Native additionally evaluates bimanual manipulation.The listed benchmarks address manipulation success, perturbation robustness, and semantic grounding.Source: §4.4 · p. 11

Appendix A.

• LIBERO measures language-conditioned manipulation, LIBERO-Plus tests perturbation robustness, VLABench evaluates semantic grounding, and RoboTwin 2.0 Native assesses bimanual manipulation.The real-world evaluation covers six tasks across stationary dexterous-hand and mobile-gripper platforms.Source: Appendix A · p. 12, §4.5 · p. 12

4.6 Results

• 98.8% on LIBERO and 84.7% on LIBERO-Plus are the 4B policy’s average success rates, with the latter achieved without further benchmark-specific adaptation.The 98.8% LIBERO result is the highest reported average in its table, while the 84.7% LIBERO-Plus total is highest among listed comparisons.Source: Table 4 · p. 13, Table 5 · p. 13, §4.6 · p. 14
• 88.6% average success on RoboTwin 2.0 Native and 51.9 average success rate across five VLABench tracks are reported for the 4B policy.RoboTwin Native includes randomized demonstrations during training, so its randomized-scene score is not zero-shot transfer.Source: Table 6 · p. 14, Table 7 · p. 14, §4.6 · p. 15
• The 4B policy has the highest observed success rate among the listed methods on all six real-world tasks after robot-specific post-training.The tasks span two dual-arm platforms: stationary dexterous hands and a wheeled mobile robot with parallel grippers.Source: §4.5 · p. 12, Table 8 · p. 15, §4.6 · p. 15

4.7.1 RynnLAM

• On LARY control regression, removing end-to-end encoder training raises relative average MSE from 0.38 to 0.52, while removing 3D-flow supervision raises it to 0.50.Without 3D-flow supervision, error increases on all four datasets in both absolute and relative settings; the probe does not measure metric accuracy of recovered 3D motion.Source: §4.7.1 · pp. 16–17
• Removing the camera branch or GRL raises relative average MSE to 0.47 or 0.49, while removing motion-token residual forcing raises it to 0.57.These ablations support the components’ utility for control predictability, but do not independently establish viewpoint invariance or identify the shortcut used without residual forcing.Source: §4.7.1 · p. 17

4.7.2 RynnVLA-Latent Policy Variants

• On the shared approximately 1,000-hour subset, RynnLAM-based supervision outperforms action-alignment alternatives on LIBERO-Plus, while explicit alignment variants score higher on LIBERO.The RynnBrain backbone, camera slots, and RynnLAM targets also outperform their respective alternatives on LIBERO-Plus.Source: Table 9 · p. 15, §4.7.2 · pp. 17, 19
• Both 2B and 4B policies reach their highest LIBERO-Plus success rates with the full latent-action pretraining corpus, though the 4B trend is not strictly monotonic.The 2B model improves monotonically across evaluated data fractions; the 4B model dips slightly from 10% to 30%.Source: Figure 8 · p. 17, §4.7.2 · p. 19

5 Conclusion

• RynnLAM creates continuous transition descriptors from video without native robot-action labels, and RynnVLA-Latent learns them with flow matching before retargeting its pretrained backbone and action expert to robot commands.RynnLAM combines residual feature prediction, camera-compensated 3D-flow pseudo-labels, contrastive warping, and camera-pose decoupling.Source: §5 · p. 19
• The 4B policy scores 98.8% on LIBERO, 84.7% on LIBERO-Plus, 51.9% average success across five VLABench tracks, and 88.3% on RoboTwin 2.0 Native.It also has the best success rate among compared baselines on all six real-world tasks, while performance varies by task and RoboTwin scores remain below the strongest listed baselines.Source: §5 · p. 19
• Estimated depth, camera poses, and flow do not guarantee calibrated metric motion or complete camera/manipulation separation, and deployment still requires embodiment-specific action-labeled demonstrations.The paper also says these results establish neither causal action identifiability nor a general scaling law.Source: §5 · p. 19

A.1 Latent Action Representation Quality

• The LARY low-level control regression track probes frozen representations by decoding end-effector trajectories with a lightweight MLP and scoring MSE across four datasets.Absolute targets use a fixed base frame; relative targets express displacement from the first-frame end-effector state to emphasize displacement prediction.Source: §A.1 · p. 25
• LIBERO and LIBERO-Plus use 40 language-conditioned manipulation tasks, with LIBERO-Plus testing seven perturbation factors without additional adaptation.VLABench evaluates implicit-instruction grounding across five public tracks; RoboTwin 2.0 Native evaluates clean and randomized scenes after training on both types of demonstrations.Source: §A.1 · p. 25

B Teleoperation Setup

• Marvin-WUJI demonstrations use trackers for arm teleoperation and data gloves for dexterous-hand control.Arm tracking computes wrist-to-chest transforms for inverse kinematics, while glove signals are retargeted to the Wuji hand joint space.Source: Figure 10 · p. 26, B Teleoperation Setup · p. 26

C Details of Action-Alignment Baselines

• The action-alignment baselines map heterogeneous state and action spaces into a shared canonical vector or use embodiment-specific input and output projections.For human videos without native controls, the canonical-vector baseline uses wrist poses and five fingertip positions as pseudo-action labels.Source: C Details of Action-Alignment Baselines · p. 26, §C.1 · p. 26, §C.2 · p. 27

C.1 Shared Vector Representation

• The shared canonical representation has 85 dimensions for dual-arm embodiments, with a per-dimension mask excluding structurally absent action dimensions from the loss.Its slots cover joints, end-effector poses, grippers, dexterous hands, torso, head, and mobile base.Source: §C.1 · pp. 26–27

C.2 Domain-specific Projection

• Domain-specific projection keeps a shared action expert but selects embodiment-specific input and output projections using a domain identifier.The lightweight boundary projections capture differences in action parameterization and distribution.Source: §C.2 · p. 27

C.3 Human Egocentric Datasets

• For human egocentric data in the domain-specific projection baseline, wrist poses and five fingertip positions fill corresponding unified-vector slots despite the lack of native control labels.Source: §C.3 · p. 27

C.4 Delta Actions and Normalization

• Training action chunks express each action relative to the robot’s current state; 6-D end-effector rotation slots remain unchanged while other slots use training-data q0.01 and q0.99 quantiles for normalization.Source: §C.4 · p. 27

Video captioning

• Videos are sampled at 1 FPS and divided into 5-second segments for Qwen3-VL to caption their subjects, actions, environments, objects, and context.Captions are limited to 512 tokens, generated at temperature 0.7, and stored in JSON.Source: D RynnVLA-Latent Dataset · p. 27

Optical flow annotation

• DPFlow estimates optical flow sequentially for frame pairs at native resolution, saving color-encoded fields as 25 FPS MP4 videos.Source: D RynnVLA-Latent Dataset · p. 27

Depth annotation

• Depth Anything 3’s DA3NESTED-GIANT-LARGE-1.1 checkpoint produces per-frame dense depth and camera-pose predictions from video sampled at 30 FPS.Processing uses a 392-pixel short-side working resolution, with resizing capped at that upper bound.Source: D RynnVLA-Latent Dataset · p. 27

Training and objective variants

• The full 3D-flow model has lower average relative-action MSE than no-flow and 2D-flow variants; adding 2D flow alone does not improve over no flow.The extended variants use the same data, budget, and probe protocol as the main table.Source: E Extended Ablations of RynnLAM · p. 28, Table 10 · p. 28
• Removing future-feature prediction or compressing zm from latent tokens instead of raw target-frame features degrades probe performance, supporting the direct feature pathway.The latter variant removes the direct feature-space pathway into the bottleneck.Source: E Extended Ablations of RynnLAM · p. 28

Latent-component probes

• Motion tokens achieve 0.54 average MSE, but the full descriptor achieves 0.38; action and camera latents alone achieve 0.93 and 0.95, respectively.The comparisons use the same checkpoint and show probe utility, not complete semantic disentanglement.Source: E Extended Ablations of RynnLAM · p. 28

Post-training data

• For policy post-training, recorded robot observations and native actions are aligned in time, with action chunks expressed in the target robot’s control space.Source: F Real-World Data Collection Details · p. 28
• Operator skeletons provide the demonstration-collection control interface, but are not latent-action targets; RynnLAM extracts those targets from video.Source: F Real-World Data Collection Details · pp. 28–29

Real-world demonstrations

• The collection contains 3,850 high-quality demonstration episodes covering all six evaluated tasks across two platforms.The task-specific data distributions are detailed in Table 11.Source: F Real-World Data Collection Details · p. 29, Table 11 · p. 29