PaperlayerView exact revision on arXiv

Source-linked AI summary

Native Video-Action Pretraining for Generalizable Robot Control

Qihang Zhang, Lin Li, Luyao Zhang, Shuai Yang, Yiming Luo, Shuaiting Li, Ruilin Wang, Junke Wang, Jiahao Shao, Gangwei Xu, Jiaming Zhou, Yishu Shen, Yudong Jin, Fangyi Xu, Shuailei Ma, Jiaqi Liao, Guanxing Lu, Zifan Shi, Yongkun Wen, Yujie Zhao, Weixuan Tang, Xinyang Wang, Chaojian Li, Jiapeng Zhu, Ka Leong Cheng, Nan Xue, Xing Zhu, Yujun Shen, Yinghao Xu

arXiv:2607.08639v2cs.ROcs.CV

TL;DR

Generic video-generation components are poorly matched to physical robot control because their representations, temporal modeling, and inference are not designed for embodiment. LingBot-VA 2.0 natively pretrains a semantic causal video-action model with efficient asynchronous inference, achieving few-shot generalization and real-time closed-loop manipulation across simulation and real-world evaluations.

  • Problem

    Generic video-generation components use reconstruction-oriented representations and bidirectional backbones, creating semantic, temporal, and inference mismatches for closed-loop robot control.

  • Method

    LingBot-VA 2.0 natively pretrains a semantic visual-action tokenizer and causal DiT, using sparse MoE inference and asynchronous re-grounded prediction for robot control.

  • Results

    Across simulation and real-world evaluations, LingBot-VA 2.0 improves over strong baselines, adapts from 10–15 demonstrations, and reaches 225 Hz peak asynchronous execution.

  • Takeaways & Limitations

    The resulting representation supports few-shot and zero-shot generalization, real-time closed-loop execution, and long-horizon high-precision manipulation.

  • Takeaways & Limitations

    The planner and policy are trained separately, latent actions come from passive video, and evaluation coverage remains focused on bimanual manipulation.

Abstract

from arXiv · show

The advent of video-action models offers a promising path for robot control. Nevertheless, we argue that repurposing video generative models designed for digital content creation is inherently inadequate for physical environments. To bridge this gap, we present LingBot-VA 2.0, a video-action foundation model built from the ground up for embodiment. Four core design principles showcase its evolution from LingBot-VA. (1) Departing from traditional reconstruction-focused VAEs, we introduce a semantic visual-action tokenizer, which aligns visual representations with both semantics and actions, improving instruction following and action precision in subsequent policy learning. (2) Given the strictly causal nature of temporal dynamics, we adopt a causal pretraining paradigm, training from scratch to circumvent the catastrophic forgetting that frequently occurs when adapting bidirectional architectures. (3) To meet the demands of high-frequency inference, our model employs a sparse MoE backbone, expanding model capacity without compromising efficiency. (4) Real-time closed-loop control is realized through an enhanced asynchronous inference scheme, which predicts future latents in parallel with action execution while re-grounding each rollout on the latest observation via learned forward dynamics. Real-world deployment validates LingBot-VA 2.0 as a robust foundation model, as evidenced by its few-shot generalization across complex manipulation tasks.

1 Introduction

LingBot-VA 2.0 addresses limitations of repurposing generic video generators for robot control through native semantic, causal, and efficient video-action pretraining. Its design supports few-shot generalization and real-time closed-loop manipulation.

  • Motivation: Generic video-action models inherit reconstruction-focused representations, slow inference, and bidirectional temporal processing that mismatch closed-loop robot control.Their pixel-oriented latents provide limited semantic and physical structure, while bidirectional pretraining conflicts with strictly forward control.
  • Core approach: Native pretraining aligns world states and latent actions in a shared semantic space while learning causal dynamics directly for robot control.The model uses a semantic visual-action tokenizer and causal DiT pretrained on web-scale image and video data.
  • Core approach: Asynchronous inference predicts future visual latents during action execution and re-grounds each rollout on the latest real observation.This addresses the serial bottleneck between world-model updates and robot control.
  • Results: 225 Hz peak asynchronous execution supports real-time closed-loop control on real robots.Sparse MoE inference, few-step consistency distillation, Foresight Reasoning, and quantized execution enable this deployment frequency.
  • Results: LingBot-VA 2.0 adapts from only 10–15 demonstrations, transfers across embodiments, and improves over strong baselines across simulation and real-world evaluations.The paper also reports zero-shot execution in some settings and more consistent long-horizon, high-precision manipulation.

2 Native Video-Action Model

LingBot-VA 2.0 builds its robot-control stack natively in two stages: a semantic visual-action tokenizer followed by a causal video-action model. A planner, MoE video stream, action model, and asynchronous rollout complete the stack.

  • Semantic visual-action tokenizer: The first stage aligns visual latents with a frozen foundation model and extracts latent actions from unlabeled video.This produces a shared visual-action latent space.
  • Causal video-action model: The second stage pretrains a causal video-action model on the shared latent space.The stack includes a high-level VLM planner, causal video and action components, and mechanisms for real-time execution.

2.1 Preliminary

The preliminary formulation describes how generic video generators compress and synthesize video, then explains how prior video-action methods adapt them for robotics. Two key modifications are causal attention and action injection.

  • Video generation: A VAE compresses raw video into compact latents and reconstructs it, with training focused on video compression.The latent sequence uses temporal and spatial downsampling factors f_t and f_s.
  • Video generation: The video generator synthesizes latent video from noise conditioned on a text prompt using flow matching.Its attention is bidirectional over all latent frames.
  • Robot-control adaptation: Prior video-action methods map temporally downsampled visual transitions to action chunks spanning f_t low-level actions.Each action chunk is aligned with a transition between consecutive visual latents.
  • Robot-control adaptation: Video causality replaces bidirectional attention with a causal mask so each latent frame depends only on the past.This matches closed-loop control, where the present cannot attend to the future.
  • Robot-control adaptation: Action injection extends video prediction to a conditional distribution over future visual latents and actions given initial visual context.Existing methods parameterize this objective through joint video-action prediction or separate video and action flow-matching losses.

2.2 Semantic Visual-Action Tokenizer

LingBot-VA 2.0 replaces compression-only visual tokenization with a semantic visual-action tokenizer that aligns visual latents with foundation-model features and learns compact action transitions. The resulting paired visual-action latents support video-action modeling of semantically meaningful, control-relevant dynamics.

  • Visual Tokenization: The tokenizer augments reconstruction with semantic alignment to foundation-model features and latent-action learning from consecutive visual latents.A frozen Perception Encoder supplies semantic targets, while latent actions are learned without action labels.
  • Visual Tokenization: Video clips are encoded into visual latents using spatial patches for the first frame and spatiotemporal tubelets for subsequent frames, with causal attention across frames.A symmetric decoder reconstructs the video from the encoded latents.
  • Visual Tokenization: Semantic alignment matches temporally pooled tokenizer representations to frozen teacher features while preserving per-frame information for reconstruction.A learnable projection maps tokenizer latents to the teacher feature dimension, and λ_align controls alignment strength.
  • Latent Action Tokenization: The latent action tokenizer uses inverse dynamics to infer compact transition variables and forward dynamics to reconstruct subsequent visual latents.Forward prediction and backward consistency are trained on unlabeled video.
  • Latent Action Tokenization: A bottleneck prevents latent actions from copying the full visual state, encouraging control-relevant changes and producing paired visual-action training targets.The resulting targets are visual latents z_0:N and latent actions ℓ_0:N−1, later written as a_t ≡ ℓ_t.

2.3 Video-action Model Pretraining and Inference

LingBot-VA 2.0 pretrains a hierarchical causal video-action model in which a VLM planner provides structured subtask context and a low-level policy predicts future visual latents and actions. Its training adds multi-chunk temporal supervision, video in-context demonstrations, cross-embodiment action alignment, and asynchronous closed-loop inference.

  • Hierarchical planning: The model separates long-horizon planning from low-level control: a VLM decomposes goals into subtasks, while the policy predicts future visual latents and latent actions at chunk granularity.The planner tracks progress and supplies structured context for each subtask.
  • Hierarchical planning: The planner runs asynchronously at approximately 2 Hz and writes structured context to a shared buffer that the policy reads at action-chunk boundaries.This decoupling prevents planner inference latency from blocking policy execution.
  • Hierarchical planning: Boundary-crossing samples train the planner both to echo the active subtask and to predict the next subtask after a segment transition.Inputs include temporally spaced keyframes, the task goal, completed-segment history, and the current subtask when applicable.
  • Multi-chunk prediction: MCP supervises representations to predict several future latent chunks, addressing the myopic tendency to copy short-term visual continuity rather than model physical state transitions.The representation must encode scene evolution across the next K chunks, not only the immediate next chunk.
  • In-context learning: Video in-context learning pairs robot videos with semantically aligned human demonstrations that provide explicit temporal and procedural guidance beyond language-only instructions.The human and robot videos need not share object instances, viewpoints, or scene layouts.
  • Cross-embodiment co-training: Human–robot co-training retargets hand poses into the robot action space because action-space and motion gaps can otherwise corrupt shared dynamics.The unified representation uses the same end-effector action layout while domain-specific heads absorb remaining motion differences.
  • Asynchronous inference: Foresight Reasoning predicts future states and actions while the robot executes, then re-grounds the rollout on real observations and executed actions to preserve closed-loop control.The forward-dynamics grounding objective conditions on the executed action, unlike the ordinary video loss that uses only past actions.

2.4 Post-training

Post-training makes LingBot-VA 2.0 practical for real-time deployment by distilling both experts and optimizing computation across model, sequence, and runtime levels.

  • Consistency distillation: Consistency models map points along probability-flow ODE trajectories directly to clean endpoints, enabling few-step sampling.The student is trained for consistent predictions at adjacent trajectory points using an exponential-moving-average target network.
  • Consistency distillation: 5 and 10 denoising steps are reduced to 2 steps each through consistency distillation for the video and action experts.The deployed policy produces every chunk in two function evaluations.
  • Inference acceleration: Inference acceleration targets transformer forward computation, long-horizon attention, and repeated runtime overhead.The three levels respectively optimize DiT execution, KV-cache handling, and host-side preparation.

3 Data Recipe

The data recipe combines broad web-scale priors, expanded and relabeled robot trajectories, and human manipulation videos converted into robot-compatible supervision and in-context demonstrations.

  • Curriculum: Training follows a curriculum from general text-to-image and text-to-video pretraining to video-action adaptation on embodied data.The image and video stages reuse LingBot-Video corpora to provide broad appearance and dynamics priors.
  • Robot data: The robot stage retains diverse public, semi-public, and internal sources while adding thousands of hours of internal demonstrations.Sources include AgiBot, RoboMind, InternData-A1, OXE, UMI-style datasets, and RoboCOIN.
  • Robot data: A unified annotation pipeline segments trajectories into atomic clips and assigns language prompts plus global task instructions.Qwen3.5-397B relabeling repairs missing or generic prompts and standardizes supervision across embodiments and datasets.
  • Human data: The human manipulation corpus contains thousands of hours of egocentric video across 65.4k episodes, five tabletop environments, more than 600 operators, and over 3.0k scene-task combinations.Hand poses are converted into robot-compatible action representations by preserving root trajectories and mapping finger joints to gripper apertures.
  • In-context learning data: For in-context learning, robot videos are sampled by task taxonomy, transformed into semantically aligned human demonstrations with a VLM, and filtered before pairing.The resulting human videos provide visual task context for robot trajectories.

4 Experiments

The model is a from-scratch causal diffusion-transformer stack with separate video and action processing, chunked generation, multi-chunk prediction, and sparse capacity scaling.

  • Architecture: The backbone is a from-scratch DiT operating on 96-channel semantic visual-action latents, with 30 video transformer blocks and 2048-dimensional video hidden states.Its joint attention space is 3072-dimensional with 24 heads of dimension 128.
  • Architecture: Action tokens use a unified 30-dimensional action space and a parallel narrower expert stream that shares text key/value projections with video.Video uses sparse MoE components while actions use a dense feed-forward pathway.
  • Generation: Causal chunked generation uses block-causal attention with a sliding history window, supporting variable chunk sizes and history lengths.Evaluation uses chunk size 2 and a 64-chunk attention window.
  • Multi-Chunk Prediction: Multi-Chunk Prediction adds auxiliary heads that predict 1–3 future chunks beyond the next chunk.The objective requires representations to encode how the scene evolves across multiple future chunks.
  • Training objective: All stages use rectified-flow flow matching with modality-specific timestep shifts and independently sampled video and action timesteps.The action modality uses an unshifted timestep, while robot video–action data use shift 5 and the MCP branch uses shift 10.

4.2 Real-world Deployment

Experiments evaluate a single generalist policy on real-world manipulation and RoboTwin simulation, including baseline comparisons, in-context transfer, tokenizer ablations, and multi-chunk prediction.

  • Real-world evaluation: The real-world benchmark covers Fruit Sorting, Pen Collection, Drawer Tidying, and Plate Handover.The study reports per-task success and average task progress using one multi-task checkpoint trained from 20 teleoperated demonstrations per task.
  • Real-world evaluation: LingBot-VA 2.0 achieves the strongest overall real-world performance, improving success rate and task progress over π0.5 and LingBot-VA.The reported advantage is especially clear on continuous, longer-horizon manipulation tasks.
  • In-context learning: In-context evaluation tests four unseen task compositions using a human reference video and current robot observation without parameter updates.The study probes transfer across new compositions, object arrangements, and object instances.
  • Simulation: RoboTwin evaluation uses clean and domain-randomized bimanual manipulation settings with 2,500 clean-scene and 25,000 randomized-scene demonstrations.All models are trained in a multi-task setup and evaluated by task success rate.
  • Simulation: 93.6% average success rate is achieved by LingBot-VA 2.0 on RoboTwin, versus 92.2% for LingBot-VA and 14.0 percentage points above π0.5.The clean-to-randomized gap is 0.6 percentage points, indicating robust control under domain variation.
  • Ablations: Tokenizer ablations compare visual tokenizers using success rate on Easy and Hard splits over 50 RoboTwin tasks.The figure caption defines the comparison scope but does not provide the individual ablation values.
  • Ablations: MCP converges faster and reaches higher final accuracy than the baseline at both 12 and 50 fps.The comparison is reported as a RoboTwin multi-chunk prediction ablation.

4.4 Ablation

The ablations evaluate tokenizer quality, multi-chunk prediction, inference acceleration, and real-world demonstrations across complementary control settings.

  • Semantic Visual-Action Tokenizer: Replacing the reconstruction-oriented VAE with the semantic visual-action tokenizer improves downstream RoboTwin success, with larger gains at longer horizons.The tokenizer preserves state information more useful for world-action modeling and control.
  • Multi-chunk prediction: 29.7 percentage points: multi-chunk prediction outperforms the baseline after 5k steps on the 50 fps randomized setting.It matches the baseline’s 45k-step accuracy using 20k steps, a 2.3× training speedup.
  • Acceleration: 142 ms per chunk: the inference acceleration pipeline reduces end-to-end latency from 927 ms through consistency distillation and system-level optimizations.Table 3 reports inference time and asynchronous control frequency under different acceleration techniques.
  • Real-world demonstrations: Four real-world demonstrations test long-horizon organization, moving-object interaction, fine-grained grasping, and reactive visual control.The demonstrations cover complementary demands including planning, temporal grounding, and manipulation precision.

5 Related Work

Related work spans industrial and visuomotor policies, world-action models that predict scene evolution for control, and latent-action methods that recover control variables from video.

  • Robot-control policies: Industrial foundation models and visuomotor policies pursue real-world robot deployment through demonstrations, reasoning, memory, or direct action-sequence regression.Examples include LingBot-VLA, Qwen-RobotManip, WALL-OSS, ACT, and Diffusion Policy.
  • World-action models: World-action models use predicted scene evolution as an intermediate representation for robot action generation.DreamZero adapts pretrained video diffusion, while LingBot-VA formulates training as causal world modeling for frame prediction and policy execution.
  • Latent-action learning: Latent-action methods make unlabeled video usable for control by recovering controllable variables, policies, world models, inverse dynamics, or motion tokens.The line includes Genie, LAPO, LAPA, Moto, UniVLA, RepWAM, and Motus.

6 Conclusion

LingBot-VA 2.0 is built natively for embodied control, combining aligned semantic-action representations, causal modeling, and accelerated closed-loop inference. Across simulation and real-world evaluations, it improves over strong baselines, adapts from few demonstrations, and supports long-horizon manipulation; broader joint training and embodiment coverage remain open directions.

  • System design: LingBot-VA 2.0 builds every component around embodied control, from a shared semantic visual-action space to causal modeling and real-time closed-loop inference.Its stack includes a semantic tokenizer, sparse causal DiT, multi-chunk prediction, human–robot co-training, Foresight Reasoning, and acceleration.
  • Evaluation outcome: Across simulation and real-world evaluations, LingBot-VA 2.0 improves over strong vision-language-action and video-action baselines.It also adapts to new tasks from only a few demonstrations and sustains long-horizon manipulation through re-grounding and hierarchical planning.
  • Open directions: The planner and policy are trained separately, latent actions come from passive video, and current embodiment coverage is narrower than a general-purpose foundation model.The authors identify tighter joint training, interactive or reinforcement signals, larger pretraining and backbone scale, and broader embodiment coverage as open directions.

7 Contributors

The paper assigns contributors across pretraining, data preparation, post-training, acceleration, and deployment and demonstrations.

  • Contributor roles: Pretraining contributors are listed separately from those responsible for pretraining data, post-training, acceleration, and deployment and demonstrations.The contributor passage organizes responsibilities by these project stages.
  • Pretraining: The named pretraining team includes Qihang Zhang, Lin Li, Junke Wang, Jiahao Shao, and additional collaborators.The passage lists the full pretraining contributor group.
  • Acceleration and deployment: Acceleration and deployment and demonstration responsibilities are assigned to distinct named contributor groups.The passage separately identifies Shuaiting Li and Chaojian Li for acceleration, and a deployment and demo team.
Loading 2607.08639v2…