PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligence

Shuailei Ma, Jiaqi Liao, Xinyang Wang, Jingjing Wang, Chaoran Feng, Zijing Hu, Chong Bao, Zichen Xi, Yuqi Gan, Weisen Wang, Yanhong Zeng, Qin Zhao, Zifan Shi, Wei Wu, Hao Ouyang, Qiuyu Wang, Shangzhan Zhang, Jiahao Shao, Yipengjing Sun, Liangxiao Hu, Lunke Pan, Nan Xue, Kecheng Zheng, Yinghao Xu, Xing Zhu, Yujun Shen, Ka Leong Cheng

arXiv:2607.07675v1cs.CV

TL;DR

Video generation models are not designed for embodied intelligence, lacking emphasis on physical correctness, controllability, and efficient scaling. LingBot-Video addresses this with sparse MoE architecture, robot-augmented data, and multidimensional rewards, achieving a balance between modeling capacity and inference efficiency.

  • Problem

    Video foundation models prioritize perceptual quality over physical correctness and controllability, while dense architectures and internet-dominated data limit efficient scaling and embodiment grounding.

  • Method

    LingBot-Video combines sparse MoE video diffusion, robot-augmented pretraining data, and rewards for physical plausibility and task-level success.

  • Results

    The model achieves a balance between modeling capacity and inference efficiency as a large-scale MoE video foundation model for embodied intelligence.

  • Takeaways & Limitations

    LingBot-Video provides an open-source MoE video foundation model intended to bridge digital video generation and physical actuation.

Abstract

from arXiv · show

Despite the recent promise in robot control, video generative models suffer from a domain mismatch due to their primary focus on content creation. For example, their design inherently prioritizes visual fidelity and creativity over computational efficiency and physical realism. In this work, we present LingBot-Video, a DiT-based video pretraining paradigm specifically tailored for embodied intelligence. From the architecture perspective, we adopt the Mixture-of-Experts (MoE), instead of dense, framework to achieve a better trade-off between modeling capacity and inference efficiency, and manage to scale it up from scratch. From the data perspective, we construct a data profiling engine that augments standard internet videos with extensive robot-oriented footage, encompassing manipulation, navigation, and egocentric perspectives, to equip the base model with an intrinsic understanding of actions and world dynamics. From the training perspective, we develop a multi-dimensional reward system to enforce the alignment regarding physical rationality and task completion, going beyond standard criteria such as aesthetics, prompt-following, and motion consistency. Comprehensive evaluations validate its performance and efficiency as a video foundation model. We contribute LingBot-Video as the inaugural large-scale, open-source MoE video foundation model to the community, in a pioneering effort to bridge digital creativity and physical actuation.

1 Introduction

The introduction identifies a gap between video generation and embodied intelligence, whose requirements include physical correctness and controllability rather than only perceptual quality. LingBot-Video addresses this gap with sparse MoE architecture, robot-augmented data profiling, and multi-dimensional rewards.

  • Motivation: Most video foundation models prioritize realism, aesthetics, and text alignment over physical correctness and controllability, leaving interaction constraints insufficiently enforced.Examples include contact stability, rigid-body dynamics, and long-horizon state consistency.
  • Challenges: Existing approaches fall short in architecture, data, and training objectives, while dense video transformers activate all parameters uniformly, limiting scalability and increasing inference costs.The introduction frames these as three tightly coupled dimensions of the embodied-intelligence gap.
  • Approach: LingBot-Video is a DiT-based video pretraining paradigm designed for embodied intelligence, integrating sparse conditional computation, robot-augmented data, and embodied training objectives.Its MoE framework is intended to improve inference efficiency while scaling capacity for complex spatiotemporal dynamics.
  • Contributions: The work introduces a sparse MoE video diffusion framework with scalable training to balance model scalability and efficiency in spatiotemporal modeling.This contribution targets the limitations of dense computation identified in the introduction.
  • Contributions: A data profiling engine analyzes, filters, and rebalances heterogeneous videos, integrating internet and embodied datasets to improve grounding in physical interactions, action semantics, and embodiment-specific dynamics.The engine supports systematic integration of large-scale internet videos with embodied datasets.
  • Contributions: A multi-dimensional reward system combines physical plausibility and task-level success signals beyond conventional perceptual and text-alignment objectives.The introduction presents this reward design as the training-objective component of the proposed framework.

2 Method

LingBot-Video uses a task-unified single-stream diffusion transformer with sparse MoE routing to combine multimodal video generation across tasks with scalable representation capacity and controlled active computation. The method further introduces sequence-wise expert balancing, enabling large parameter capacity while keeping per-token computation comparable to dense models.

  • Unified Input Formulation: Unified inputs concatenate projected visual-latent patches and condition features into one token sequence for T2I, T2V, and TI2V generation.Image targets are represented as a special single token.
  • Single-Stream Diffusion Transformer: A single-stream diffusion transformer lets visual and condition tokens share transformer blocks, maximizing parameter reuse and supporting efficient scaling.The architecture uses lightweight modality-specific input projections before shared transformer processing.
  • Sequence-Wise Auxiliary Loss: Sequence-wise auxiliary balancing addresses routing imbalance within individual packed video sequences rather than relying only on batch-level expert statistics.This design is motivated by long spatio-temporal token sequences, where batch-level statistics can conceal per-video imbalance.
  • Sparse Mixture-of-Experts: Sparse MoE routing decouples total parameter capacity from per-token active FLOPs, allowing experts to store diverse physical priors without increasing dense-like computation.Each token is routed to K_r active experts, while total capacity grows with the expert pool size N_r.
  • Scaling and Efficiency: MoE 13B-A1.4B outperforms Dense 1.3B under a similar active compute budget, while MoE 30B-A3B closely approaches Dense 14B and larger sparse models scale predictably to 120B total parameters.The scaling series increases total parameters from 13B to 120B and active parameters from 1.4B to 11B; at 1M tokens, MoE 30B-A3B reaches a 0.97× latency ratio versus Dense 3B.

3 Data

LingBot-Video builds scalable, distribution-aware data infrastructure that profiles heterogeneous media into structured records, organizes concepts and actions for targeted sampling, and supports dense captions with inference-time prompt rewriting. This pipeline addresses data quality, diversity, and train–inference prompt mismatch through profiling, graph-based curation, and captioning.

  • Data Profiling Engine: The Data Profiling Engine converts images and videos into fixed-schema records covering structural, semantic, motion, camera, and quality attributes.These records drive filtering, balanced sampling, and captioning across heterogeneous media.
  • Captioning: A two-stage Caption Rewriter expands and formats brief user prompts into the structured caption representation used during training.This design bridges the distribution gap between lengthy attribute-rich training captions and brief free-form inference prompts.
  • Data Profiling Engine: Motion profiling separates camera, subject, and tracked motion, using VLM estimates and LocoTrack-based geometry to screen near-static or degenerate clips.The tracked-motion signal helps prevent spurious motion scores from allowing unsuitable clips into training.
  • World-Knowledge Topological Graph: The World-Knowledge Topological Graph organizes samples into semantic and video-specific action trees for distribution-aware curation.Training feedback and graph statistics up-weight rare or difficult concepts, including manipulation, physical contact, and long-tail human activities, while down-weighting saturated generic video.
  • Captioning: All training data receive dense JSON captions spanning images, videos, VLA videos, and egocentric videos under a shared schema.Video, VLA, and egocentric captions add fields to the image schema, providing attribute-rich supervision for controllability.

4 Infrastructure · 4.1 Pre-Training Infrastructure

LingBot-Video’s pre-training infrastructure is co-designed for heterogeneous, variable-length video workloads through token-budgeted packing, composable parallelism, memory management, compile-first execution, and asynchronous monitoring. These mechanisms target throughput, memory efficiency, long-context support, sparse-expert scaling, and reduced accelerator stalls.

  • 4.1 Pre-Training Infrastructure: The infrastructure addresses six bottlenecks in large-scale video pre-training, including heterogeneous streams, variable sequence lengths, token-budgeted loading, multidimensional parallelism, and activation memory.Its design is co-optimized for throughput and memory efficiency rather than adapted from image- or language-only workloads.
  • 4.1.1 Heterogeneous Data Pipeline and Token-Budgeted Packing: Mini-batches are assembled by target token budgets, with online length-aware scheduling estimating visual and conditioning tokens before appending samples.This accommodates variability from image and video resolution, duration, and conditioning inputs.
  • 4.1.1 Heterogeneous Data Pipeline and Token-Budgeted Packing: Samples are packed into one-dimensional sequences with cumulative lengths and attention masks, eliminating rectangular padding waste while supporting variable-length inputs.The packed format concatenates visual and conditioning segments as [x1, y1, x2, y2, . . . , xN, yN].
  • 4.1.1 Heterogeneous Data Pipeline and Token-Budgeted Packing: A smart-fill sampler searches rank-local candidate pools to use residual token capacity when the next video clip exceeds the remaining batch budget.This improves utilization of each packed batch without violating its token limit.
  • 4.1.2 Composable Parallel Training: The distributed stack composes data, fully sharded data, sequence, and expert parallelism through named multidimensional device meshes.Runs can select parallel modes compositionally without modifying the core training loop.
  • 4.1.2 Composable Parallel Training: FSDP shards parameters, gradients, and optimizer states, while SP handles long packed sequences and EP distributes sparse MoE experts with routed-token all-to-all communication.FSDP is the primary model-state scaling mechanism for dense transformer blocks and non-expert modules; SP transposes token and head shards during attention.
  • 4.1.3 Activation Checkpointing: Activation checkpointing recomputes selected activations to control long-sequence memory, with configurable block-, layer-, operation-, and memory-budget-level granularities.The configuration balances recomputation overhead against memory constraints.
  • 4.1.4 Compile-First Graph Capture: Combining full-block activation checkpointing with compile-first graph capture improves Model Flops Utilization (MFU) by approximately 1.9×.The pipeline applies torch.compile with Inductor before FSDP2 sharding so local block computations can be fused before sharding hooks are attached.

4.2 Post-Training Infrastructure

Section 4.2 presents a diffusion-native RL infrastructure for large MoE video generation, addressing token-scale trajectories, gigabyte-scale intermediate states, and synchronization pressure while supporting multiple optimization and finetuning modes. It achieves fast distributed execution with 20-second weight synchronization, sub-50-millisecond state exchange, and 43.9% end-to-end MFU.

  • Infrastructure demands: Large MoE video RL requires infrastructure for ~100K-token samples, gigabyte-scale latent trajectories and sampling statistics, and substantial parameter synchronization and memory management.These demands arise because video RL intermediate states are much larger than text-token representations.
  • Diffusion-native infrastructure: The system unifies conditioning, latent trajectories, rewards, and transition-level training data under a diffusion-native data abstraction.This abstraction is designed specifically for video diffusion RL post-training.
  • Optimization and finetuning: It supports GRPO-style reverse-process optimization, forward-process objectives, LoRA-style parameter-efficient finetuning, and full-model finetuning.The infrastructure accommodates both objective families and both parameter-update regimes.
  • Efficiency: 20 seconds per step is sufficient for full-parameter weight synchronization of the 30B model.This synchronization performance is reported for the complete model at each step.
  • Efficiency: 50 milliseconds is sufficient to exchange gigabyte-scale intermediate states across multiple GPU nodes, while end-to-end MFU reaches 43.9% over the full RL step.The reported communication and utilization figures characterize distributed rollout-to-training execution.

4.3 Serving Infrastructure

LingBot-Video uses a Diffusers-compatible package and SGLang Diffusion runtime to support open deployment, regression testing, and optimized multi-GPU long-video generation. The serving stack supports T2I, T2V, and TI2V generation, with fidelity-first and speed-first execution profiles.

  • Serving Infrastructure: The serving stack combines a Diffusers-compatible model package with an SGLang Diffusion runtime for portable deployment, numerical regression testing, and optimized multi-GPU long-video generation.The same runtime supports text-to-image, base text-to-video, and text-image-to-video generation.
  • Serving Infrastructure: The Diffusers-compatible release enables deployment through standard Diffusers APIs in generic PyTorch environments without complex configuration or custom binary compilation.The packaging scheme is intended to maximize accessibility and lower the deployment barrier for the open-source community.
  • Serving Infrastructure: SGLang provides optimized CUDA kernels and distributed scheduling policies through three execution paths: direct Diffusers, generic SGLang Diffusion, and a third path described in the deployment framework.The passage identifies these as distinct runtime execution paths but does not provide the third path’s name in the supplied text.
  • Serving Infrastructure: The deployment framework recommends context parallelism for sharding long-video token sequences across GPUs and batched CFG for efficient conditional and unconditional branch evaluation.Users select between specialized execution modes based on deployment objectives.
  • Serving Infrastructure: The fidelity-first profile follows the training codebase for scheduler logic, routed-expert execution, and precision policy to support regression testing and numerical-consistency checks.It preserves grouped GEMM, vectorized token padding and restoration, and a hybrid precision layout.
  • Serving Infrastructure: The speed-first profile targets high-throughput serving, rapid visual screening, and low-latency interaction by replacing standard routed-expert execution with FP8 SGLang Triton kernels.The supplied passage states that this reduces memory footprint and kernel execution times, combined with sequence sharding and parallelized guidance evaluation.

5 Training

Training uses a progressive curriculum to stabilize large-scale sparse-MoE video pretraining, then applies multidimensional rewards to address visual, temporal, and physical failure modes. The model is further distilled with DMD2 into a few-step generator for improved inference efficiency.

  • Progressive Pre-training Curriculum: A progressive curriculum separates visual-prior learning, temporal modeling, task conditioning, data harmonization, and high-resolution refinement to stabilize optimization and gradual router specialization.The staged order reduces the risk of instability, routing collapse, and suboptimal sample quality when complexity is introduced incrementally.
  • Progressive Pre-training Curriculum: Stages 1–3 progress from image-only T = 1 sequences to low-resolution T2V and then joint T2V/TI2V training, building spatial priors, temporal dynamics, and visual-condition handling.TI2V training requires preserving the provided initial frame while predicting temporally coherent future frames.
  • Progressive Pre-training Curriculum: Stages 4–5 use weighted sampling to balance data quality and source distribution, then refine high-frequency details, local artifacts, and temporal sharpness in high-resolution videos.The weighted sampler selects high-value and high-quality sources using predefined weights, while the final refiner targets issues such as texture flickering.
  • Reward Modeling: The reward system replaces coarse holistic scores with separate signals for vision quality, action-centric text-video alignment, dynamic degree, motion coherence, and human-motion consistency.These rewards target blurriness, caption misalignment, inappropriate motion intensity, unnatural playback, and human-motion artifacts including hand deformity and limb-count errors.
  • Inference Distillation: DMD2 distillation matches a student generator to the teacher distribution, converting LingBot-Video into a few-step generator for improved inference efficiency.The student is conditioned on the unified condition c and generates x0 = Gθ(z, c).

6 Evaluation

LingBot-Video is evaluated on internal general-quality and embodied-domain benchmarks spanning T2V and TI2V generation, alongside public robotics and physics benchmarks. It achieves state-of-the-art open-source performance on internal TI2V evaluation and ranks first among evaluated open-source models on Physics-IQ Verified.

  • Internal benchmark: The internal benchmark evaluates T2V and TI2V across general visual quality and embodied scenarios involving physical interactions and real-world robot conditions.General quality emphasizes visual appeal, temporal consistency, and semantic accuracy; embodied evaluation predicts future interactions from an initial frame and control command.
  • Internal benchmark: General-quality metrics cover motion quality, prompt following, visual consistency, and aesthetic quality.These assess movement naturalness and plausibility, instruction adherence, identity and scene preservation, and cinematic visual execution.
  • Internal benchmark: Embodied-domain categories include human interaction, physical simulation, robotics, egocentric perspective, and navigation.They probe fine-grained interactions, intuitive physical laws, diverse robot platforms, first-person rendering, and spatial layouts or motion planning.
  • Internal benchmark results: LingBot-Video achieves state-of-the-art performance among open-source competitors on TI2V, ranking first in both general quality and embodied-domain scores.The comparison includes NVIDIA Cosmos 3 Super-Image-to-Video, Wan 2.2 A14B, LongCat-Video, Hunyuan Video 1.5, and LTX-2.3.
  • Public benchmarks: RBench evaluates 650 text-image prompts across five task-oriented interaction types and four embodiment-specific robot-morphology categories.It is designed to assess correctness of robot-centric interactions and complements the internal robotics evaluation.
  • Public benchmarks: 40.4 is LingBot-Video’s Physics-IQ Verified score, ranking first among evaluated open-source models and narrowly surpassing Cosmos 3 at 39.5.Physics-IQ Verified uses 66 controlled experiments spanning solid dynamics, fluid dynamics, thermodynamics, optics, and magnetism.

Conclusion and Discussion

The conclusion presents LingBot-Video as an MoE-based video foundation model for embodied intelligence, balancing modeling capacity with inference efficiency. It outlines applications in data generation, policy evaluation, and action planning, while open-sourcing the model to support community collaboration.

  • Core contribution: LingBot-Video is an MoE-based video foundation model tailored for embodied intelligence, bridging digital creativity and physical actuation.The model scales an MoE-based Diffusion Transformer from scratch.
  • Applications: The Data Engine synthesizes high-fidelity, low-cost training data at scale to mitigate robotics data scarcity.
  • Applications: The Policy Evaluator acts as a visual simulator for safety-critical robot-policy evaluation without real-world risks.
  • Applications: The Action Planner predicts what happens next to support robots’ real-time decision-making and planning.
  • Community release: Open-sourcing LingBot-Video is intended to foster community collaboration on embodied physical engines and next-generation robot brains.

A Structured Caption Details · A.1 Example Structured Captions

The section presents representative structured captions reproduced verbatim for image, video, VLA, and egocentric data. These examples combine scene descriptions, camera attributes, prominent elements, and temporally grounded actions.

  • A.1 Example Structured Captions: Image captions describe visual scenes with structured camera fields, including an opulent palace interior, extreme-wide framing, low angle, balanced composition, and hard light.The caption identifies vaulted ceilings, red wall panels, fluted columns, parquet flooring, a portrait, throne, and chandelier as prominent elements.
  • A.1 Example Structured Captions: Across modalities, the structured format records comprehensive descriptions alongside camera properties, prominent elements, object attributes, and timestamped actions.The examples include image, video, VLA, and egocentric data categories, each represented by a structured caption reproduced from training data.
  • A.1 Example Structured Captions: Video captions represent an outdoor cooking scene in which meat rolls cook on a black griddle over an open fire while smoke rises and a hand adds rolls.The camera remains stationary in an eye-level close-up, and the principal action spans [0.0s - 7.1s].
  • A.1 Example Structured Captions: The video example decomposes objects into attributes such as location, relative size, shape and color, texture, appearance, relationship, and orientation.For the meat rolls, the caption records several cylindrical red, green, and orange objects placed on the griddle, with visible toothpicks and stuffing.
  • A.1 Example Structured Captions: VLA captions describe a first-person robotic workspace containing a shopping cart, produce-filled display shelf, and two robotic arms.The fixed camera uses a slightly high-angle view, while the right arm grasps a green cucumber, releases it, and retracts upward and rightward.
  • A.1 Example Structured Captions: The VLA example links actions to timestamps: the right robotic arm moves down while grasping a cucumber, opens its gripper, then moves upward and right.The recorded intervals are [0.0s - 1.5s], [1.5s - 5.5s], and [5.5s - 7.2s].
  • A.1 Example Structured Captions: Egocentric captions model body-mounted video with first-person camera motion, machinery, hands, environmental context, and object interactions.The example shows a person working on an orange lawnmower in a garage or workshop, with the right hand pulling a black cable while the left hand remains stationary.
Loading 2607.07675v1…