Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligence

• The approach responds to a persistent mismatch: perceptually strong video models often lack physical correctness, controllability, embodiment priors, and long-horizon consistency under intervention.Internet-scale visual diversity alone does not guarantee fidelity to contact stability, rigid-body dynamics, or embodied interaction constraints.Source: §1 · pp. 1–2
• Its sparse MoE framework uses conditional computation to improve inference efficiency while scaling capacity for complex spatiotemporal dynamics.The framework targets a scalability–efficiency trade-off in video modeling rather than uniformly activating all parameters.Source: §1 · pp. 2–3
• LingBot-Video is presented as the first large-scale, open-source MoE video foundation model for embodied intelligence, bridging video generation and physical actuation.It is a DiT-based pretraining paradigm designed specifically for embodied intelligence.Source: §1 · pp. 2–3
• A data profiling engine combines internet-scale videos with manipulation, navigation, and egocentric robot datasets to strengthen grounding in interactions, actions, and embodiment-specific dynamics.The engine systematically analyzes, filters, and rebalances heterogeneous video sources.Source: §1 · pp. 2–3
• A multi-dimensional reward system adds physical plausibility, rationality, and task-level success signals beyond conventional aesthetics and text-video alignment objectives.These signals encourage dynamics and interactions consistent with embodied environments and address physical feasibility and task completion.Source: §1 · pp. 2–3

2.1 Task-Unified Single-Stream Diffusion Transformer

• The unified input formulation concatenates visual-latent and condition tokens, supporting T2I, T2V, and TI2V generation with images represented as T = 1 videos.A single-stream transformer shares blocks across modalities, enabling dense cross-modal interaction and reducing tensor reorganization, communication, and synchronization overhead relative to dual-stream designs.Source: §2.1 · p. 3

2.2 Scaling with Sparse Mixture-of-Experts

• Sparse MoE replaces the dense FFN while preserving shared experts and top-Kr routed experts, decoupling total parameter capacity from active per-token computation.Fine-grained experts specialize heterogeneous physical and task-dependent patterns, while sequence-wise auxiliary loss and online bias correction balance routing without increasing per-token FLOPs linearly.Source: §2.2 · pp. 4–6

2.3 Sparse Mixture-of-Experts Recipe Exploration

• With active parameters fixed at 1.4B, increasing routed experts from E = 64 to 128 to 256 consistently improves losses, but gains beyond E = 128 are marginal.Under a fixed 13B total budget, fine-grained MoE 13B-A1.4B-E128 with Kr = 8 outperforms coarse MoE 13B-A1.5B-E64 with Kr = 4.Source: §2.3 · p. 7, Figure 3 · p. 7

2.4 Scaling Experiments

• At 1M tokens, MoE 30B-A3B reaches 0.97× latency versus Dense 3B and achieves 1.50×, 2.59×, and 3.18× speedups versus Dense 6B, 14B, and 30B.The model also consistently outperforms dense baselines at comparable active compute, while larger sparse models achieve lower training loss through 120B total parameters.Source: §2.4 · pp. 7–9, Figure 7 · p. 9

2.5 Cascaded Refiner

• The cascaded refiner upsamples base-generated videos from 480p to 1080p using degraded low-resolution conditions encoded into VAE latents.Conditional rectified flow restricts denoising to t ≤ τ with τ ∼ Uniform(0.85, 0.95), preserving global motion while restoring high-frequency details and local artifacts.Source: §2.5 · pp. 9–10
• The refiner upsamples base-stage video, adds Gaussian noise at t = τinf, and integrates the rectified flow ODE backward to generate the clean latent.The noisy starting latent is xτinf = (1 − τinf)xlr + τinfϵ, with τinf = 0.85 given as an example.Source: §2.5 · p. 10
• The refiner significantly enhances face appearance, restores high-frequency details, and produces sharp, legible OCR text.Source: §2.5 · p. 10

3 Data

• The integrated infrastructure closes the training-to-deployment gap by combining structured profiling, semantic-action organization, dense supervision, prompt rewriting, and stage-wise control of data composition.The curriculum governs the volume and composition consumed during progressive pre-training, while the profiling and graph modules support balanced long-tail coverage.Source: §3 · pp. 10–11, Figure 10 · p. 12

3.1 Data Profiling Engine

• The Data Profiling Engine converts heterogeneous image and video samples into queryable records spanning structural, semantic, motion, camera, and quality attributes for filtering, sampling, and captioning.Figure 9 shows these five dimensions driving downstream filtering, balanced sampling, and captioning.Source: §3.1 · p. 11, Figure 9 · p. 11

3.2 World-Knowledge Topological Graph

• The World-Knowledge Topological Graph combines a 50,000-leaf, 1,000-intermediate, 25-top-level semantic hierarchy with several hundred canonical action nodes for distribution-aware curation.Training loss and coverage statistics up-weight rare or difficult manipulation and long-tail nodes, down-weight saturated modes, and guide targeted data acquisition.Source: §3.2 · pp. 12–13

3.3 Dense Structured Captioning

• Dense structured JSON captions unify images, videos, VLA videos, and egocentric videos while adding camera, entity, temporal, and action information appropriate to each modality.Video captions specify what happens, camera motion, and timestamped actions; VLA and egocentric captions describe robotic or first-person interactions.Source: §3.3 · p. 13

3.4 Caption Rewriter

• The Caption Rewriter bridges brief inference prompts and lengthy training captions through concise action-centric expansion followed by bounded JSON mapping, limiting hallucinated details.The two-stage pipeline supports both T2V and TI2V, with duration constraints on predicted action timestamps and first-frame grounding for TI2V.Source: §3.4 · p. 14

3.5 Data Curriculum

• The five-stage data curriculum progresses from filtered 192p images to joint image-video training, 480p re-filtering, challenge-focused embodiment rebalancing, and a tiny 1080p refinement subset.Stage 2 injects over 70,000 hours of robot manipulation, navigation, egocentric, and related embodiment footage, while Stage 5 retains well under 1% of the initial video pool.Source: §3.5 · pp. 14–15

4.1 Pre-Training Infrastructure

• The infrastructure addresses six bottlenecks—token-budgeted loading, multidimensional parallelism, activation memory, graph capture, communication prefetching, and non-blocking I/O—created by heterogeneous image-video streams and variable sequence lengths.These challenges distinguish large-scale video pre-training from image-only or language-only workloads.Source: §4.1 · p. 15

4.1.1 Heterogeneous Data Pipeline and Token-Budgeted Packing

• Token-budgeted packing dynamically schedules samples using visual and conditioning lengths, while one-dimensional packing and smart filling eliminate padding waste across heterogeneous inputs.Packed metadata enables variable-length attention without cross-sample leakage, and rank-local candidate search fills residual token capacity when clips exceed the remaining budget.Source: §4.1.1 · pp. 15–16

4.1.2 Composable Parallel Training

• Four composable device-mesh dimensions—DP, FSDP, SP, and EP—unify throughput scaling, model-state sharding, long-context partitioning, and MoE token routing.FSDP shards dense and non-expert states, SP distributes sequence length across accelerators, and EP dispatches routed tokens to local experts using accelerated DeepEP communication.Source: §4.1.2 · p. 16

4.1.3 Activation Checkpointing

• Activation checkpointing supports configurable block-, layer-, operation-, and memory-budget-level recomputation to balance memory constraints against overhead for long packed video sequences.The non-reentrant implementation prevents lifetime conflicts between activation recomputation and expert communication collectives.Source: §4.1.3 · p. 16

4.1.4 Compile-First Graph Capture

• Approximately 1.9× higher MFU results from combining full-block activation checkpointing with compile-first graph capture before FSDP2 sharding.The pipeline applies torch.compile with the Inductor backend before FSDP2 so local block computations are fused before sharding hooks are attached.Source: §4.1.4 · p. 16

4.1.5 Asynchronous Monitoring and Distributed Checkpoint I/O

• Asynchronous monitoring queues logging and serialization work to background processing so comprehensive diagnostics, including MFU, step-time, MoE balance, and routing statistics, do not stall accelerators.GPU tensors are moved to host memory before queueing, and bounded queues prevent logging backpressure from blocking computation.Source: §4.1.5 · p. 17

4.2 Post-Training Infrastructure

• Video diffusion post-training requires specialized infrastructure because denoising-coupled optimization, high-dimensional latent trajectories, and MoE scale break token-centric assumptions about log probabilities, credit assignment, and rewards.A single sample involves roughly 100K tokens, while intermediate video-RL states reach gigabyte scale and strain synchronization and memory management.Source: §4.2 · p. 17, §4.2.1 · p. 17

4.2.1 Diffusion-Native RL System Design

• The diffusion-native system unifies conditioning, latent trajectories, rewards, and transition data while supporting GRPO-style reverse-process optimization, forward-process objectives, LoRA, and full-model finetuning.Rollout, reward evaluation, and training are decoupled, with communication-aware latent handling and a service-oriented reward layer for heterogeneous models, server-side decoding, and batching.Source: §4.2.1 · p. 17

4.2.2 RL System Performance

• 43.9% end-to-end MFU is sustained over the full RL step, with 30B-model weight synchronization taking 20 seconds per step and gigabyte-scale state exchange under 50 milliseconds.These measurements demonstrate the infrastructure’s speed across synchronization, inter-node communication, and complete RL execution.Source: §4.2.2 · p. 17

4.3 Serving Infrastructure

• The serving stack combines portable packaging, a numerically aligned reference path, and optimized multi-GPU execution under one runtime supporting T2I, T2V, and TI2V generation.A second-stage refiner can optionally follow base video generation.Source: §4.3 · p. 17

4.3.1 Diffusers-Compatible Model Package

• Diffusers-compatible packaging enables out-of-the-box deployment in generic PyTorch environments while sharing conditioning and autoencoder weights between base and refiner DiTs.A lightweight overlay builder exposes the selected DiT as the active transformer without duplicating shared weights or tensors.Source: §4.3.1 · p. 18

4.3.2 SGLang-Native Serving Backend

• SGLang provides optimized CUDA kernels and distributed scheduling through direct Diffusers, generic SGLang Diffusion, and LingBot-Video-native execution paths.The direct path remains a readable numerical reference, while the native adapter preserves denoising alignment and integrates custom serving hooks.Source: §4.3.2 · p. 18

4.3.3 Distributed Video Serving

• Distributed serving shards long-video token sequences across GPUs with context parallelism and accelerates conditional and unconditional branches through batched CFG evaluation.These configurations target diverse production requirements, including regression testing, rapid screening, and low-latency interactive applications.Source: §4.3.3 · p. 18
• The fidelity-first mode preserves training-code behavior with BF16 transformer computation and FP32 numerically sensitive parameters to keep deviations within expected tolerance.It retains grouped GEMM, vectorized token padding and restoration, and a hybrid precision layout for regression testing.Source: §4.3.3 · p. 18
• The speed-first mode replaces routed-expert execution with FP8 SGLang Triton kernels, reducing memory and kernel time while maintaining scheduler and model interfaces.Sequence sharding and parallelized guidance evaluation provide substantial throughput improvements with limited observed impact on video quality.Source: §4.3.3 · p. 18

5.1 Progressive Pre-Training

• The five-stage curriculum progresses from image-only priors through low-resolution temporal learning, T2V/TI2V conditioning, weighted data harmonization, and high-resolution refinement.This ordering separates static appearance from dynamics, lets the sparse router specialize gradually, balances heterogeneous data sources, and restores fine spatial details while preserving motion.Source: §5.1 · p. 19

5.2.1 Reinforcement Learning with Multi-Aspect Rewards

• Six specialized rewards jointly optimize vision quality, text-video alignment, dynamic degree, motion coherence, human-motion consistency, and physical plausibility with task completion.The physical evaluator checks causality, object permanence, non-penetration, material-kinematic realism, action occurrence, manipulation, and the intended final spatial state from visible frame evidence.Source: §5.2.1 · pp. 19–20

5.2.2 On-Policy GRPO Training

• On-policy GRPO uses one group-shared stochastic denoising step, CPS sampling, inverse-gain timestep reweighting, and independently normalized rewards for precise and balanced credit assignment.Only the sampled transition receives the policy update; CPS keeps samples on the deterministic sampler’s marginal path, while analytic normalization equalizes update magnitudes across timesteps.Source: §5.2.2 · p. 21

5.2.3 Negative-Aware Finetuning with Real-World Videos

• Negative-aware finetuning pairs real videos as chosen samples with same-prompt generations as rejected samples, using single-step forward-process optimization and active/EMA policy pathways.The method avoids trajectory-level likelihood computation, assigns real videos r = 1 and generated videos r = 0, and regularizes against a frozen base model to limit drift.Source: §5.2.3 · pp. 22–23

5.3 Action-to-Video Post-Training

• LingBot-Video-A2V converts raw robot commands into aligned relative-action latents injected into transformer blocks, using leakage-filtered initial-state captions and joint backbone/action-branch training.Post-training uses Fourier GR-1 data for 8k steps with batch size 64 and learning rate 1e-5; the resulting model achieves high-quality action following for embodied world simulation.Source: §5.3 · pp. 25–26

5.4 Distillation

• DMD2 distillation trains a few-step student generator by matching its distribution to a teacher across diffusion noise levels while retaining a lightweight GAN objective for visual quality.The auxiliary score model estimates the student score online, while teacher scores provide real-distribution supervision under the unified condition c.Source: §5.4 · p. 26

6.1 Internal Benchmark

• The internal benchmark measures visual quality through motion quality, prompt following, visual consistency, and aesthetics, and tests embodied generation through human interaction, physical simulation, robotics, egocentric perspective, and navigation.The embodied setting conditions future interaction prediction on an initial frame I0 and control command T, targeting physical understanding and spatial reasoning.Source: §6.1 · p. 26
• LingBot-Video achieves state-of-the-art TI2V performance among open-source competitors in both general quality and embodied-domain scores, while ranking second in T2V general quality and outperforming Cosmos on T2V embodied-domain scores.These results are reported on an internal benchmark spanning T2V and TI2V, with general-quality and embodied-domain evaluations.Source: §6.1 · p. 27, Figure 15 · p. 27

6.2 Public Benchmark

• On RBench, LingBot-Video demonstrates embodied-generation capability that generalizes beyond the internal suite across task-oriented interactions and four robot morphologies.RBench contains 650 text-image prompts covering manipulation, spatial relationships, collaboration, long-horizon planning, visual reasoning, and single-arm, dual-arm, humanoid, and quadruped robots.Source: §6.2 · p. 28, Table 1 · p. 28
• LingBot-Video scores 40.4 on Physics-IQ Verified, ranking first among evaluated open-source models and narrowly surpassing Cosmos 3 at 39.5, while exceeding Hunyuan Video 1.5 at 33.4 and Wan 2.2 A14B at 32.2.The evaluation uses the image-to-video setting and tests prediction of real physical processes across controlled experiments spanning solid, fluid, thermal, optical, and magnetic phenomena.Source: §6.2 · pp. 28–29, Figure 16 · p. 28

6.3 User Study

• In blind GSB comparisons over 400 prompts per pair, LingBot-Video’s Good rate exceeds its Bad rate against several open-source T2V baselines and all evaluated open-source TI2V baselines, while trailing stronger commercial models.The T2V comparisons are closer against HunyuanVideo 1.5 and Cosmos 3, whereas TI2V shows more consistent gains across open-source competitors.Source: §6.3 · p. 29, Figure 17 · p. 29

6.4 Action-to-Video Post-Training

• On EgoDex Eval and DreamDojo-HV Eval, LingBot-Video-A2V generalizes to novel objects and actions absent from GR-1 post-training data, adhering better to physical laws and following actions more accurately than DreamDojo.The evaluations test out-of-distribution action following rather than memorization of training rollouts.Source: §6.4 · p. 29, Figure 18 · p. 30

• LingBot-Video scales an MoE-based Diffusion Transformer from scratch to balance modeling capacity and inference efficiency for embodied intelligence.The model is intended to bridge digital creativity and physical actuation and support next-generation robot brains.Source: Conclusion and Discussion · p. 36
• As a Data Engine, LingBot-Video synthesizes high-fidelity, low-cost training data at scale to mitigate robotics data scarcity.Source: Conclusion and Discussion · p. 36
• As a Policy Evaluator, it acts as a visual simulator for safety-critical robot-policy evaluation without real-world risks.Source: Conclusion and Discussion · p. 36
• As an Action Planner, it predicts what happens next to assist robots’ real-time decision-making and planning.Source: Conclusion and Discussion · p. 36
• Open-sourcing LingBot-Video aims to foster community collaboration on embodied physical engines and next-generation robot brains.Source: Conclusion and Discussion · p. 36

A.1 Example Structured Captions

• The appendix provides one representative structured caption per data category, including separate examples for image and video data.The image example is explicitly labeled, followed later by a video example.Source: §A.1 · pp. 44–45
• The image caption describes a grand palace hall through comprehensive scene prose and structured fields for framing, angle, composition, and lighting.Its object annotations specify prominent elements such as the vaulted ceiling, red wall panels, columns, parquet floor, portrait, throne, and chandelier.Source: §A.1 · p. 44
• The video caption represents an outdoor cooking scene with a stationary close-up camera and time-localized action describing meat rolls cooking on a griddle.Structured annotations capture scene content, camera movement, prominent objects, timestamp, action, location, size, color, texture, and appearance details.Source: §A.1 · p. 46
• The robotic manipulation example tracks object state and spatial relations, from the cucumber held by the gripper to its final resting position inside the plastic bag.It specifies the cucumber’s locations, actions, orientation changes, and relationship to the arm and bag, while separately marking the left arm and cart as stationary.Source: §A.1 · p. 48
• The structured captions represent objects with names, descriptions, locations, relative sizes, visual attributes, relationships, orientations, and timestamps.Examples include a display shelf containing artificial vegetables and a receptacle for a dropped cucumber.Source: §A.1 · p. 49
• The egocentric example describes a first-person workshop scene where a right hand grasps and pulls a black lawnmower cable while the left hand remains stationary for stability.The caption identifies the orange lawnmower, concrete floor, visible clothing, cable interaction, and action interval from 2.0s to 5.0s.Source: §A.1 · pp. 49–50
• The camera metadata specifies a body-mounted first-person view with continuous head-linked panning and tilting, a close-up shot, medium frame size, high angle, and ultra-wide or fisheye lens.Lighting is recorded as hard daylight with centered composition.Source: §A.1 · p. 49
• From 0.0s to 2.0s, the referenced entity remains stationary; from 2.0s to 5.0s, it is grasped by the right hand and pulled upwards and to the right.Source: §A.1 · p. 51