PaperlayerView exact revision on arXiv

Source-linked AI summary

SolarWM: Open Data and Scalable Training for Long-Horizon Video World Models

Junchao Huang, Guian Fang, Shengju Qian, Xianghao Kong, Zhuoran Zhao, Wei Huang, Yihua Du, Zixin Zhang, Justin Cui, Yuchao Gu, Yukang Chen, Xinting Hu, Tianyu He, Shaoshuai Shi, Zhuotao Tian, Xin Wang, Mike Zheng Shou, Li Jiang

arXiv:2609.02886v1cs.CV

TL;DR

Interactive video world-model training must reconcile heterogeneous datasets with incompatible video backbones while remaining reproducible. SolarWM addresses this with a unified data engine, shared backbone interfaces, and a three-stage recipe, producing causal models that roll out for minutes to hours from 5s training sequences.

  • Problem

    Heterogeneous datasets and video backbones create inconsistent supervision and model-specific implementations, leaving the field without a unified, reproducible foundation.

  • Method

    SolarWM converts multiple sources into a canonical frame-aligned contract and adapts four backbones through shared interfaces and bidirectional, autoregressive, and DMD training stages.

  • Results

    After training only on 5s sequences, the resulting causal models support real-time interaction and open-ended rollouts lasting minutes to hours.

  • Takeaways & Limitations

    SolarWM provides an open, reproducible, and extensible foundation spanning data preparation, model training, and long-horizon inference.

  • Takeaways & Limitations

    The in-domain validation protocol uses samples from the same source families as training, while excluding validation samples from optimization.

Abstract

from arXiv · show

We introduce SolarWM, a fully open foundation for building interactive video world models from data preparation through long-horizon inference. Training across heterogeneous data sources and video backbones is challenging: datasets differ in temporal scale, camera geometry, visual quality, motion, and captioning styles, while video generators use distinct representations and architectures. Naive data mixing and model-specific implementations therefore produce inconsistent supervision and make results difficult to reproduce and compare. SolarWM addresses this coupling with a reconfigurable multi-source data engine and a backbone-native adaptation framework. The engine converts 1.43 million canonical clips from 10 datasets into a unified, frame-aligned contract covering visual observations, metric camera geometry, captions, quality metadata, selection decisions, and provenance, while decoupling source processing from mixture construction. Under shared camera-conditioning, training, and inference interfaces, we instantiate four 5B--33B models based on Wan2.2, LTX-2.5, and MiniMax-H3 while preserving their native representations and objectives. A unified three-stage recipe combines bidirectional adaptation, teacher-forced autoregressive initialization, and distribution matching distillation. The resulting causal models enable real-time interaction over rollouts ranging from minutes to hours after being trained on only 5s sequences. By releasing the resulting data, pipeline, recipes, weights, and framework, SolarWM provides a reproducible and extensible foundation for interactive world-model research.

1. Introduction

SolarWM presents an open foundation that coordinates heterogeneous data preparation, backbone adaptation, and long-horizon interactive video modeling. Its unified data infrastructure and three-stage training recipe yield scalable causal models supporting minute-to-hour rollouts from short training sequences.

  • 5s training sequences suffice for open-ended minute-to-hour rollouts without additional long-sequence fine-tuning or attention-sink mechanisms.The recipe combines bidirectional training, autoregressive adaptation, and distribution matching distillation; most optimization occurs during bidirectional training.
  • SolarWM integrates reconfigurable multi-source data infrastructure with backbone-native adaptation across data construction, training, and long-horizon inference.
  • 1.43 million clips from 10 datasets are processed into a reusable corpus spanning real-world, synthetic, and game environments.Source preprocessing is decoupled from mixture construction, allowing filtering criteria, sampling ratios, and source weights to change without rerunning preprocessing.
  • Four 5B–33B models preserve backbone-native representations while sharing interfaces for camera conditioning, optimization, and rollout.The model family covers Wan2.2, LTX-2.5, and MiniMax-H3 backbones and supports camera-controllable generation across multiple environment types.

2. Open-Source Release

SolarWM’s open-source release covers the processed corpus and the artifacts needed to reproduce and reconfigure its training pipeline. The release separates expensive source processing from downstream mixture design.

  • The release includes 1.43 million canonical clips from 10 source datasets with annotations, metadata, quality assessments, selection decisions, and provenance.
  • Researchers can alter filtering criteria, sampling ratios, source weights, or split definitions without rerunning computationally intensive source preprocessing.
  • SolarWM will release training recipes, model weights, and implementations of its training stages and inference framework.

3. Related Work

Related work spans interactive world models, causal adaptation of bidirectional video generators, and partially open research stacks. SolarWM targets the remaining reproducibility gap with a unified, recipe-complete, cross-backbone framework and an auditable data engine.

  • Interactive video world models: Interactive world models extend passive video generation into visual simulation conditioned on actions, camera motion, or semantic instructions.
  • Causal adaptation: Bidirectional video generators require staged adaptation for causal rollout, with prior work using autoregressive training, generated context, and distillation objectives.
  • Open research stacks: Existing systems often lack complete processed data, selection records, executable source-to-training pipelines, or exact optimization recipes needed for direct reproduction.
  • SolarWM framework: SolarWM applies one three-stage recipe across four models while sharing data, camera-conditioning, optimization, and rollout interfaces.The framework retains only backbone-native representations and objectives, making the recipe reusable across backbones rather than model-specific.
  • Data construction: Heterogeneous interactive-world-modeling sources differ in temporal scale, resolution, camera evidence, coordinate conventions, metric scale, captions, and motion distributions.
  • Data construction: SolarWM normalizes these sources into a frame-aligned representation and separates source processing from mixture construction, releasing records and executable recipes for reconfiguration.

4. Training Pipeline

SolarWM trains world models through three stages: bidirectional adaptation, teacher-forced AnyFlow initialization, and DMD-based causal training. The pipeline progressively adapts pretrained video generators, introduces causal prediction, and aligns the student with the inference-time rollout distribution.

  • Training Pipeline: SolarWM uses bidirectional adaptation, teacher-forced AnyFlow autoregressive initialization, and DMD-based causal training.The three stages respectively adapt the pretrained generator, produce a few-step causal initializer, and address the exposure gap during causal rollout.
  • 4.1. Bidirectional Adaptation: Bidirectional adaptation trains on camera-conditioned world data with complete-window attention while preserving unrestricted temporal context.Fused-PRoPE injects camera control into the existing self-attention path without a separate control branch or attention pass.
  • 4.2. Teacher-Forced AnyFlow Initialization: Teacher-forced AnyFlow hides future blocks, uses clean ground-truth history, and directly supervises flow transitions between arbitrary noise levels.This produces a few-step autoregressive checkpoint without separate Causal ODE or Causal CD initialization stages.
  • 4.3. DMD-based Causal Training: DMD-based causal training rolls out the student under the inference-time temporal rule and uses a frozen bidirectional teacher to estimate the target distribution.A trainable fake-distribution model tracks the student’s evolving rollout distribution, and their score difference supplies the student’s distribution-matching direction.

5. Open Data Engine

SolarWM’s Open Data Engine turns heterogeneous video sources into a traceable, frame-aligned corpus while separating physical data, recipes, and model views. It supports source-aware filtering, quality assessment, and reproducible temporal-view construction.

  • Corpus organization: 1.43M clips from 10 datasets are organized into 14 independently addressable dataset owners for filtering, weighting, and recipe construction.The corpus spans real-world, synthetic, and game environments with diverse camera motions, temporal scales, and motion patterns.
  • Unified data contract: Every sample aligns video, camera poses, intrinsics, captions, audio when available, metadata, quality metrics, and provenance to a deterministic frame interval.Metric camera-to-world transforms and per-frame intrinsics remain traceable to source processing and lineage.
  • Reconfigurable infrastructure: The engine separates canonical samples from logical recipes and backbone-specific model views, so filtering, source mixtures, and temporal windows can change without duplicating videos.Users can alter thresholds, source weights, and temporal views without rerunning expensive source-level processing.
  • Clean Plate processing: Clean Plate derivatives remove dynamic people and vehicles while preserving scene layout and camera motion, then recompute captions, metrics, and camera diagnostics.The pipeline treats cleaned outputs as new data requiring quality reassessment because transformation artifacts and altered motion statistics can occur.
  • Released corpus: 876k of 1.4M canonical clips are retained: 471k high-tier and 404k xhigh clips, while 549k processed clips remain rejected with provenance.The complete corpus occupies approximately 25.85 TB across 29k shards; temporal aggregation shows 858k clips in the 153–956-frame range.

6. SolarWM Model Family

SolarWM provides one interactive world-model framework across four heterogeneous video backbones. Shared camera, data, training, and inference interfaces enable comparison while preserving backbone-native representations and contracts.

  • Model family: Four models span Wan 2.2-5B, Wan 2.2-14B, LTX-2.5-22B, and MiniMax-H3-33B under a common SolarWM contract.The organization supports both backbone comparison and route-specific adapters.
  • Camera interface: A shared camera condition aligns poses and intrinsics with video frames, then route adapters map the motion into each backbone’s expected representation.The common condition preserves the same underlying camera motion across routes.
  • Camera interface: Fused-PRoPE injects camera motion into query, key, and value attention rotations rather than using a separate condition token or camera branch.Native video RoPE is applied first, followed by projective rotations and the matching output transform.
  • Backbone adaptation: The framework preserves each backbone’s native temporal grid, text representation, attention mask, training objective, and essential settings.The shared interface standardizes access without erasing backbone-specific semantics.
  • Backbone-specific contracts: Wan2.2-5B uses a 48-channel VAE latent with the first latent as a clean image anchor excluded from video loss.The remaining latents are denoised as video targets under the shared camera condition.
  • Backbone-specific contracts: Wan2.2-14B becomes one dense model over all timesteps, with an explicit image condition and a noisy, supervised first target latent.The released two-expert routing rule and noise boundary are not retained.

7. Experiments

SolarWM evaluates bidirectional pretrained models and distilled causal models across in-domain, OOD, minute-scale, and hour-scale settings. The qualitative results test camera following, appearance generalization, scene consistency, and sustained autoregressive generation.

  • Bidirectional OOD evaluation: Four bidirectional backbone routes generate 10-second OOD videos from synthesized first frames under prescribed translation, rotation, and mixed 6-DoF camera paths.This stage tests transfer of the shared data and camera contract across unseen visual domains.
  • Causal evaluation protocol: Causal evaluations use one image, a fixed text prompt, and a frame-aligned camera trajectory, with later frames conditioned on model-generated history.The reported causal results cover 10-second, minute-scale, and hour-scale horizons at 16 fps with four sampling steps.
  • In-domain evaluation: In-domain tests include third-person 10-second rollouts that assess prescribed camera following, subject appearance, and spatial relations to the surrounding scene.Minute-scale in-domain sequences cover real-world, synthetic, and game scenes under compound camera paths.
  • OOD evaluation: OOD stress tests synthesize initial images spanning stylized, surreal, unusual, and dramatically lit scenes, then apply multiple camera trajectories to separate appearance shift from control responsiveness.The 10-second rollouts require newly generated video beyond the externally supplied still image.
  • Long-horizon generation: Minute-scale OOD rollouts preserve principal layouts and appearances while revealing compatible new regions and continuing to follow compound camera motion beyond the 5-second training sequences.Five visually distinct scenes are evaluated without resetting the model.
  • Long-horizon generation: 60-minute endpoints remain recognizable and visually coherent while generated views continue responding to camera control in uninterrupted hour-scale rollouts.These evaluations start from real first frames in the held-out validation pool, despite training only on 5-second sequences.

8. Conclusion

SolarWM combines a reconfigurable multi-source data engine, backbone-native adaptation, and a unified training recipe for heterogeneous video world models. It yields camera-controllable causal models supporting real-time interaction and hour-scale rollouts after training on 5-second sequences, while releasing the resources needed for reproducible research.

  • SolarWM combines a reconfigurable multi-source data engine, backbone-native adaptation, and a unified three-stage training recipe.
  • Four camera-controllable causal models span 5B–33B parameters and support real-time interaction with continuous hour-scale rollouts.These models are trained on only 5-second sequences.
  • Releasing the data, processing pipeline, training recipes, model weights, and training-and-inference framework provides a reproducible and extensible basis for future research.
Loading 2609.02886v1…