PaperlayerView exact revision on arXiv

Source-linked AI summary

Holo-World: Unified Camera, Object and Weather Control for Video World Model

Xiangchen Yin, Wenzhang Sun, Jiahui Yuan, Zijie Liu, Yinda Chen, Wei Li, Dachun Kai, Chunfeng Wang, Xiaoyan Sun

arXiv:2606.20083v3cs.CV

TL;DR

Existing video world models rarely unify camera, object, and weather control from a single image. Holo-World introduces unified source-to-state supervision and factorized scene-weather control, preserving controlled scene structure while transferring weather and outperforming video-to-video baselines on weather-state generation.

  • Problem

    Video world models rarely unify camera, object, and weather controls from a single image while preserving the observed world and transferring weather state.

  • Method

    Holo-World uses HoloStateData and factorized preservation- and weather-oriented residual subspaces with decomposed guidance for single-image source-to-state video generation.

  • Results

    86.00% Weather Alignment and 68.51 VLM Evaluation were achieved, outperforming video-to-video methods across all weather metrics while preserving controlled scene structure.

  • Takeaways & Limitations

    Holo-World supports unified camera, object, and weather control from a single image while preserving background and scene structure during weather transfer.

  • Takeaways & Limitations

    The paper’s scope is controllable video generation rather than a full physical simulator.

Abstract

from arXiv · show

Video world models are moving toward preserving an observed world under controllable camera and object motion while allowing its environmental state to change. Yet these controls remain isolated, and weather generation typically relies on a source video or reconstructed scene that already specifies future structure. We study a first-frame-anchored source-to-state setting, where the model starts from a single image and follows explicit camera and object controls and an optional weather instruction, then generates a video that either preserves the source world or transfers it to a target weather state. To address these challenges, we first build HoloStateData, a state video dataset that turns diverse videos into unified control samples for camera, object, and weather supervision. Second, we introduce Holo-World, a unified controllable video world model that jointly controls the scene from a single image. Its Unified Scene Adapter factorizes world preservation and weather transfer into distinct parameter subspaces, using rendered background, geometry buffers, and object controls to maintain controlled scene structure while modeling weather-dependent appearance and particle effects. Additionally, Scene-Weather Decomposed CFG guides scene and weather residuals separately, strengthening target weather effects without over-amplifying the full condition. Quantitative and qualitative experiments demonstrate that Holo-World maintains precise camera and object controls with consistent scene structure while transferring scenes into diverse target weather states, outperforming video-to-video weather editing baselines on weather-state generation. Our project page is available at https://xiangchenyin.github.io/Holo-World/

1 INTRODUCTION

The introduction frames unified video world modeling as controlling camera motion, object dynamics, and weather state from a single image. It presents HoloStateData and Holo-World as the dataset and model contributions addressing shared supervision and joint control.

  • Motivation: Current video world models are moving toward dynamic, controllable worlds that specify scene viewpoint, object motion, and world state.The introduction distinguishes structured world control from merely generating visually plausible videos.
  • Problem: The paper studies first-frame source-to-state generation, synthesizing future frames from one image under explicit camera and object controls while rendering a target weather state.Weather is treated as part of the same control interface rather than as a separate capability.
  • Data obstacle: Existing control methods provide separate signals but lack training samples organized around a shared camera-object-weather interface.The stated data obstacle concerns jointly aligning camera pose, background geometry, and other control signals.
  • Modeling obstacle: The model must preserve background appearance and geometry while transferring weather state, because entangled adapter parameters can suppress weather changes or disrupt scene consistency.Real videos supervise source appearance and geometry, whereas paired weather videos supervise transfer under matched structural controls.
  • Contributions: HoloStateData organizes diverse videos into unified samples aligned with camera poses, geometry anchors, object controls, scene text, and weather-state supervision, while Holo-World jointly controls camera, objects, and weather from one image.The contributions explicitly formulate unified state control and introduce the dataset and unified controllable video world model.

2 RELATED WORK

Prior work develops video world models for dynamic scene evolution, increasingly incorporating explicit 3D structure for exploration and navigation. Related controllable-generation methods separately address camera and object motion or weather editing, often relying on structural inputs such as trajectories, reconstructed scenes, or source videos.

  • Video world models: Video world models predict future world states, with recent approaches using transformer or diffusion backbones for action-conditioned generation.Geometry-aware methods additionally incorporate explicit 3D reconstruction for world exploration and navigable scene generation.
  • Video generation with camera and object motion control: Camera- and 3D-aware video methods inject trajectories, poses, rendered geometry, or learned camera motion to enable viewpoint control.Controllable video generation has progressed from text-only conditioning toward explicit structural controls.
  • Weather editing and environment-state generation: Weather methods synthesize or edit environmental effects including rain, snow, fog, wet surfaces, snow coverage, and overcast skies.Some treat weather control as video-to-video editing, where a full source video already specifies future layout, motion, and temporal continuity; others rely on reconstructed 3D scenes.

3 HOLOSTATEDATA

HoloStateData represents clips as first-frame-anchored source-to-state records with factorized camera, object, geometry, and weather supervision. Its construction combines diverse video sources, automatic annotations, rendered controls, and target-weather videos while preventing target-weather pixels from entering model-facing controls.

  • Source-to-state representation: Each record uses the first frame and source-side controls to define the observed world, with optional target video and weather prompt specifying the synthesized scene state.Source controls include camera-conditioned background motion, static geometry anchors, and object boxes; the interface supports both background preservation and weather transfer.
  • Construction pipeline: HoloStateData is constructed through data collection, video annotation, and scene construction, producing rendered background controls, geometry buffers, object controls, and target-weather supervision.The pipeline gathers real, simulated-weather, and V2V weather-transferred videos, then extracts factorized annotations and renders source-side controls.
  • Construction pipeline: Target-weather pixels are never used to render model-facing controls; they serve only as the supervised target video for weather-state transfer.This separation preserves source-side control conditioning while retaining paired supervision for the target weather state.
  • Automatic annotation: For each retained 81-frame clip, automatic annotation generates factorized text, object, and geometry controls while separating scene descriptions from target-weather prompts.Qwen3-VL produces scene prompts covering identity, objects, actions, temporal evolution, and camera motion while excluding weather; Simulation and V2V samples additionally receive weather prompts.
  • Data subsets: The dataset organizes its sources into Real, Simulation, and V2V subsets that respectively provide source-world stabilization, aligned synthetic weather pairs, and real-video weather-transfer exposure.The Real subset lacks a weather-edit target, Simulation shares scene structure across weather states, and V2V complements synthetic pairs with real trajectories.
  • Metrics: Real-subset evaluation measures video quality, camera and object-control errors, and background preservation using VBench-I2V, RotErr, TransErr, ObjMC, PSNR, SSIM, and LPIPS.Background metrics compare generated frames with rendered RGB on valid static regions to assess controlled-background stability.

4 HOLO-WORLD

Holo-World uses a first-frame-anchored source-to-state interface that jointly preserves controlled world structure and transfers the scene to a target weather state. Its Unified Scene Adapter and Scene-Weather Decomposed CFG factorize preservation and weather residuals during training and inference.

  • Unified Scene Adapter: UniSA injects preservation-oriented and weather-oriented residuals into a frozen Wan backbone through parameter-disjoint World and State Adapter subspaces.The World Adapter preserves camera, background geometry, and object layout; the State Adapter learns weather transfer from the source appearance anchor and target-weather text.
  • Scene-Weather Decomposed CFG: Scene-Weather Decomposed CFG separates sampling into scene and weather residuals, using low scene guidance to preserve the source and a state-scaled weather term to strengthen target changes.Real samples suppress the weather residual, whereas Weather samples concentrate concrete weather changes in it.
  • First-frame source-to-state interface: Holo-World represents source-side world controls and source appearance separately from scene and target-weather prompts in a first-frame source-to-state interface.Cworld includes rendered RGB, depth, normal, and object-control videos, while Cstate uses rendered source RGB as the appearance anchor.
  • Unified Scene Adapter: Rendered background, depth, and normal controls anchor camera-conditioned structure, while object-control videos constrain dynamic entities and reduce structure drift during weather transformation.The weather prompt defines the target scene state relative to the source appearance anchor.
  • Scene-Weather Decomposed CFG: Ordinary classifier-free guidance entangles scene and weather effects, so increasing its single scale can overdrive no-edit samples with color shifts, over-saturation, or texture changes.The mixed direction combines scene and weather semantics, appearance shifts, adapter effects, and control-text interactions.

5 EXPERIMENTS

Experiments evaluate Holo-World on held-out Real and Weather subsets under first-frame-anchored control, showing strong world preservation and weather transfer despite stricter inputs than video-to-video baselines. Ablations further support geometry anchors, UniSA, and Scene-Weather Decomposed CFG as complementary components.

  • Benchmark: The held-out HoloStateData benchmark contains 150 curated samples split evenly across Real, Simulation, and V2V subsets.Each subset contains 50 samples and is stratified by source type and evaluation goal.
  • World Preservation: On Real, Holo-World achieves the best overall VBench-I2V score while maintaining strong subject and background consistency under camera and object controls.Specialized baselines remain competitive on some perceptual submetrics, motivating evaluation beyond video quality alone.
  • Weather Transfer: On Weather, Holo-World reaches 86.00% Weather Alignment and 68.51 VLM Evaluation, outperforming video-to-video methods on all reported weather metrics from only the first frame.Unlike the V2V baselines, it cannot inherit future layout, motion, or temporal continuity from a complete source video.
  • Model Design Ablation: Ablations show that G-buffer controls improve background preservation and weather controllability, while UniSA further improves all three background metrics and VLM Evaluation but slightly lowers Weather Alignment.The results indicate that explicit geometry anchors support weather effects while preserving the controlled scene, whereas architectural separation can make visible weather effects more conservative.
  • Scene-Weather Guidance: Scene-Weather Decomposed CFG increases Weather Alignment and VLM Evaluation while keeping Real-subset background metrics close to the no-CFG setting, unlike vanilla CFG.Vanilla CFG improves weather metrics only modestly and degrades background preservation.

6 CONCLUSION

The paper presents Holo-World as a unified model for camera, object, and weather control from one image, combining source-to-state supervision with separated training and sampling mechanisms. Experiments show scene preservation during diverse weather transfer and improved performance over video-to-video baselines.

  • 6 CONCLUSION: Holo-World enables unified camera, object, and weather control from a single observed image.HoloStateData provides source-to-state supervision, while UniSA and Scene-Weather Decomposed CFG implement the separation during training and sampling.
  • 6 CONCLUSION: Experiments show that Holo-World preserves background and scene structure while transferring diverse weather states.
  • 6 CONCLUSION: Holo-World outperforms video-to-video baselines on weather-state generation.

A APPENDIX · A.1 FLOW MATCHING OBJECTIVE

This appendix defines Holo-World’s flow-matching objective as transport along a linear path between clean video latents and Gaussian latents. The method preserves Wan2.1-T2V’s latent flow and velocity-prediction objective while introducing conditioning through trainable residual adapters.

  • A.1 FLOW MATCHING OBJECTIVE: Flow Matching learns a time-dependent velocity field transporting samples between the data distribution and a simple prior distribution.The formulation uses z0 as a clean video latent and z1 ∼N(0, I) as a Gaussian latent.
  • A.1 FLOW MATCHING OBJECTIVE: The probability path linearly interpolates between the clean and Gaussian latents as zτ = (1 −τ)z0 + τz1.The interpolation parameter satisfies τ ∈[0, 1].
  • A.1 FLOW MATCHING OBJECTIVE: The corresponding velocity target is constant along this linear probability path.This constant target provides the training signal for the flow-matching model.
  • A.1 FLOW MATCHING OBJECTIVE: Rather than predicting the clean latent directly, the model predicts the velocity of the latent path.Holo-World retains this velocity-prediction formulation.
  • A.1 FLOW MATCHING OBJECTIVE: Holo-World uses Wan2.1-T2V-14B as its pretrained latent video backbone, with a 3D VAE, text encoder, and DiT denoiser.The DiT predicts the flow target over spatio-temporal latent tokens.
  • A.1 FLOW MATCHING OBJECTIVE: Holo-World keeps Wan2.1-T2V’s flow objective unchanged while UniSA injects conditioning through trainable residual adapters in selected Wan DiT blocks.Conditioning includes the first frame, source-side world controls, source appearance anchors, and factorized text conditions; the latent flow path and velocity objective remain unchanged.

A.2 ADDITIONAL CAPABILITY COMPARISON

This section compares controllable video-generation methods by their exposed control axes and evidence requirements rather than generation quality. It contrasts image-to-video methods using rendered geometry with weather-oriented video-to-video editors requiring complete source videos.

  • Capability comparison: Table 6 compares methods by the control axes exposed in their primary inference interfaces, not by generation quality.A checkmark denotes an explicit control axis, while blank cells indicate that the state factor is not independently exposed.
  • Capability comparison: Existing image-to-video methods provide camera or joint camera-object control through 3D-rendered geometry.
  • Capability comparison: Weather-oriented baselines are generally video-to-video editing models that require a complete source video as input.

A.3 ADDITIONAL RESULTS

Additional results show that Holo-World transfers diverse weather states while preserving source-world camera and object controls, scene layout, and temporal evolution. Intermediate visualizations further show geometry-aligned control and spatially grounded weather changes in generated videos.

  • Evaluation setup: Additional evaluations fix the source world, first frame, camera trajectory, rendered geometry controls, and object controls while varying only the target-weather prompt.This setup tests world preservation and weather transfer under a unified source-to-state interface.
  • Multi-weather control: Changing the weather prompt produces distinct rain, fog, cloud, and snow states while retaining the same viewpoint progression and scene layout.Rain darkens scenes and adds wet appearance; fog reduces long-range visibility; clouds shift illumination; snow changes ground and surface appearance.
  • Multi-weather control: Repeated temporal columns remain aligned within each sample, indicating that weather transfer preserves the underlying camera-conditioned world evolution.The weather variants change environmental appearance without overriding the controlled scene progression.
  • Intermediate representations: Rendered RGB, depth, and normal buffers guide background and geometry alignment, preserving road boundaries, building facades, vegetation layout, and object placement over time.Real examples follow rendered controls without unnecessary weather or global restyling.
  • Weather transfer: Weather examples preserve scene structure while changing appearance: fog lowers distant contrast and visibility, whereas snow shifts streets and facades toward a snowy state.Both cases maintain road curvature, guardrails, mountainside boundaries, building geometry, and viewpoint trajectory.

A.4 HOLOSTATEDATA DETAILS

HoloStateData converts videos into first-frame-anchored source-to-state samples with source-world controls and optional target-weather supervision. Its split design and factorized annotations support separate evaluation and training of world preservation and weather transfer.

  • Sample construction: Each sample pairs a source frame with rendered background RGB, geometry buffers, camera motion, and object controls that define the world to follow.Paired-weather samples additionally provide a target-weather video and weather prompt condition.
  • Control-supervision separation: Source controls remain separated from target supervision, preventing target-weather pixels from leaking into model-facing controls while supporting both no-edit preservation and weather transfer.Real videos provide no-edit supervision, whereas paired-weather videos use target-weather clips only as supervised outputs.
  • Dataset splits: The training set contains about 15K samples across Real, Simulation, and V2V subsets, while the held-out benchmark contains 150 samples balanced equally across them.The benchmark separately evaluates no-edit world preservation and target-weather generation through a stratified diagnostic split.
  • VLM text annotation: Qwen3-VL generates separate scene and weather text conditions, with scene annotations describing source content and motion while excluding weather and weather annotations describing only the weather state.The two-prompt design uses low-randomness decoding and keeps scene, object, camera, and weather descriptions factorized.
  • Automatic construction: Automatic construction converts samples into model-facing controls using SpatialVID-HQ source clips for V2V and paired simulated weather videos rendered with Unreal Engine.The weather-transfer model is fine-tuned from Wan2.1-Fun-Control-14B and used with proprietary video-editing models to generate candidate target-weather data.

A.5 EVALUATION OF HOLO-WORLD … A.5.8 BACKGROUND PRESERVATION METRICS

The evaluation uses separate Real- and Weather-subset protocols to assess video quality, control fidelity, weather generation, human alignment, preference, and source-supported background preservation. Weather metrics are restricted to Simulation/V2V edits, while Real-subset metrics evaluate generation and control without weather-state editing.

  • A.5.1 METRIC OVERVIEW: The evaluation separates Real-subset generation and control metrics from Weather-subset weather metrics, with valid weather runs fixing splits, manifests, prompts, sampling, and study design.Weather-subset scores are never averaged with Real-subset scores or collapsed into a mixed benchmark.
  • A.5.2 VBENCH-I2V METRICS: Real-subset video quality uses eight official VBench-I2V dimensions, including imaging, aesthetics, dynamics, smoothness, background, subject, and I2V consistency.Scores are averaged over the Real subset and reported on a percentage scale.
  • A.5.3 CAMERA AND OBJECT-CONTROL METRICS: Camera and object-control fidelity compares trajectories recovered from generated videos with target trajectories using shared geometry annotations and Hungarian matching for object identities.Camera fidelity uses rotation and translation errors, while unmatched target objects receive a fixed penalty; evaluation covers specified person, vehicle, and animal classes.
  • A.5.4 WEATHER ALIGNMENT: Weather Alignment is a binary pass-rate metric that checks whether the target core weather family is clearly visible while excluding scene, object, camera, and general-quality factors.It evaluates 100 Simulation/V2V samples and may inherit vocabulary or calibration bias when Qwen3-VL is used for annotation and evaluation, motivating GPT-based validation.
  • A.5.5 VLM EVALUATION: VLM Evaluation assigns a 0–100 weather-generation score covering weather-background editing, weather dynamics, and video quality rather than camera or object-control accuracy.The metric does not reward unchanged backgrounds, because the Weather subset evaluates weather-state generation rather than background copying.
  • A.5.6 HUMAN-METRIC ALIGNMENT: Human-metric alignment computes Pearson correlations between automatic weather scores and averaged ratings from five annotators across 100 shared Weather-subset samples.Human ratings cover weather correctness, weather dynamics, and geometry preservation under generated weather.
  • A.5.7 USER STUDY: The user study presents two randomly ordered, anonymized generated videos and reports separate pairwise preferences for Holo-World against Cosmos-Transfer2.5 and Wan2.7-Edit.Participants judge natural weather generation and weather-background editing, not camera or object accuracy.
  • A.5.8 BACKGROUND PRESERVATION METRICS: Real-subset background metrics compare generated frames with rendered reference backgrounds on shared valid, source-supported pixels, excluding placeholders and unrendered regions.Higher Background PSNR/SSIM and lower Background LPIPS indicate better preservation under the no-edit contract.
Loading 2606.20083v3…