PaperlayerView exact revision on arXiv

Source-linked AI summary

UniWorld: A Unified World Model for Real-Time Visual World Simulation

Xin Zhang, Yabo Chen, Shuaiwei Wang, Zixuan Duan, Haibin Huang, Feng Xu, Xuelong Li

arXiv:2610.32381v3cs.CV

TL;DR

Continuous visual world simulation must couple changing geometry with persistent appearance and preserve consistency across finite sampling windows. UniWorld combines depth trajectories and reference appearance in a shared generator, coordinating visual and geometric histories. Its evaluations span multiple depth sources and settings, with measured steady-state DiT execution achieving real-time throughput.

  • Problem

    Continuous rendering must transform evolving geometry into temporally consistent visual observations while preserving the depth conditions associated with earlier observations across finite sampling windows.

    Source: §1 · p. 2

  • Method

    UniWorld uses image-aligned depth and reference appearance in a shared generator, with noise-aligned overlapping continuation and rolling depth state to coordinate visual and geometric histories.

    Source: §1 · p. 2

  • Results

    Across multi-scenario generation, fixed-window interventions, and long-horizon trajectories, the measured steady-state DiT path achieves real-time throughput while maintaining bounded active state.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    One shared generator connects estimated depth, native simulation data, and on-demand world trajectories through a common rendering interface.

    Source: §1 · p. 2, §5 · p. 10

Abstract

from arXiv · show

Continuous visual world simulation requires coupling evolving scene geometry with temporally persistent appearance. We present UniWorld, a unified framework that integrates depth trajectories and reference appearance within a shared video generation model. The central design maintains visual and geometric histories on a common timeline. Noise-aligned overlapping continuation places generated history at the prescribed sampling noise level and commits only newly generated latents. Rolling depth latent state preserves the corresponding geometric controls, incrementally reads newly available geometry, and keeps the conditioning fixed within each sampling window. Together, these mechanisms ensure exact preservation of latent history while bounding the active depth state by the window size. We evaluate continuous generation, paired history interventions, and long-horizon behavior across diverse scene categories. The same generator supports estimated, native, and on-demand simulator depth, while cached and incremental execution paths produce bitwise-identical outputs. Appearance changes and geometric control branches operate through the same interface without modifying committed history. The measured steady-state DiT execution path achieves real-time generation throughput, demonstrating the efficiency of UniWorld for continuous visual world simulation. Project page: https://zephipage.github.io/UniWorld.

1 Introduction

Continuous visual rendering must coordinate evolving geometry with persistent appearance while preserving prior outputs across finite sampling windows. UniWorld combines depth and reference appearance in a shared generator, synchronizing visual and geometric histories for continued generation.

1 Introduction

  • Across diverse scene categories, the framework evaluates estimated and native depth, fixed-window interventions, and extended trajectories; its measured steady-state DiT path achieves real-time throughput.An on-demand engine also demonstrates appearance changes and branch-conditioned continuation through the same interface.Source: §1 · p. 2
  • UniWorld represents heterogeneous geometric trajectories as image-aligned depth and combines them with reference appearance in a common generative rendering interface.The interface is intended for recorded scenes, simulators, and on-demand world trajectories.Source: §1 · p. 2
  • Noise-aligned overlapping continuation constrains generated history at each flow time and commits only the newly generated suffix.This preserves prior latents while retaining temporal context for continued generation.Source: §1 · p. 2
  • Rolling depth latent state preserves geometric controls for committed observations, reads new depth incrementally, and bounds active state independently of sequence duration.Depth conditioning remains fixed within each sampling window.Source: §1 · p. 2

2 Related Work

Related work spans interactive visual world models, conditional video generation, and methods for extending video over time. UniWorld complements these approaches with a shared rendering interface and a temporal contract for aligned visual and geometric histories.

2.1 Visual World Models

  • Visual world models connect observations with actions or latent controls, while UniWorld renders appearance from geometry supplied by an upstream simulator.The cited examples include Genie [Bruce et al., 2024] and Astra [Zhu et al., 2026].Source: §2.1 · p. 3

2.2 Conditional Video Generation

  • Conditional video methods provide text, image, structural, camera, or multiple control signals; UniWorld focuses on retaining conditioning inputs and generated history across windows.The implementation builds on H3, Video DeltaNet, Qwen3-VL, and VideoX-Fun Union control branch [MiniMax, 2026, Xi et al., 2026, Bai et al., 2025, aigc-apps, 2026].Source: §2.2 · p. 3

2.3 Long-Video Generation

  • Long-video methods use appearance memory, compressed context, per-token noise, generated histories, or unified noise initialization; UniWorld applies a joint temporal contract to an eight-evaluation flow renderer.The contract aligns noisy video history, overlapping depth latents, and incremental output with the global VAE phase.Source: §2.3 · p. 3

3 Approach

UniWorld generates video from a reference image, optional text, and aligned depth while coordinating visual history, geometric controls, and sampling noise across windows. Its approach combines noise-aligned history preservation with incremental depth state and a shared temporal plan.

3.1 Task Definition and Preliminaries

  • The method takes a reference image, optional text, and aligned depth sequence as inputs, and produces video at a playback rate; windows are generated sequentially while each window remains bidirectional.The stated target generation rate for real-time operation is 14 FPS.Source: §3.1 · p. 3

3.2 UniWorld: A Unified Framework for Visual World Simulation

  • Source adapters and standardizers provide a common depth representation, while original-image/text context and a separate first-frame condition provide appearance conditioning.The reference image supplies the initial first-frame condition; later windows use generated RGB at their start.Source: §3.2 · p. 4

3.3 Noise-Aligned Overlapping Continuation

  • The inherited video prefix is aligned with the current flow noise before each model evaluation, and only the newly generated suffix is retained for commitment.After each update, the prescribed history trajectory replaces the predicted prefix; at the terminal flow time, it exactly recovers the inherited history.Source: §3.3 · pp. 4–5

3.4 Rolling Depth Latent State

  • Rolling depth retains canonical latents for overlapping positions, reads only newly requested positions, and keeps the depth condition fixed during model evaluations.Its active-state bound covers active canonical depth rows, not transient copies, fused features, model activations, or source-owned storage.Source: §3.4 · pp. 5–6

3.5 Joint Generation Algorithm

  • A single global temporal plan aligns video history, depth state, and noise positions to combine rolling depth, history preservation, and boundary-image updates.The plan keeps the three state streams on corresponding latent-time positions.Source: §3.5 · p. 6

4 Experiments

The experiments evaluate UniWorld’s video quality, continuous generation, history mechanisms, runtime, and long-horizon behavior across multiple scene categories and depth sources. They report a higher VBench Quality Score for DMD8 than H3 50-step, real-time throughput at both tested resolutions, and exact state-consistency checks.

4.1 Experimental Setup

  • The experimental setup uses 100 thirty-second clips, balanced across four scene categories, with VDA-estimated depth and fidelity evaluation against timestamp-aligned GT.Additional studies include 24 sources under nine paired conditions, 25 native-depth inputs, 30 long videos, and three PyBullet conditions.Source: §4.1 · pp. 6–7

4.2 Video Quality Comparison

  • DMD8 achieves the highest VBench Quality Score and four other quality dimensions, exceeding H3 50-step by 0.67 Quality Score points.Table 1 averages 5- and 14-second outputs equally across the same 100 sources.Source: §4.2 · p. 7, Table 1 · p. 7

4.3 Multi-Scenario Continuous Generation

  • One checkpoint completes 100 clips across four categories and 300 windows, with overall PSNR of 13.3973 dB.The same model completes all 25 native-depth inputs by changing only the source adapter.Source: §4.3 · pp. 7–8, Table 2 · p. 8

4.4 Mechanism Analysis and Ablations

  • Full-method replay, rolling/direct reads, aligned depth re-encoding, and online/cached pairs match exactly in all 24 state-consistency cases.In paired fixed-window comparisons, full projection reduces seam LPIPS by 0.00195 relative to boundary RGB only; continuation PSNR and seam temporal-error intervals include zero.Source: §4.4 · p. 8, Table 3 · p. 9

4.5 Runtime and Resource Usage

  • 14.472 new FPS at 960 × 544 and 21.361 new FPS at 672 × 384 meet the 14-FPS target for full-continuation windows.Timings include synchronized H3/VDN and Union forwards with FSDP/Ulysses communication on eight H100 GPUs.Source: §4.5 · p. 9
  • Active depth state occupies 19.050 MiB at 960 × 544 and 9.413 MiB at 672 × 384, independently of trajectory length.Rolling reads fewer depth positions than direct slicing over 30-, 60-, and 120-second outputs while preserving identical output.Source: §4.5 · p. 9

4.6 Long-Horizon Behaviour and Applications

  • Full-video PSNR/LPIPS are 14.4542/0.49032 for 17 sixty-second videos and 14.4277/0.54645 for thirteen 120-second training-split stress tests.Figure 5 reports ten-second aggregates with category-stratified source-group confidence bands; duration groups use different sources.Source: §4.6 · p. 9, Figure 5 · p. 9
  • Three PyBullet conditions test appearance changes, a camera revisit, and a control branch; on-demand and cached depth produce bitwise-identical final latents in all three pairs.The control branch preserves the first window and changes the new suffix; warm/cool references share geometry while producing different appearance.Source: §4.6 · pp. 9–10

5 Conclusion

UniWorld unifies depth trajectories and reference appearance for continuous visual world simulation. Its shared generator supports multiple depth sources while preserving visual history and matching geometric controls with bounded active state.

5 Conclusion

  • UniWorld represents heterogeneous geometric trajectories with image-aligned depth and reference appearance in a common generative rendering interface.The interface covers recorded scenes, simulators, and on-demand world trajectories.Source: §5 · p. 10
  • Its joint state contract preserves committed video latents and matching geometric controls while incrementally reading new depth with bounded active state.Exact replay and cached/online equivalence verify the connection in real model runs.Source: §5 · p. 10
  • The measured steady-state DiT path achieves real-time throughput while maintaining bounded active state for continuous generation.Source: §5 · p. 10

A Proofs and State Invariants

History projection constrains the saved visual prefix throughout sampling, while rolling depth state preserves canonical geometric overlap and limits active storage. These guarantees depend on stated traversal and finite-precision conditions.

A.1 History Projection

  • Each sampler step projects masked history to its noise-aligned value, and at the terminal time the saved prefix is recovered.The global committed prefix is separately preserved by concatenating only new positions; bitwise claims apply to the stated FP32 implementation, not arbitrary precision changes or globally coherent scenes.Source: §A.1 · p. 13

A.2 Depth Overlap and Incremental Reads

  • Identical-value copying preserves overlapping depth values and dtype across windows when source reads return canonical values in the same order and dtype.A failed read or restarted request may reread positions, and transient old-plus-new buffers may exceed one active state.Source: §A.2 · p. 13
  • Under sequential, gap-free traversal, rolling depth reads each canonical position once and keeps active storage bounded by the maximum window length.The active tensor contains at most LmaxS rows of 96 FP32 elements; the storage bound excludes transient copies and model state.Source: §A.2 · p. 13

B Conditioning and Implementation

UniWorld reuses pretrained video, control, and conditioning components, with separate paths for appearance, depth, and video history. Its sampling lifecycle validates and reuses rolling state while keeping window inputs fixed.

B.1 Model Components

  • The 50-block H3 backbone retains bidirectional within-window processing, alongside a separate five-block Union control branch and fixed-width Qwen3-VL image/text conditioning.The implementation uses existing H3, Union, Qwen3-VL, and video VAE components without additional local joint training or a new distillation objective.Source: §B.1 · p. 14

B.2 Independent First-Frame Conditions

  • The first-frame condition uses a fixed noise realization and FP32 arithmetic, while the original Qwen representation remains fixed across windows.The method does not refresh Qwen embeddings from generated boundaries or use ground-truth boundaries during inference.Source: §B.2 · p. 14

B.3 Depth Rows and Union Packing

  • Depth rows use posterior mode, FP32 normalization, and patchification; explicit target-row indices prevent modality tags alone from selecting the wrong rows.The depth tensor and control carrier are reused within a window, while Union hints depending on the noisy target and time are recomputed at each step.Source: §B.3 · p. 14, Table S1 · p. 14

B.4 Sampling and State Lifecycle

  • Sampling projects history before the first model evaluation and after each update, while rolling depth validates new data and commits it atomically for retries.Each window uses eight actual velocity evaluations; retrying the current window reuses the validated depth state.Source: §B.4 · p. 15

C.1 Datasets and Source Selection

The evaluation combines diverse short and long sources with estimated, native, and simulator depth conditions. It reports fidelity, perceptual, temporal, and geometry measures using fixed evaluation protocols and source-group bootstrap intervals.

C.1 Datasets and Source Selection

  • The 100 primary clips contain 96 distinct source groups; ten human-activity clips also appear in the configured 10,000-sample local training set, and overlap checks do not characterize original pretraining data.The 24 ablation sources comprise six distinct source groups per category, selected by a fixed deterministic ordering independent of quality. The 17 sixty-second sources use existing validation/test splits; all 13 120-second sources belong to existing training splits and are reported as stress tests.Source: §C.1 · p. 15

C.2 Depth Representation and Ground-Truth Alignment

  • Native-minus-estimated depth on 25 matched sources changes PSNR by −0.70859 dB and LPIPS by +0.02943, while the reported geometry intervals include zero.Both geometry evaluations use common native ground truth.Source: §C.2 · p. 16, Table S3 · p. 16

C.3 Fidelity, Perceptual, and Temporal Metrics

  • Across the primary set, SSIM is 0.431918, LPIPS is 0.49997, and depth rank correlation is 0.87809.Excluding frame zero gives 13.3915 dB PSNR; the common continuation gives 12.6700 dB PSNR and 0.54241 LPIPS.Source: §C.3 · p. 16, Table S4 · p. 17

C.4 Geometric Metrics and Valid Support

  • Primary geometry compares generated-video VDA output with the VDA conditioning trajectory, while native comparisons use a shared native target but retain generated-side estimator bias.Depth calibration is fitted over the first five seconds; later frames do not refit it.Source: §C.4 · p. 17

C.5 Cluster Bootstrap and Aggregate Intervals

  • Intervals use 10,000 category-stratified source-group bootstrap replicates and marginal 95% percentiles, without family-wise adjustment across exploratory arms and metrics.The procedure estimates variation across sources under a fixed seed per source, not seed-to-seed generation variability.Source: §C.5 · p. 17

C.6 Paired Mechanism Protocol and Additional Results

Paired interventions distinguish exact state-preservation checks from quality measurements across seams and full continuations. Model replays and rolling-versus-direct depth access also satisfy reported tensor and output consistency checks.

C.6 Paired Mechanism Protocol and Additional Results

  • 24/24 boundary-only and endpoint-only variants produce identical committed outputs, and direct slicing and rolling access also match in 24/24 cases.Endpoint restoration cannot change an already generated suffix; independent phase-aligned re-encoding is exact in these cases because the encoder uses independent 17-frame chunks.Source: §C.6 · p. 17, Figure S1 · p. 19
  • The boundary-only seam LPIPS difference is 0.00195 across 24 sources, while its continuation temporal-error difference is −0.000324.The 22-source sensitivity-subset seam interval includes zero; seam behavior and the full generated interval are distinct temporal scopes.Source: §C.6 · pp. 17, 19, Table S8 · p. 18, Table S6 · p. 18

C.7 State Consistency Checks

  • All 24 full-method replays reproduce primary outputs exactly, and constrained history rows have zero maximum absolute error.Rolling and direct depth access also produce bitwise-equal video and audio latents in two 60-second cases; the quality evaluation concerns visual outputs.Source: §C.7 · p. 19, Table S9 · p. 19

C.8 VBench Quality Comparison: Protocol and Paired Results

The comparison evaluates quality on a 100-source collection using paired uncertainty estimates; the measured full-continuation DiT path meets the 14-FPS target at both tested resolutions. Resource measurements distinguish rolling depth-state reads from duration-growing global storage.

C.8 VBench Quality Comparison: Protocol and Paired Results

  • The VBench protocol compares 800 videos from four configurations across 100 sources, aggregates two durations with equal weight, and normalizes dimension scores without clipping.VBench 0.1.5 uses its official custom-video pipeline; source fidelity and depth agreement are evaluated separately.Source: §C.8 · pp. 19–20
  • All six paired Quality Score intervals are positive for DMD8 minus each comparator, using category-stratified source-group bootstrap intervals without multiplicity adjustment.The comparison uses 10,000 replicates and evaluates each duration separately.Source: §C.8 · p. 20, Table S12 · p. 21, Table S13 · p. 21

D.1 Controlled DiT Timing

  • 21.361 and 14.472 new FPS at 672 × 384 and 960 × 544, respectively, meet the 14-FPS target on full-continuation paths.Timing covers synchronized model forward passes, including the H3/VDN backbone, Union branch, and FSDP/Ulysses communication; other pipeline operations are excluded.Source: §D.1 · pp. 22–23, Table S16 · p. 22

D.2 Measured Depth-State and Model Resources

  • At 960 × 544, rolling reads request 217/427/852 unique depth positions for 30/60/120-second deliveries, versus 241/475/960 for direct reads.These are logical source requests; global depth, noise, and committed outputs still grow with duration.Source: §D.2 · p. 23

D.3 Long-Horizon Results

Long-horizon measurements show declining fidelity metrics over time, while online tests report bitwise agreement between cached and incremental outputs. Appearance and control variations, overlap configurations, and paired visualizations probe the same continuous rendering setup.

D.3 Long-Horizon Results

  • For 60-second trajectories, PSNR falls from 18.257 to 13.136 dB and fixed-scale NMAE rises from 0.0580 to 0.1428 across the first-to-last ten-second blocks.The 120-second values change from 18.665 to 13.307 dB and from 0.0903 to 0.2248; the 60-second set has 17 sources, while the 120-second set has 13 stress-test sources from a different split.Source: §D.3 · p. 24, Table S20 · p. 24

D.4 On-Demand Geometry, Appearance, and Control

  • All three online final video-latent outputs match cached outputs bitwise, and newly acquired depth intervals match the cached oracle.Warm/cool generated RGB differs by a mean 0.19322 versus 0.14456 for engine images, while the control pair has identical initial-window latents and suffix latent MAE 0.039076.Source: §D.4 · p. 24, Table S21 · p. 24

D.5 Window and Overlap Configurations

  • The full-video ranking favors 39-frame overlap, while the shared continuation slightly favors shorter overlaps; a separate two-scene comparison showed no consistent PSNR/SSIM advantage from history.These are configuration observations, not isolated estimates of the history constraint.Source: §D.5 · p. 24, Table S22 · p. 25

D.6 Qualitative Selection and Paired Visualizations

  • The paired figures show source RGB beside generated observations at identical timestamps, including continuation boundaries and online variations in reference appearance and future geometry.Figure 4 selects the first ablation source from each category independently of quality scores.Source: §D.6 · p. 25

E AI Use and Reproducibility Statements

The authors report generative AI assistance with manuscript preparation while retaining responsibility for results and the final manuscript. They identify the inference specifications, state proofs, implementation details, and measurement protocols provided for reproducibility.

E.1 AI use statement

  • Generative AI assisted manuscript organization, drafting, editing, formatting, and preparation of tables and figures; the authors remain responsible for the experimental results and final manuscript.Quantitative results and qualitative examples derive from completed model runs, while method diagrams are illustrations.Source: §E.1 · p. 26

E.2 Reproducibility statement

  • Inference is specified in Equations 4–11 and Algorithm 1, with appendices covering state proofs, model components, conventions, controls, statistics, and measurement protocols.Supplementary tables report aggregate results and uncertainty under those protocols.Source: §E.2 · p. 26

Related papers

Loading 2610.32381v3…