PaperlayerView exact revision on arXiv

Source-linked AI summary

OctWorld: Long-Range World-Consistent Video Generation with Octree-Based 3D Mapping

Zelong Lv, Sicheng Xu, Jianfeng Xiang, Ruicheng Wang, Yue Dong, Yu Deng, Guangzhong Sun, Jiaolong Yang

arXiv:2609.03919v1cs.CV

TL;DR

Long-range video generation must preserve spatial consistency across extended camera paths, especially when scenes are revisited. OctWorld uses a persistent, adaptive 3D memory to condition chunk-wise video generation, and experiments report better results than prior methods and alternative memory representations. Its demonstrated generation scope is static scenes; dynamic-object memory remains unsupported.

  • Problem

    Long-range generation with extended camera paths remains underexplored, particularly the challenge of keeping revisited regions geometrically and visually consistent.

    Source: §1 · p. 2

  • Method

    OctWorld combines chunk-wise autoregressive video generation with OctMap, an extensible global 3D memory that fuses generated RGB frames and depth observations using an adaptive octree-based TSDF.

    Source: §1 · pp. 2–3

  • Results

    OctWorld outperforms prior methods on existing benchmarks and longer-range generation cases, and OctMap performs better than point-cloud and naive TSDF alternatives.

    Source: §1 · p. 3

  • Takeaways & Limitations

    OctMap is designed to preserve geometric and appearance detail across diverse scene scales while maintaining low memory overhead.

    Source: §1 · p. 3

    OctWorld demonstrates long-range generation for static scenes but does not yet support generating or retaining dynamic objects.

    Source: §D.2 · p. 29

Abstract

from arXiv · show

We present OctWorld, a video diffusion framework with persistent 3D memory for generating explorable, world-consistent, and high-fidelity visual scenes. Given a single image, OctWorld performs stable autoregressive world generation along user-specified camera trajectories. We focus on long-range generation, characterized by extended camera paths and wide viewpoint coverage, where preserving spatial consistency is particularly challenging when previously generated regions are revisited. To address this problem, we introduce OctMap, an extensible and spatially adaptive 3D memory that progressively fuses generated visual observations and their corresponding depth maps into a global representation. OctMap employs TSDF fusion within a dynamic sparse octree whose spatial resolution adapts to image evidence. This design preserves geometric and appearance details across diverse scene scales while maintaining low memory overhead. Experiments demonstrate that OctWorld generates long-range, spatially consistent videos and outperforms prior methods on both existing benchmarks and challenging long-range generation settings. OctMap also provides clear advantages over point-based caches and fixed-resolution TSDF volumes. Project page: https://maxtirerror.github.io/octworldpage/

1 Introduction

OctWorld addresses the limited long-range scope of prior geometry-aware generation by maintaining a fused global 3D memory across extended camera paths. Its adaptive octree memory supports diverse scene scales, and experiments report stronger long-range generation and memory-representation results.

1 Introduction

  • Long-range generation remains underexplored, and revisiting regions after extended camera travel requires regenerated content to remain geometrically and visually consistent.Earlier memory approaches accumulate point clouds or store RGB frames but lack principled fusion and global integration, leaving them susceptible to geometric drift.Source: §1 · p. 2
  • OctWorld combines chunk-wise autoregressive video generation with OctMap, a fused, extensible global 3D memory updated from generated RGB frames and depth observations.Depth may be jointly predicted or supplied by an external streaming estimator.Source: §1 · p. 2
  • Fixed-resolution TSDF volumes face a scalability-versus-detail trade-off in open-domain scenes with uncertain, potentially evolving spatial scale.High resolution can scale poorly, whereas low resolution can lose scene detail.Source: §1 · p. 3
  • OctMap uses a dynamic sparse octree that subdivides regions when incoming RGB-D evidence shows a voxel is too large to preserve detail.Adaptive resolution is intended to handle diverse scene scales and geometric complexity while maintaining low memory overhead.Source: §1 · p. 3
  • OctWorld outperforms prior methods on existing benchmarks and longer-range generation cases, while OctMap surpasses point-cloud and naive TSDF alternatives.The experiments use large-scale RGB-D sequences derived from public real-world videos, predominantly featuring static scenes.Source: §1 · p. 3

2 Related Work

The related-work discussion contrasts camera-conditioned video methods and memory-based world-consistent generation with OctWorld’s explicit integration of depth into global spatial memory. The training pipeline constructs memory from history frames and renders it as a condition for target views.

Camera-Controlled Video Generation

  • Camera-controlled methods use Plücker rays or related inputs, while OctWorld integrates predicted depth into dynamic 3D memory as a spatial condition for synthesis.The cited passage contrasts OctWorld with approaches that often treat depth as an auxiliary output.Source: §2 · p. 3

Training pipeline

  • In training, history frames form an OctMap that is rendered to target views as spatial conditions, with depth generation optional for joint video-depth generation.The training sequence is split into history and target frames.Source: Figure 2 · p. 4

World-Consistent Video Generation

  • Prior world-consistent methods retrieve stored features or frames, warp and inpaint, or progressively build 3D scenes, while OctWorld incrementally fuses RGB-D observations into global spatial memory.The passage characterizes prior explicit-scene methods as relying on simple aggregation or feature fusion.Source: §2 · p. 4

3 Method

OctWorld generates video in chunks along a user-specified camera trajectory, using a persistent OctMap built from generated RGB-D observations. The method renders that memory into spatial conditions and updates it as generation proceeds.

3 Method

  • Given an initial image and camera trajectory, OctWorld generates video chunk by chunk, integrating each chunk’s RGB frames and depth maps into OctMap.Depth can be jointly generated or obtained from an external streaming estimator; the updated memory is rendered for the next target view.Source: §3 · p. 5

3.1 OctMap

  • OctMap allocates finer voxels nearby and coarser voxels at a distance, with octree subdivision guided by incoming RGB-D evidence.The method samples points along camera rays within the truncation margin and allocates or splits nodes along the octree path before TSDF fusion.Source: §3.1 · pp. 5–6

3.2 World-Consistent Video Generation Network

  • The video-generation network conditions diffusion on rendered world-memory observations, packed history frames, camera inputs, and noise augmentation.OctWorld-1.0 jointly generates RGB and depth on a FramePack backbone; OctWorld-1.5 uses a Wan2.1 14B image-to-video backbone with depth from an external streaming estimator.Source: §3.2 · pp. 7–8

3.3 Inference for Long-range Video Generation

  • At inference, the initial image and estimated depth initialize OctMap, and each trajectory chunk is generated and integrated to produce the final video.The trajectory is divided according to the generation window, with each generated chunk’s RGB frames and depth maps added to memory.Source: §3.3 · p. 8

4 Experiments

OctWorld is evaluated for world generation, spatial memory maintenance, long-range synthesis, and its memory and noise-augmentation choices. Results report strong consistency across camera trajectories and favorable user preferences, while ablations compare memory representations and noise settings.

4.1 Implementation Details

  • Training uses filtered RGB-D sequences from RealEstate10K, DL3DV, and static SpatialVid, with estimated per-frame depth and camera trajectories.WorldScore and Re10K evaluate generation and memory maintenance, respectively; long-range tests use manually constructed trajectories.Source: §4.1 · pp. 8–9

4.2 World Generation Evaluation

  • OctWorld ranks highest in 3D and photometric consistency on WorldScore, while placing second in camera control and style consistency.The evaluation uses 200 static photorealistic scenes and 108 frames per scene; Table 1 reports the benchmark comparison.Source: §4.2 · pp. 9–11, Table 1 · p. 10

4.3 Spatial Memory Maintaining

  • OctWorld achieves the best LPIPS and Chamfer Distance scores on the Re10K reverse-trajectory test, measuring appearance and geometric consistency.The test evaluates revisiting regions by retracing camera paths; qualitative results report minimal artifacts and coherent appearance and geometry.Source: §4.3 · pp. 11–12, Table 2 · p. 12

4.4 Long Range Generation

  • In a user study with 22 participants, OctWorld receives the highest preference, and qualitative comparisons report better consistency and generation quality than Gen3C and VMem.Long-range evaluation uses 17 outdoor scenes and trajectories of up to 433 frames.Source: Table 3 · p. 12, §4.4 · pp. 12–13, Figure 7 · p. 14

4.5 Ablation Study.

  • For world generation, TSDF fusion outperforms point clouds at f = 16, OctWorld improves further, and removing OctMap removes spatial conditioning and reduces performance.Qualitative comparisons report noisy patterns for point clouds, lower generation quality for TSDF, and imprecise camera control without OctMap.Source: §4.5 · pp. 13–14, Table 4 · p. 14
  • Noise augmentation trades spatial detail against generation quality; the selected level of 3 slightly reduces spatial quality while significantly improving 3D consistency.The authors use this setting in all experiments.Source: §4.5 · pp. 13–14
  • OctMap achieves better reprojection quality than point-cloud and TSDF alternatives while keeping processing time and memory comparable to TSDF at voxel size 0.002.The comparison uses 100 SpatialVID scenes; point-cloud fusion has the lowest PSNR and SSIM, while smaller TSDF voxels increase memory consumption.Source: §4.5 · pp. 14–15, Table 5 · p. 15

5 Conclusion

OctWorld combines video diffusion with an extensible, spatially adaptive octree-based TSDF memory for long-range world-consistent generation. The paper reports stable generations and memory advantages over point-based caches and conventional TSDF volumes, alongside efficiency analyses and additional baseline comparisons.

5 Conclusion

  • OctWorld integrates generated RGB-D frames into a global, spatially adaptive octree-based TSDF representation for scalable scene fusion across variable spatial scales.The conclusion reports stable, consistent long-range videos and advantages over point-based caches and conventional TSDF volumes in memory efficiency.Source: §5 · p. 15

Appendix

  • The appendix separates OctWorld-1.0 analyses and settings from further method details and results for OctWorld-1.5.The described failure-case, limitation, and future-work discussion is in Appendix D.Source: Appendix · p. 20

A.1 OctMap Inference Efficiency Analysis

  • OctMap’s integration and mesh-extraction times are averaged per 36-frame chunk across 17 long-range sequences, with peak CUDA memory and allocated nodes also recorded.The sequences contain 505 frames each, according to the table caption.Source: §A.1 · p. 20, Table I · p. 20

A.2 OctWorld Inference Efficiency Analysis

  • OctWorld remains competitive in model size, inference time, and runtime memory; its spatial memory module adds 4–6 FPS of overhead.The video-generation backbone remains the dominant inference-time component.Source: §A.2 · p. 20, Table II · p. 20

A.3 Additional Method Comparison

  • Additional qualitative comparisons report that InvisibleStitch struggles on long trajectories, OASIS drifts on real-world scenes, and HYGameCraft struggles with precise camera control.OctWorld maintains geometric alignment with explicit trajectory conditioning.Source: §A.3 · p. 21, Figure I · p. 21

A.4 Additional Experiments on First-Frame Depth Initialization

The depth-initialization experiments compare MoGe, Pi3, and noisy MoGe inputs. OctWorld corrects noisy inputs within one chunk, and initial depth is treated as a controllable input.

A.4 Additional Experiments on First-Frame Depth Initialization

  • OctWorld corrects noisy initial depth within one chunk, producing consistent visual outcomes without requiring flawless initial estimates.The comparison visualizes RGB, spatial memory, and depth before and after one chunk for MoGe, Pi3, and noisy MoGe with σ = 0.05.Source: §A.4 · p. 21, Figure II · p. 22
  • First-frame depth provides a precise 3D spatial scale and is treated as a controllable user input.The framework could be adapted to predict initial depth for an end-to-end system.Source: §A.4 · p. 21

A.5 More Long-Range Generation Results

Additional results demonstrate OctWorld’s long-range generation across diverse scenes and camera trajectories. The reported videos reach 433 frames while preserving spatial memory.

A.5 More Long-Range Generation Results

  • OctWorld produces 433-frame, 3D-consistent videos that navigate diverse scenes while preserving spatial memory.The appendix points to supplementary video for detailed visual results.Source: §A.5 · p. 22
  • The OctWorld-1.0 architecture diagram identifies denoising targets, diffusion timestep, noise injection, history frames, and the patchify embedder.Activation and normalization layers are omitted for clarity.Source: Figure IV · p. 23

B.1 OctWorld-1.0 Architecture Details

OctWorld-1.0 extends a pretrained video diffusion model with OctMap conditioning and training adaptations. The appendices specify data preparation, adaptive scene mapping, and evaluation protocols.

B.1 OctWorld-1.0 Architecture Details

  • OctWorld-1.0 extends FramePack with depth-prediction channels, a control branch conditioned on spatial-memory renderings, and noise-level augmentation.The RGB and depth projections are injected through added DiT blocks; the noise level enters the timestep embedding through a dedicated conditioning layer.Source: §B.1 · p. 23

B.2 Model Training

  • Zero-initialized conditioning pathways preserve the pretrained FramePack backbone at training start, then incorporate new conditions gradually.The added weights and the control-branch projection are zeroed before training.Source: §B.2 · p. 24

B.3 Data Selection

  • Training data are filtered for reliable geometry, scene stability, and illumination, while dataset selection excludes many dynamic scenes and human content.Filtering includes depth-confidence and scene-transition thresholds, then removes segments with maximum luminance change above 22.Source: §B.3 · pp. 24–25

B.4 OctMap 3D Mapping

  • OctMap initializes within a canonical cube, extends its bounds as needed, and splits nodes when finer spatial resolution is required.Root extension preserves existing features with O(1) memory and computational overhead; child nodes initialize from parent weights and features by nearest-neighbor interpolation.Source: §B.4 · p. 25

C.1 World Generation

  • Long-range evaluation uses 200 photorealistic static WorldScore scenes with camera motion magnitude of at least three directional changes.Images and trajectories are standardized across methods; Voyager is reported using single-chunk results because it lacks an official autoregressive inference pipeline.Source: §C.1 · pp. 25–26, Table III · p. 26

C.2 Spatial Memory Maintenance

  • Spatial-memory evaluation starts from the 80 longest RealEstate10K test clips and compares forward/reversed reconstructions using Chamfer Distance.The user-study interface presents the input, prompt, and camera-path description alongside three randomized videos of equal duration.Source: §C.2 · p. 26, Figure V · p. 27

C.3 Long Range Generation

  • Participants complete 20 comparisons, with preference reported as each method’s share of votes; the two trajectories contain 433 and 505 frames.The reported results use the correct trajectory lengths despite an inaccurate main-text description.Source: §C.3 · p. 27

C.4 Ablation Study

The ablation appendix describes point-cloud and TSDF fusion procedures and a failure case involving locally processed camera trajectories.

C.4 Ablation Study

  • Point-cloud fusion filters unreliable depth boundaries and rejects points inconsistent with existing views, yet drift still accumulates after several chunks.Incoming depth maps are warped into existing views to identify redundant or inconsistent points.Source: §C.4 · pp. 27–28
  • The TSDF ablation initializes a fixed-resolution volume from normalized depth and the bounding box of camera frustums.The appendix specifies λsize = 0.002 for Fig. 3 visualization and λsize = 0.007 for Table 4 quantitative evaluation.Source: §C.4 · p. 28

D.1 Failure Case

  • When a camera path intersects or is occluded by previously generated objects, local-chunk trajectory conditioning can produce visual drifting artifacts.The appendix suggests injecting global camera trajectories beyond a single chunk as a possible mitigation.Source: §D.1 · p. 28

D.2 Limitations and Future Work

The paper identifies OctMap scalability and dynamic-scene support as limitations, and describes a depth-free OctWorld variant with streaming depth estimation.

D.2 Limitations and Future Work

  • OctMap lacks coarse-node features, preventing view-dependent level-of-detail rendering that could aggregate fine voxels for distant or homogeneous regions.The appendix also identifies voxel pruning as necessary to discard unobserved or redundant voxels.Source: §D.2 · p. 28
  • A trajectory intersecting or occluded by previously generated objects can cause visual drifting artifacts under local-chunk trajectory conditioning.The example is a doorway incorrectly rendered as a solid wall.Source: Figure VI · p. 29
  • Because all OctMap nodes remain cached on the GPU, memory consumption rises as the scene expands.The paper identifies view-dependent memory offloading as a direction for addressing this scalability bottleneck.Source: §D.2 · p. 29
  • OctWorld’s quality depends on training data and its backbone; the current system handles static-scene generation but not dynamic-object generation or memory retention.The paper identifies 4D scene modeling and dynamic world memory as open challenges.Source: §D.2 · p. 29

E A Depth-free Model with Enhanced Capabilities

  • OctWorld-1.5 is a depth-free variant built on the Wan2.1 14B image-to-video backbone, evaluated on WorldScore and RealEstate10K.It uses a decoupled depth-estimation pipeline, backbone-specific conditioning, and an updated training pipeline.Source: E A Depth-free Model with Enhanced… · p. 29

E.1 Streaming Long-Range Depth Estimation

  • OctWorld-1.5 uses streaming Depth Anything 3 for long-range depth estimation, requiring fewer fine-tuning steps and avoiding possible degradation from joint depth prediction.Source: §E.1 · p. 29

E.2 Additional Architecture Details

OctWorld-1.5 uses Wan2.1 14B’s pretrained conditioning interface for partial OctMap renders, with camera and history inputs. It also introduces token-wise noise conditioning and jointly trains the FramePack history module.

E.2 Additional Architecture Details

  • OctWorld-1.5 performs better on every WorldScore metric except 3D and photometric consistency, where the authors attribute lower scores to a text-alignment trade-off.Dense descriptions in the static split can encourage overly compact, spatially implausible layouts.Source: Table IV · p. 30
  • OctWorld-1.5 achieves better LPIPS at every history noise level and performs well with inference noise level 0, unlike OctWorld-1.0, which relies on inference-time noise.The authors use inference noise level 0 for OctWorld-1.5 in WorldScore experiments.Source: Table V · p. 30
  • Rendered OctMap observations use Wan’s pretrained image-and-mask conditioning interface as y and m, alongside camera and temporal-history conditions.The partial RGB observation is encoded in Wan’s latent space, and its visibility mask is downsampled to the latent resolution.Source: §E.2 · p. 30
  • This design reuses Wan’s learned conditioning mechanism rather than adding a separate branch for rendered world-memory observations.The authors state that this lets the model exploit Wan2.1 14B’s generative prior while remaining compatible with partial renders.Source: §E.2 · pp. 30–31
  • FramePack is reinitialized and trained jointly with the other conditioning modules in one stage, without an intermediate training stage.Source: §E.2 · p. 31
  • Token-wise AdaLN modulation assigns noise level 0 to clean anchor tokens, α to history tokens, and t to target tokens.The conditioning timestep directly corresponds to each token’s applied noise level.Source: §E.2 · p. 31

E.3 Data and Training Pipeline

OctWorld-1.5 is trained exclusively on reprocessed DL3DV data, with preprocessing aligned between training and inference. The authors report strong qualitative results after 7,000 optimization steps.

E.3 Data and Training Pipeline

  • DL3DV clips are standardized to 15 fps and capped at 961 frames, with partial observations and masks preprocessed as at inference time.This is intended to eliminate spatial-conditioning discrepancies between training and testing.Source: §E.3 · p. 31
  • After 7,000 optimization steps, OctWorld-1.5 produces strong qualitative results; training used FSDP on 16 NVIDIA H100 GPUs with global batch size 64.Source: §E.3 · p. 31
Loading 2609.03919v1…