PaperlayerView exact revision on arXiv

Source-linked AI summary

PanoWorld: Real-World Panoramic Generation

Haoyuan Li, Dizhe Zhang, Yuemei Zhou, Xiangkai Zhang, Haoran Feng, Xiaofan Lin, Wenjie Jiang, Bo Du, Ming-Hsuan Yang, Lu Qi

arXiv:2607.09661v1cs.CV

TL;DR

Panoramic world models struggle with physically consistent geometry and illumination over space and time, while perspective-based memory can misalign under distortion and rotation. PanoWorld uses rotation-equivariant panoramic rays, DPRC, GMA, and World360 to improve controllable generation, outperforming alternatives in scene persistence and structural integrity.

  • Problem

    Panoramic world models struggle to maintain physical consistency across space and time, while perspective-based memory can misalign under distortion and rotation-induced viewpoint shifts.

  • Method

    PanoWorld leverages rotation-equivariant panoramic representations, decouples rotation from translation, and uses DPRC, GMA, and a three-stage training pipeline for controllable generation.

  • Results

    PanoWorld significantly outperforms existing alternatives in scene persistence, structural integrity, visual quality, and trajectory control on World360.

  • Takeaways & Limitations

    World360 supports evaluation of panoramic world modeling under real-world physical variations, and PanoWorld provides a robust approach for unconstrained panoramic world synthesis.

  • Takeaways & Limitations

    Replicating the input as the first generated frame creates a quality gap from the second frame that accumulates over longer horizons and makes synthesis dependent on the first frame.

Abstract

from arXiv · show

In this work, we aim to address the challenge of long-range memory in panoramic world models by exploiting the rotation-equivariant property of omnidirectional representations, where rotation can be treated as an implicit geometric transformation.Building on this insight, we propose PanoWorld, which simplifies camera trajectories into translations via fixed headings for both current-action modeling and long-range memory through Dense Panoramic Ray-Conditioning (DPRC) and Geometry-aware Memory Augmentation (GMA).Then, a three-stage training pipeline is introduced to progressively optimize each component. To better evaluate physical consistency under large-scale spatial variations and diverse illumination conditions, where existing datasets are relatively stable, we construct World360, a large-scale dataset consisting of both real-world video clips collected via panoramic unmanned aerial vehicles and high-quality simulated clips generated by AirSim360.Extensive experiments on World360 demonstrate the effectiveness of PanoWorld, outperforming alternative methods by a large margin.Our models, training code, and dataset will be publicly available. More information can be found on our project page: https://lihaoy-ux.github.io/panoworld-page/.

1 Introduction

Panoramic world models struggle to maintain physical consistency across space and time because perspective-based memory mechanisms misalign under distortion and rotation-induced viewpoint shifts. PanoWorld addresses this challenge by exploiting ERP rotation equivariance, modeling translation explicitly, and combining DPRC, GMA, and a three-stage training pipeline with the World360 dataset.

  • Problem: Panoramic world models face amplified challenges in preserving geometric and illumination consistency across space and time because they capture the full 360◦ field of view.Equirectangular projections have emerged as a mainstream panoramic representation.
  • Problem: Existing memory mechanisms inherit perspective assumptions, causing misaligned retrieval under severe distortion and rotation-induced viewpoint shifts.These mechanisms include 3D points and KV caches used in 3DGS-based and video-generation paradigms.
  • Method: PanoWorld leverages ERP rotation equivariance by treating rotation as a geometric transformation and modeling only translation explicitly.Rotations mainly alter the distortion pattern while preserving the underlying scene content.
  • Dataset and evaluation: World360 contains 70K real-world clips collected via Anti-Gravity and 50K high-fidelity simulations from the AirSim360 platform.The dataset addresses the limitations of predominantly indoor or simulation-based datasets with relatively stable physical conditions.
  • Method: PanoWorld combines Dense Panoramic Ray-Conditioning (DPRC), Geometry-aware Memory Augmentation (GMA), and a three-stage training pipeline.DPRC provides dense geometric conditioning for panoramic motion modeling, while GMA enhances memory retrieval and spatiotemporal consistency.

2 Related Work

Prior work explores panoramic video generation, camera-controlled diffusion, and memory augmentation, but existing approaches face limitations in structural consistency, motion control, and long-horizon scene persistence. These directions motivate more coherent and controllable panoramic world models.

  • Panoramic Video Generation: Panoramic vision offers a holistic geometric perspective for globally coherent scene synthesis, with WorldPrompter and 360DVD advancing panoramic generation and benchmarking.360DVD introduced the text-to-panoramic video generation pipeline and the WEB360 dataset.
  • Camera-controlled Video Diffusion Models: Camera-controlled video diffusion uses extrinsic parameters, point clouds, or spherical motion conditioning to control camera motion during generation.CameraCtrl and ViewCrafter provide representative forms of controllable generation, while 360DVD applies conditioning in panoramic settings.
  • Memory-Augmented Video Synthesis: Long-horizon video generation suffers from scene drift, while existing memory methods are clip-bound, require full-sequence processing, or remain unsuitable for open-world generation.Context-as-Memory fails to capture long-range dependencies, and Captain Safari improves consistency using implicit 3D features but requires full-sequence processing.

3 Data Pipeline and Benchmark

This section presents a physically grounded data curation engine and introduces World360, a large-scale panoramic world-modeling benchmark covering real-world physical variations and complex 3D paths.

  • Data Pipeline: The data engine enforces physically grounded consistency across temporal, geometric, and illumination dimensions for controllable panoramic video synthesis.It is designed for large-scale data generation and evaluation of panoramic world models.
  • Data Curation: Rotation Decoupling factors out relative rotations and re-projects panoramic frames into a unified heading, making translation the primary driver of visual change.This simplifies learning spatial depth and motion parallax within a panoramic ray-conditioned manifold.
  • World360: World360 contains 120,000 high-quality sequences: 70,000 curated real-world clips and 50,000 high-fidelity AirSim360 simulations.The benchmark is constructed from the proposed curation pipeline for panoramic world modeling under real-world physical variations.
  • World360: World360 supports evaluation of structural integrity across complex 3D paths, bridging constrained modeling and unconstrained panoramic synthesis.It introduces diverse multi-altitude aerial trajectories, unlike existing large-scale real datasets dominated by planar street-level motion.

4 Our Method

PanoWorld simplifies panoramic generation by separating rotation from translation, modeling view-dependent motion with DPRC, and preserving long-range consistency through GMA. Its three-stage training progressively adapts panoramic geometry, learns motion control, and activates memory-anchored synthesis.

  • Core framework: PanoWorld removes rotational ambiguity, injects geometry-aware motion control and historical memory into a unified diffusion backbone, and decodes controllable panoramic video.The framework focuses diffusion modeling on translation-induced parallax while treating rotation as a geometric transformation.
  • Dense Panoramic Ray-Conditioning: DPRC models view-dependent translation by mapping equirectangular pixels to spherical rays and encoding translational parallax in a ray-based motion manifold.The resulting representation is integrated into diffusion transformer blocks through Projective Positional Embeddings on a downsampled latent grid.
  • Geometry-Aware Memory: GMA retrieves historical features through an implicit geometric manifold, using ray correspondence rather than explicit image-space warping to preserve consistency when the camera revisits locations.Confidence-guided adaptive fusion anchors synthesis to high-confidence memory and reduces hallucination, flickering, and structural drift.
  • Stages 2–3: Motion and Memory: Stages 2 and 3 respectively train DPRC for 3D translational control and activate GMA for memory-anchored synthesis with confidence-guided landmark anchoring.The progressive sequence stabilizes optimization and assigns each component its intended role before full integration.
  • Stage 1: Panoramic Video Fine-tuning: Stage 1 uses LoRA and latitude-aware reconstruction to adapt the backbone to equirectangular geometry, learning polar-distortion and horizontal wrap-around consistency.The resulting backbone respects 360° topological constraints and prevents structural tearing and boundary artifacts during synthesis.

5 Experiments

Experiments show that PanoWorld improves panoramic visual quality, trajectory adherence, geometric stability, and scene persistence over competing methods and ablations. The study also extends the model toward real-time generation through causal forcing.

  • Implementation Detail: PanoWorld is fine-tuned from Wan2.2-5B with LoRA under a decoupled multi-stage pipeline that independently optimizes action-conditioning and memory modules.Parameter freezing is used during the independent optimization stages.
  • Evaluation Metrics: The evaluation covers distribution fidelity, regional panoramic structure, visual quality, temporal consistency, clarity, and aesthetic coherence using FID, FIDpole, FIDequ, FAED, NIQE, and Q‑Align metrics.Q‑Align includes QAqual. and QAaes.
  • Baselines: PanoWorld is compared with Imagine360, Matrix-3D, and OmniRoam under unified resolutions and de-rotated camera trajectories from the same evaluation dataset.The reported resolutions are 480p (480 × 960) and 720p (720 × 1440).
  • Qualitative Results: PanoWorld maintains precise trajectory adherence and geometric stability, while Matrix-3D exhibits blurring and structural voids and OmniRoam shows vertical-control failures, ghosting, and artifacts.The qualitative comparisons use real-world outdoor sequences with diverse movement patterns.
  • Quantitative Analysis of Visual Quality: 27.64 FID, 47.21 FIDpole, and 26.00 FIDequ are achieved by PanoWorld, indicating stronger structural integrity across the panoramic manifold.The method suppresses pole distortions and equatorial blurriness that affect other panoramic models.
  • Ablation Study: GMA significantly improves geometric stability and long-term trajectory adherence over w/o GMA and Random Memory variants, with gains increasing alongside sequence length.The full model preserves consistent structures, whereas the ablations produce spatial sliding, blurring, or broken geometry.

6 Conclusion

PanoWorld is a controllable panoramic video generation framework that simplifies motion learning by decoupling rotations through rotation-equivariant omnidirectional representations. Its DPRC and GMA modules target geometric, radiometric, and temporal consistency, supported by the World360 dataset.

  • 6 Conclusion: PanoWorld is a framework for controllable panoramic video generation that decouples rotations to simplify motion learning.The approach leverages the rotation-equivariance of omnidirectional representations.
  • 6 Conclusion: DPRC and GMA modules use panoramic-ray properties to ensure geometric, radiometric, and temporal consistency.These modules are designed around the unique properties of panoramic rays.
  • 6 Conclusion: World360 is a large-scale dataset featuring real-world UAV captures and high-quality simulated content.The supplied passage introduces World360 to support research on panoramic video generation.

Supplementary Material for PanoWorld: Real-World · A Datasets Pre-processing · A.1 Scene-Reality Real Data

The supplementary material details PanoWorld’s real-world Scene-Reality dataset and its preprocessing pipeline. The pipeline synchronizes panoramic video with 6-DoF poses, removes rotational and pose artifacts, resamples trajectories spatially, aligns frames, and creates fixed-length clips.

  • Supplementary Material for PanoWorld: Real-World: Scene-Reality Data comprises large-scale panoramic videos captured in diverse physical environments, exposing models to natural ego-motion and complex visual textures.It is paired with AirSim360-generated synthetic data as the second training dataset.
  • A Datasets Pre-processing: The Scene-Reality Learning Dataset contains 124 long-duration 360° videos, corresponding 6-DoF trajectories, and over 6 million frames across varied outdoor environments.Trajectories are derived from a high-precision GPS-aided Inertial Navigation System.
  • A.1 Scene-Reality Real Data: Temporal alignment anchors the camera exposure start to pose time t0, matches calculated frame timestamps to pose records, and linearly interpolates poses.The camera logs at 59.94 FPS, while the GPS-fused system logs at 200 Hz; leading frames without valid poses are discarded.
  • A.1 Scene-Reality Real Data: Rotational decoupling aligns ENU and camera coordinates, optimizes relative rotations from spherical features, and stabilizes ERP frames using visual-inertial processing.Feature-deficient regions use an INS-prior yaw estimate with latitude-weighted phase correlation, followed by Kalman smoothing.
  • A.1 Scene-Reality Real Data: A two-stage pose-noise suppression strategy mitigates turbulence, motor vibration, GPS multipath, high-frequency noise, and sudden jitter in synchronized UAV trajectories.The stated goal is to reconstruct physically plausible, smooth kinematic trajectories for high-fidelity world modeling.
  • A.1 Scene-Reality Real Data: Spatial normalization addresses variable UAV velocity by computing cumulative three-dimensional Euclidean arc length, explicitly including vertical motion.Using 3D distance prevents planar metrics from distorting trajectories with climbs and descents.
  • A.1 Scene-Reality Real Data: Uniform spatial re-indexing filters near-zero spatial steps, interpolates kinematic states onto the new grid, and compresses redundant static segments.Because rotations were already stabilized, roll, pitch, and yaw can be interpolated without phase-wrapping artifacts.
  • A.1 Scene-Reality Real Data: Temporal blending aligns visual frames to spatial samples, maintaining a constant inter-frame distance of 0.05 m before fixed-length trajectory slicing.Sliding windows extract L1 = 81-frame samples for motion-conditioned generation and longer sequences for geometry-aware memory optimization.

A.2 Trajectory-Precise Synthetic Data … B.2 Action Model Training Pipeline

The paper builds trajectory-precise synthetic data with synchronized panoramic frames and poses, filters videos using semantic and visual-quality criteria, and trains frozen-generator action models in three stages. Its training setup combines WAN2.2-5B with LoRA and synchronized rotational augmentation under constrained batch sizes.

  • A.2 Trajectory-Precise Synthetic Data: The synthetic dataset spans 8 outdoor environments and provides noise-free panoramic frames with perfectly synchronized ground-truth 6-DoF poses.It is sourced from the Air360 simulator to avoid sensor noise, clock drift, and interpolation residues.
  • A.2 Trajectory-Precise Synthetic Data: The trajectory planner voxelizes free space at 1 m × 1 m × 1 m and samples valid altitude layers on a 0.05 m sub-grid.This hierarchy supports efficient planning while accommodating elevation changes and precise spatial-uniform sampling.
  • A.2 Trajectory-Precise Synthetic Data: The simulator generates diverse flight paths through three trajectory modes, smooths anchor-point transitions, and renders videos at a constant spatial increment of ∆s = 0.05 m.The resulting benchmarks contain complex elevation changes and rich geometric constraints.
  • A.3 Prompt Construction and Filtering Pipeline: Text supervision uses sparse 21-frame captioning focused on environmental characteristics, spatial layouts, and ambient illumination rather than camera motion.Descriptions are produced with Qwen3-VL and omit ego-motion and dynamic-object movement.
  • A.3 Prompt Construction and Filtering Pipeline: The filtering pipeline flags under-exposure at µI < 30, low contrast at σI < 20, and over-exposure when Rover > 40%.It also logs failed sequence identifiers in an exposure anomaly catalog.
  • A.3 Prompt Construction and Filtering Pipeline: Remaining frames receive Q-Align perceptual quality scores Q ∈[0, 1], after which exposure-only or dual-attribute filtering produces the clean subset.The two schemes trade rapid throughput against maximum visual purity.
  • B.1 Video Model Training Pipeline: The video model uses WAN2.2-5B with rank-64 LoRA fine-tuning on 70k video-trajectory samples resized to 480 × 960.Training is conducted with DiffSynth-Studio.
  • B.2 Action Model Training Pipeline: The upstream generator remains frozen while the action model trains independently for three stages of 2 epochs, using synchronized rotational augmentation and batch size 1 with gradient accumulation 4.Rotational perturbations diversify sequences and paired 6-DoF poses while maintaining an effective optimization batch size under memory constraints.

B.3 Memory Model Training Pipeline … C Application

PanoWorld freezes the upstream video and action models while progressively training its geometry-aware memory module. Evaluation uses World360 trajectories spanning diverse 3D maneuvers and combines panoramic visual metrics with explicit 6-DoF trajectory reconstruction.

  • B.3 Memory Model Training Pipeline: The upstream video and action models remain frozen while the geometry-aware memory module is optimized independently.This preserves established visual generation and action control capabilities.
  • B.3 Memory Model Training Pipeline: The first training stage performs geometric warm-up on 20k frame pairs with corresponding ground-truth camera poses.The frame pairs are extracted from collected video sequences.
  • B.3 Memory Model Training Pipeline: The second stage uses 50k fixed-length video clips, randomly samples start frames, and builds memory anchors from four earliest frames within [0.2, 2.0] m displacement.Training uses block-wise video data loading and clips of 161 frames.
  • B.5 Inference: Inference benchmarks 90 World360 video-trajectory sequences covering forward, backward, lateral, vertical, arc, and loop maneuvers.The trajectory taxonomy includes up/down translations and complex loop trajectories.
  • B.5 Inference: Evaluation measures visual and geometric fidelity with FVD, FID, FIDpole, and FIDequ across the omnidirectional spherical manifold.The metrics assess structural integrity and projection consistency under equirectangular distortion.
  • B.5 Inference: ViPE reconstructs continuous 6-DoF camera trajectories from generated panoramic frames for trajectory-focused validation beyond pixel-level metrics.Recovered trajectories are aligned and compared against reference trajectories.

C.1 Real-Time Application via Causal-Forcing · D Discussion

PanoWorld is adapted into a causal autoregressive Wan2.2-TI2V-5B generator for controllable panoramic video synthesis, using chunk-wise training and a three-stage Causal Forcing pipeline. Rolling Forcing extends generation beyond the 21-latent-frame training limit, while experiments show strong quality, controllability, and inference-efficiency gains.

  • C.1 Real-Time Application via Causal-Forcing: Qualitative results show that the Causal Forcing framework, advanced to Wan2.2-TI2V-5B, avoids the severe structural blurring and motion inconsistencies affecting previous methods.Given one panoramic image and camera trajectories, the model generates panoramic video with the intended controllable setup.
  • C.1 Real-Time Application via Causal-Forcing: The Wan2.2-TI2V-5B diffusion model is converted from a dual-stream architecture into a causal autoregressive framework via Causal Forcing.This adaptation targets a high-fidelity, controllable interactive environment simulator.
  • C.1 Real-Time Application via Causal-Forcing: The 21 latent frames are partitioned into [1] + [4 × 5], with the initial frame used as an I2V conditioning anchor and the remaining 20 frames trained in 4-frame teacher-forced chunks.The initial frame is excluded from loss computation, while ground-truth and noisy latent sequences are combined to preserve dual-stream processing under causality.
  • C.1 Real-Time Application via Causal-Forcing: Causal Forcing compresses the autoregressive model through autoregressive diffusion training, causal ODE initialization, and asymmetric distribution matching distillation.At 480p, Stage-1 uses 3,000 steps and Stage-3 DMD uses 10,500 steps, producing a 4-step generator.
  • C.1 Real-Time Application via Causal-Forcing: Rolling Forcing extends chunk-wise DMD from 21 latent frames (81 pixel frames) to 41 latent frames (161 pixel frames) using autoregressive rollouts and a sliding-window KV cache.DMD supervision is applied to the last 21 frames, and distillation uses 1,998 pre-encoded panoramic clips.
  • C.1 Real-Time Application via Causal-Forcing: The full model achieves the best scores across almost all reported visual-quality, perceptual-quality, and trajectory-controllability metrics, including FID, FAED, QA, and PSNR.The evaluation covers panoramic video generation and motion controllability in Table 5.
  • C.1 Real-Time Application via Causal-Forcing: The causal-forcing variant incurs only a marginal quality drop relative to the full model while delivering an orders-of-magnitude improvement in inference efficiency.On one NVIDIA H20 GPU, the full model requires 4 minutes and 48 seconds for generation, whereas Matrix-3D and OmniRoam are substantially slower.

D.1 Limitation · D.2 Future Work

The framework accelerates inference by replicating the input image as the first generated frame, but this creates a distribution discrepancy and subsequent quality gap. Future work targets smoother input-to-generation transitions through boundary refinement, latent-space alignment, and related efficiency improvements.

  • D.1 Limitation: D.1 Limitation: Directly replicating the input image as the first generated frame is the framework’s primary bottleneck.The strategy bypasses initial-frame denoising to accelerate inference.
  • D.1 Limitation: D.1 Limitation: Replication introduces a distribution discrepancy between the high-fidelity real input and subsequent model-generated frames.The discrepancy arises from combining a real input with generated continuation frames.
  • D.1 Limitation: D.1 Limitation: A distinct quality gap emerges starting from the subsequent generated frames.The passage attributes this gap to the discrepancy between the replicated input and generated sequence.
  • D.2 Future Work: D.2 Future Work: Lightweight boundary-refinement modules are proposed to bridge the domain gap caused by direct input-image injection.This direction replaces hard replication with a smoother transition into the generated sequence.
  • D.2 Future Work: D.2 Future Work: Latent-space alignment techniques are proposed to reduce the domain gap from direct input injection.The goal is to optimize the transition from the real input to the generated latent sequence.
  • D.2 Future Work: D.2 Future Work: Optimizing the transition from the real input to the generated latent sequence could preserve visual fidelity in subsequent frames.The proposed transition is intended to avoid the limitations of hard replication.
Loading 2607.09661v1…