Source-linked AI summary
OmniRoam: World Wandering via Long-Horizon Panoramic Video Generation
Yuheng Liu, Xin Lin, Xinke Li, Baihan Yang, Chen Wang, Kalyan Sunkavalli, Yannick Hold-Geoffroy, Hao Tan, Kai Zhang, Xiaohui Xie, Zifan Shi, Yiwei Hu
TL;DR
Perspective video models provide limited scene observations, creating challenges for completeness and global consistency. OmniRoam uses panoramic representations in a global-to-local preview–refine framework, and experiments show stronger visual quality, controllability, and long-term coherence than perspective-based methods. The framework also supports real-time preview and downstream 3D scene reconstruction.
Problem
Perspective video models synthesize limited scene observations, creating issues of scene completeness and global consistency.
Method
OmniRoam generates a trajectory-controlled panoramic preview, refines it into a high-quality long-horizon video, and trains on hybrid panoramic data with precise trajectories.
Results
OmniRoam outperforms perspective-based methods in visual quality, controllability, and long-term scene coherence.
Takeaways & Limitations
Generated panoramic wandering videos support real-time preview and downstream 3D scene reconstruction for scene-level modeling.
Abstract
from arXiv · showhide
Modeling scenes using video generation models has garnered growing research interest in recent years. However, most existing approaches rely on perspective video models that synthesize only limited observations of a scene, leading to issues of completeness and global consistency. We propose OmniRoam, a controllable panoramic video generation framework that exploits the rich per-frame scene coverage and inherent long-term spatial and temporal consistency of panoramic representation, enabling long-horizon scene wandering. Our framework begins with a preview stage, where a trajectory-controlled video generation model creates a quick overview of the scene from a given input image or video. Then, in the refine stage, this video is temporally extended and spatially upsampled to produce long-range, high-resolution videos, thus enabling high-fidelity world wandering. To train our model, we introduce two panoramic video datasets that incorporate both synthetic and real-world captured videos. Experiments show that our framework consistently outperforms state-of-the-art methods in terms of visual quality, controllability, and long-term scene consistency, both qualitatively and quantitatively. We further showcase several extensions of this framework, including real-time video generation and 3D reconstruction. Code is available at https://github.com/yuhengliu02/OmniRoam.
1 Introduction
OmniRoam addresses limited scene coverage and long-term consistency in perspective-based scene generation through panoramic representations and a global-to-local preview–refine framework. It combines scalable panoramic data and loop consistency evaluation to support coherent, controllable scene wandering.
- Motivation: Perspective-based scene generation is limited by narrow field-of-view observations, while panoramic representations provide holistic coverage and global spatial memory.Panoramic frames retain global context that would otherwise need to accumulate over time.
- Method: OmniRoam first establishes global scene structure with a trajectory-controlled panoramic preview, then progressively refines local details through higher spatial and temporal resolution.The preview stage produces an efficient overview from an input image or video and specified camera trajectory.
- Data and Evaluation: The framework uses synthetic and real-world panoramic videos with accurate camera trajectories to support scalable training.Its data pipeline uses a canonical coordinate system for panoramic scene generation.
- Data and Evaluation: Loop consistency quantitatively evaluates long-term global consistency in generated scene videos.The metric is introduced alongside the panoramic data generation pipeline.
- Results: Experiments report improvements over state-of-the-art methods in visual quality, controllability, and long-term scene consistency.The comparison covers qualitative and quantitative evaluation.
2 Related Work
Perspective video generation remains constrained by narrow fields of view, limiting holistic scene context and global spatial consistency during large-scale exploration. Panoramic video generation addresses these spatial limitations through representations that provide broader geometric cues.
- Perspective Video Generation: Perspective-based methods are confined to narrow fields of view, limiting holistic scene context and global spatial consistency during large-scale exploration.This limitation is presented as a consequence of perspective video representations.
- Panoramic Video Generation: Panoramic vision provides holistic geometric cues that address the spatial limitations of perspective views.Existing panoramic approaches include distortion-aware and projection-driven methods.
3 OmniRoam: Long Panoramic Video Wander
OmniRoam uses a global-to-local pipeline that previews camera-controlled panoramic videos before refining them into temporally extended, high-resolution outputs. Its training data and loop-consistency metric target accurate trajectories and long-term scene coherence.
- Pipeline: OmniRoam generates a mid-resolution panoramic preview from an input image or video and camera trajectory, then refines it into a high-quality long-horizon video.The preview uses trajectory conditioning, while refinement performs scale alignment and segment-wise diffusion.
- Trajectory Control: The preview model controls camera motion by separating global scale from per-frame flow directions.Scale represents displacement magnitude and playback speed, whereas flow encodes normalized 3D displacement directions for frame-wise control.
- Trajectory Control: The scale embedding is projected and injected globally into transformer blocks, uniformly modulating temporal tokens.The logarithmic representation supports a wide range of playback speeds.
- Long-Horizon Refinement: Refinement expands temporal resolution by aligning preview and target scales, dividing the video into segments, and conditioning each segment with selected preview frames.Visibility masks identify sparse preview frames used during segment-wise generation.
- Data and Evaluation: The hybrid dataset combines stabilized real-world panoramic footage with synthetic 3DGS sequences using physically feasible trajectories in a canonical panoramic coordinate system.The coordinate system removes camera self-rotation and defines trajectories through translation relative to the ERP center.
- Data and Evaluation: Loop consistency rewards similarity between beginning and ending frame buffers while preserving variation relative to intermediate frames.The metric uses CLIP cosine similarity and a five-frame buffer to balance loop closure with exploration.
4 Experiments
OmniRoam is evaluated for visual quality, trajectory controllability, and loop consistency at 480p and 720p using multiple camera trajectories, including held-out trajectory types. Across qualitative, quantitative, and long-horizon analyses, it follows trajectories more faithfully, preserves clearer geometry and details, and maintains stronger consistency than the compared alternatives.
- Evaluation Protocols: The evaluation measures visual quality, trajectory controllability, and loop consistency at 480p preview and 720p refine resolutions across seven camera trajectories, none directly used in training.Each trajectory-method pair produces 24 test videos; qualitative inputs include Internet panoramic images to assess generalization beyond training data.
- Quantitative Results: Our method outperforms prior approaches across FAED, SSIM, LPIPS, trajectory-controllability PSNR, and loop-consistency metrics.FAED, SSIM, and LPIPS are evaluated on 81 frames, while loop consistency uses full sequences; trajectory controllability is compared with Matrix-3D across three temporal windows.
- Qualitative Results: Qualitative comparisons show that OmniRoam follows specified trajectories while producing semantically coherent content, clearer object boundaries, and more stable geometric structure.Compared methods produce blurry artifacts, semantic inconsistencies, or geometric distortions; Imagine360 often fails to generate meaningful panoramic content.
- Design Analysis: Replacing panoramic videos with perspective videos degrades visual quality, trajectory controllability, and loop consistency in both 81-frame preview and 641-frame refined sequences.The design analysis also compares the global-to-local strategy with direct autoregressive generation using long-video metrics including PSNR over late temporal windows.
- Long-Horizon Generation: Over 641-frame loop trajectories, OmniRoam returns near the initial visual state with high consistency, whereas autoregressive and perspective-video variants drift or degrade structurally.Its similarity to the first frame decreases as the camera moves away and recovers as the loop closes; the final frames closely resemble the initial frame.
- Extensions and Applications: A distilled real-time previewer generates an 81-frame panoramic video in 7 seconds, while generated long videos can also support coherent 3D Gaussian Splatting reconstruction.The original preview model takes approximately 5 minutes, and Matrix-3D approximately 11 minutes; reconstruction samples 100 frames and five perspective crops per frame.
5 Conclusion
OmniRoam enables coherent, controllable long-horizon scene wandering through panoramic video generation, with demonstrated support for real-time preview and downstream 3D reconstruction.
- OmniRoam uses panoramic representations and a global-to-local preview–refine design for controllable long-horizon scene wandering.The framework targets coherent exploration of large scenes with explicit camera control.
- Additional qualitative comparisons report better geometric stability and finer textures during scene exploration.
- Real-time previewer-based generation produces videos at 480p and subsequently refines them to 720p.
- 641-frame loop-trajectory videos preserve strong long-term consistency, with final frames closely resembling the initial frame.