Source-linked AI summary
CubeComposer: Spatio-Temporal Autoregressive 4K 360° Video Generation from Perspective Video
Lingen Li, Guangzhi Wang, Xiaoyu Li, Zhaoyang Zhang, Qi Dou, Jinwei Gu, Tianfan Xue, Ying Shan
TL;DR
Existing perspective-to-360° methods are constrained by full-attention diffusion, typically limiting native output to about 1K and requiring post-processing for higher resolution. CubeComposer decomposes panoramas into cubemap faces and autoregressively generates them across spatial and temporal dimensions with efficient context and continuity-aware designs. It natively generates 4K 360° videos with reported gains in native resolution, spatial seamlessness, and temporal consistency.
Problem
Existing perspective-to-360° methods generally support at most 1K native resolution and rely on post-processing super-resolution, despite VR applications demanding native 4K or higher.
Method
CubeComposer represents panoramas as cubemaps and autoregressively generates faces across planned temporal windows using sparse context attention and continuity-aware designs.
Results
CubeComposer natively generates 4K 360° videos and shows consistent gains over state-of-the-art methods in native resolution, spatial seamlessness, and temporal consistency.
Takeaways & Limitations
The method supports native high-resolution perspective-to-360° video generation without super-resolution while reducing peak memory through cubemap spatio-temporal autoregression.
Takeaways & Limitations
The formulation does not consider the temporal difference between pixel space and latent space in the VAE.
Abstract
from arXiv · showhide
Generating high-quality 360° panoramic videos from perspective input is one of the crucial applications for virtual reality (VR), whereby high-resolution videos are especially important for immersive experience. Existing methods are constrained by computational limitations of vanilla diffusion models, only supporting $\leq$ 1K resolution native generation and relying on suboptimal post super-resolution to increase resolution. We introduce CubeComposer, a novel spatio-temporal autoregressive diffusion model that natively generates 4K-resolution 360° videos. By decomposing videos into cubemap representations with six faces, CubeComposer autoregressively synthesizes content in a well-planned spatio-temporal order, reducing memory demands while enabling high-resolution output. Specifically, to address challenges in multi-dimensional autoregression, we propose: (1) a spatio-temporal autoregressive strategy that orchestrates 360° video generation across cube faces and time windows for coherent synthesis; (2) a cube face context management mechanism, equipped with a sparse context attention design to improve efficiency; and (3) continuity-aware techniques, including cube-aware positional encoding, padding, and blending to eliminate boundary seams. Extensive experiments on benchmark datasets demonstrate that CubeComposer outperforms state-of-the-art methods in native resolution and visual quality, supporting practical VR application scenarios. Project page: https://lg-li.github.io/project/cubecomposer
1. Introduction
Perspective-to-360° video generation aims to complete unseen regions from ordinary camera footage, but existing diffusion methods are limited to about 1K native output. CubeComposer addresses this with cubemap-based spatio-temporal autoregression, efficient context handling, and continuity-aware synthesis for native 4K generation.
- Motivation: Perspective-to-360° generation completes unobserved scene regions from perspective video while preserving the original context and dynamics.The approach could reduce dependence on specialized multicamera or 360° capture hardware.
- Motivation: Prior full-attention diffusion methods support at most 1K native resolution and often rely on post-processing super-resolution for higher-resolution output.External upscaling lacks intrinsic generative reasoning and can introduce error cascades.
- CubeComposer: CubeComposer generates one cubemap face within a fixed temporal window at each autoregressive step, reducing peak memory and enabling native 4K-scale generation.The 360° video is represented as six cubemap faces rather than synthesized in one diffusion pass.
- CubeComposer: A coverage-guided spatial order tied to camera motion prioritizes well-conditioned faces during causal temporal autoregression.The method plans face generation within each time window according to spatial coverage.
- CubeComposer: Sparse context attention uses generated history, current-window contents, and relevant future perspective fragments while reducing attention computation.The generation sequence receives full self-attention, while context attention is restricted locally.
- CubeComposer: Cube-aware positional encoding, padding, and blending use cubemap topology to produce seamless transitions between autoregressively generated faces.Adjacent-face overlaps are aligned during generation and blended in pixel space.
- Evaluation: 4K360Vid contains 11,832 high-resolution 360° video clips, and experiments report native 4K generation with superior visual quality over previous methods.The dataset includes clips at or above 4K resolution and supports global and face-wise captions.
2. Related Work
Related work spans 360° image and video generation, video foundation models, and autoregressive methods for temporal extension. CubeComposer builds on these directions while applying autoregression to spatial cubemap synthesis for perspective-to-360° video.
- 360° Generation: Early 360° generation research primarily addressed images before recent video foundation models enabled 360° video methods.The related work includes text- and image-conditioned 360° video diffusion and multi-view attention for cross-view consistency.
- Video Diffusion Models: Large-scale latent video diffusion models and DiT-based architectures provide video priors and generalization that CubeComposer uses for native 4K 360° generation.The cited foundation models learn in compressed VAE spaces and use high-capacity backbones and high-quality data.
- Autoregressive Video Generation: Recent autoregressive video methods mainly target temporal extension, streaming, and infinite generation rather than joint spatial-temporal 360° synthesis.These methods commonly convert bidirectional models into autoregressive ones through scheduling or post-training distillation.
3. Methodology
CubeComposer converts perspective video into masked cubemap inputs and autoregressively generates faces across planned temporal windows. Its context management and continuity-aware designs support efficient, coherent high-resolution 360° synthesis.
- Problem Formulation and Notation: Perspective frames are projected into masked cubemap faces, and the completed cubemap is assembled into an equirectangular video up to 4K resolution.The six faces are front, right, back, left, up, and down; cubemaps avoid the non-uniform distortion of equirectangular representations.
- Model Overview: CubeComposer divides the video into temporal windows and generates one cube face per step in a causal, coverage-prioritized spatio-temporal order.Faces with greater conditional spatial coverage are generated earlier within each window, prioritizing better-conditioned content.
- Context Mechanism: Each generation step combines history tokens, current-window tokens, and future-fragment tokens selected from spatially adjacent faces.Future fragments are chosen from temporally nearest content whose short-horizon spatial coverage exceeds threshold r.
- Context Mechanism: Sparse context attention keeps full attention for the generation sequence while restricting context self-attention to a diagonal-banded local mask.This changes context self-attention from quadratic to O(C · K) operations in context length C and bandwidth K.
- Continuity-aware Designs: Cube-aware positional encoding, topology-aligned padding, and overlap blending address discontinuities introduced at shared cube-face boundaries.Adjacent-face strips are rotated or flipped during padding, then overlapping decoded regions are weighted-averaged when updating context.
4. Experiments
CubeComposer is evaluated against prior perspective-to-360° methods using high-resolution datasets, quantitative and qualitative comparisons, and ablations of context and continuity designs. Results show native 4K generation with stronger visual quality, temporal coherence, and boundary continuity.
- Experimental Settings: The experiments use 4K360Vid, containing 11,832 high-resolution 360° video clips, together with the high-resolution ODV360 subset.The data include global and frame-wise captions and are filtered for low-quality or anomalous content.
- Experimental Settings: CubeComposer is compared with Argus, Imagine360, and ViewPoint using LPIPS, CLIP, FID, FVD, and VBench quality and consistency metrics.Evaluations use corresponding target resolutions, with ground truth resized to match each model's supported resolution.
- Comparison: CubeComposer natively generates 4K 360° videos with higher overall visual quality and detail than prior methods limited to 1K native resolution or 2K with VEnhancer.The comparison reports that VEnhancer post-processing can make previous outputs appear more unnatural, while CubeComposer operates natively at 4K.
- Ablation and Analysis: Excluding future tokens substantially degrades context-design performance, whereas the proposed mechanism matches full-context metrics while using fewer TFLOPs and achieving slightly better FVD.The analysis evaluates 27-frame videos on the ODV360 test set.
- Ablation and Analysis: Removing either cube-aware positional encoding or cube-aware padding and blending produces boundary artifacts and inconsistent degradations across LPIPS, CLIP, FID, and FVD.Enabling both continuity-aware components yields the best overall performance and reduces cross-face seams and temporal discontinuities.
5. Conclusion
CubeComposer natively generates 4K 360° videos from perspective inputs without super-resolution. Its cubemap-based spatio-temporal autoregression reduces peak memory while supporting global coherence, seamlessness, and temporal consistency.
- Conclusion: CubeComposer represents panoramas as cubemaps and plans generation across faces and temporal windows to reduce peak memory while preserving global coherence.Its design combines spatio-temporal autoregression, sparse context attention, and continuity-aware techniques to suppress seams.
- Conclusion: Experiments show consistent gains over state-of-the-art methods in native resolution, spatial seamlessness, and temporal consistency.Ablations confirm contributions from the model's components.