PaperlayerView exact revision on arXiv

Source-linked AI summary

360Anything: Geometry-Free Lifting of Images and Videos to 360°

Ziyi Wu, Daniel Watson, Andrea Tagliasacchi, David J. Fleet, Marcus A. Brubaker, Saurabh Saxena

arXiv:2601.16192v2cs.CV

TL;DR

Perspective-to-panorama methods often require camera metadata for geometric alignment, limiting their robustness on in-the-wild inputs. 360Anything instead learns the mapping with concatenated latent token sequences in a diffusion transformer and uses Circular Latent Encoding to address VAE-induced seams. It achieves state-of-the-art image and video panorama generation without camera metadata, while remaining constrained by base-model capacity, data bias, and current video resolution and length limits.

  • Problem

    Existing perspective-to-panorama approaches depend on precise camera metadata for ERP alignment, which is often unavailable or noisy in in-the-wild data.

  • Method

    360Anything concatenates perspective and noisy panorama latent tokens in a diffusion transformer and uses circular padding during VAE encoding to remove latent seam discontinuities.

  • Results

    360Anything achieves state-of-the-art panoramic image and video generation without camera metadata, outperforming methods with extra camera information and showing competitive zero-shot camera estimation.

  • Takeaways & Limitations

    The framework supports geometry-free perspective-to-360° generation and consistent panoramic videos for 3D scene reconstruction.

  • Takeaways & Limitations

    The video model is limited to 81 frames at 512×1024 resolution, while higher-resolution upsampling can reintroduce seams and distort ERP structure.

Abstract

from arXiv · show

Lifting perspective images and videos to 360° panoramas enables immersive 3D world generation. Existing approaches often rely on explicit geometric alignment between the perspective and the equirectangular projection (ERP) space. Yet, this requires known camera metadata, obscuring the application to in-the-wild data where such calibration is typically absent or noisy. We propose 360Anything, a geometry-free framework built upon pre-trained diffusion transformers. By treating the perspective input and the panorama target simply as token sequences, 360Anything learns the perspective-to-equirectangular mapping in a purely data-driven way, eliminating the need for camera information. Our approach achieves state-of-the-art performance on both image and video perspective-to-360° generation, outperforming prior works that use ground-truth camera information. We also trace the root cause of the seam artifacts at ERP boundaries to zero-padding in the VAE encoder, and introduce Circular Latent Encoding to facilitate seamless generation. Finally, we show competitive results in zero-shot camera FoV and orientation estimation benchmarks, demonstrating 360Anything's deep geometric understanding and broader utility in computer vision tasks. Additional results are available at https://360anything.github.io/.

1 Introduction

360Anything targets robust perspective-to-360° generation without camera metadata. It learns geometric correspondence from token sequences and addresses ERP seam artifacts at their VAE-encoding source.

  • Standard perspective generators capture narrow views, limiting their utility for fully immersive 3D worlds.
  • Prior perspective-to-panorama methods often require camera metadata to geometrically align perspective inputs with ERP targets.Required metadata includes Field-of-View and camera pose, which may be unavailable or noisy for in-the-wild inputs.
  • 360Anything concatenates perspective and panorama tokens in a diffusion transformer to learn their geometric relationship from data.This removes the camera estimation step and supports varying FoVs and camera poses.
  • Circular padding during VAE encoding produces circularly continuous latents, removing the training-stage source of ERP seam artifacts.
  • 360Anything achieves state-of-the-art panoramic image and video generation without camera metadata and shows competitive FoV and pose estimation.The generated panoramic videos also support consistent 3D scene reconstruction.

2 Related Work

Panorama generation has progressed from GAN and autoregressive models to diffusion methods, including text-to-panorama and perspective outpainting. Recent perspective-to-panorama systems still commonly depend on camera estimation and specialized geometric processing.

  • Early panorama generators used GANs or autoregressive models, while recent methods have shifted toward diffusion models.
  • Text-to-panorama methods improve panorama representations or panorama-aware operations to reuse knowledge from perspective generators.
  • Perspective-to-panorama outpainting methods include systems for narrow field-of-view images and videos.
  • Recent video methods use specialized architectures or representations, but rely on camera-metadata estimation and inference-time seam mitigation.

3 Method

360Anything uses latent diffusion transformers and sequence concatenation to learn perspective-to-ERP mapping without pixel-aligned geometric conditioning. Circular Latent Encoding addresses seam discontinuities introduced by VAE boundary padding.

  • Task formulation: The task maps a perspective video, with images treated as T=1, to a panoramic video represented in ERP space.
  • Overview: The method uses pre-trained latent diffusion transformers and generates panoramas in a gravity-aligned canonical space.
  • 3.1 Background: A flow-matching denoiser reverses a noise process from standard normal noise to the panorama-data distribution, conditioned on captions and perspective inputs.
  • 3.1 Background: Panorama data are encoded by a VAE, decoded back to pixels, then patchified and flattened into one-dimensional DiT tokens.
  • 3.2 Geometry-Free Scalable Panorama Generation: Instead of projecting perspective inputs into ERP space, the model concatenates perspective and noisy panorama latent sequences for global self-attention.The DiT learns their relationship in a purely data-driven way, avoiding dependence on precise FoV and orientation estimates.
  • 3.2 Geometry-Free Scalable Panorama Generation: Training data are canonicalized by estimating per-frame camera poses, aligning frames to a reference, and estimating gravity for vertical alignment.
  • 3.3 Seam-free Generation via Circular Latent Encoding: Zero-padding in convolutional VAE encoding creates latent boundary discontinuities that can produce seams in generated ERP panoramas.
  • 3.3 Seam-free Generation via Circular Latent Encoding: Circular Latent Encoding pads opposite panorama boundaries before encoding, then drops padded latents to preserve sequence length and produce seamless targets.The method uses w′=W/8 columns in the reported experiments and adds no training or inference overhead.

4 Experiments

Experiments evaluate 360Anything for perspective-to-360° image and video generation, camera estimation, seam removal, and design choices. The method achieves strong quantitative and qualitative results across these settings.

  • Experiments cover image and video generation, camera estimation, and ablations of the framework's design choices.
  • Panorama Image Generation: 360Anything achieves a clear improvement across image-generation metrics on Laval Indoor and SUN360, with only a marginal CLIP-FID disadvantage on Laval Indoor.FAED error is reduced by nearly 50% relative to the state of the art.
  • Panorama Image Generation: 360Anything produces less distorted image panoramas than CubeDiff when input FoV is below 90°, avoiding cubemap seams and boundary object stretching.The method estimates the input camera FoV and orientation for ERP placement.
  • Panorama Video Generation: 360Anything outperforms all baselines across video-generation metrics on both camera-trajectory subsets.The results include imaging quality, aesthetic quality, motion smoothness, and other reported metrics.
  • Camera Calibration: The zero-shot FoV estimator reaches a 4.93-degree average error and ranks among the top three on most datasets.It outperforms several supervised baselines and trails DUSt3R and MoGe by roughly 1–2 degrees.
  • Ablation Study: Circular latent encoding dramatically reduces seam discontinuity without adding generation-time overhead.It is compared with blended decoding as a seam-elimination technique.
  • Ablation Study: Canonicalized panorama-video training improves visual quality and fidelity-related metrics, while non-canonicalized training can improve PSNR and LPIPS.Canonicalization simplifies generation by producing gravity-aligned upright panoramas.

5 Conclusion

360Anything performs perspective-to-panorama generation without explicit geometric unprojection or camera information by learning correspondence through token sequences in a diffusion transformer. It also addresses seam artifacts and generalizes to diverse real-world media.

  • 360Anything uses sequence concatenation within a DiT to learn perspective-to-panorama correspondence without camera information.The framework replaces explicit geometric unprojection with data-driven learning.
  • VAE encoder padding is identified as the root cause of panorama seam artifacts, motivating a principled fix.
  • The approach achieves state-of-the-art performance on panoramic image and video benchmarks and demonstrates zero-shot generalization to real-world media.

A Detailed Experimental Setup

This section documents the datasets, baselines, evaluation settings, and training and inference implementation used for the model.

  • The experimental setup covers the datasets used to train and evaluate the model.
  • The section specifies the baselines used for comparison.
  • The section reports evaluation settings and training and inference implementation details.

A.1 Training Data

Training combines multiple panorama image datasets with camera-parameter augmentation and panorama-video data processed through gravity alignment and varied camera trajectories. The video pipeline supplements simulated motion with trajectories extracted from real-world videos.

  • Image data: Image training uses Polyhaven, Humus, Structured3D, and Pano360, with Structured3D contributing mostly synthetic indoor-room renderings.All three Structured3D subsets are used, and around 90% of the data are synthetic Structured3D renderings.
  • Image data: Camera augmentation samples FoV from [30°, 120°], pitch from [−60°, 60°], and roll from [−15°, 15°].Horizontal roll augmentation is also applied to panorama images.
  • Video data: Panorama videos are canonicalized by estimating camera orientation, averaging after outlier removal, and aligning gravity with the vertical axis.
  • Video data: Training samples camera trajectories from simulated linear motion with noise and real-world videos, using an 80%/20% mixture.The real-world trajectories are included because simple linear motion does not generalize well to complex motion.

A.2 Implementation details

The image and video models fine-tune pretrained diffusion transformers using VAE-encoded perspective and panorama token sequences, with task-specific positional handling and sampling configurations.

  • Image model: The image model fine-tunes FLUX.1-dev by separately VAE-encoding, flattening, and concatenating perspective and panorama tokens.The model applies 3D RoPE using shared spatial token indices and uses 50 sampling steps.
  • Image model: For 1024×2048 images, FLUX uses a cutoff timestep-shifting value of exp(1.15) ≈ 3.16 because the resolution exceeds its 4096-token limit.Larger shifting values degraded results in the reported experiments.
  • Video model: The video model fine-tunes Wan2.1-14B with the same overall design, offsetting perspective-token time indices by 0.1 in 3D RoPE.The offset avoids confusion with tokens from subsequent frames.
  • Video model: Wan training uses a learning rate of 1 × 10^-5 and batch size 64, followed by 50-step sampling with timestep shifting 3.0.The video model uses CFG weights of 3.0 for text and 2.0 for perspective-video conditioning.

A.3 Evaluation Setup

The evaluation measures perspective-to-360° image and video generation with established visual, geometric, and video-quality metrics, comparing against prior methods and their representation choices.

  • Evaluation Metrics: Image quality is evaluated on Laval Indoor and SUN360 using FID, KID, CLIP-FID, and panorama-specific FAED metrics.FID, KID, and CLIP-FID are computed on ten perspective crops sampled from non-overlapping azimuth ranges.
  • Evaluation Metrics: Video evaluation uses masked PSNR and LPIPS, ERP-video FVD, and VBench measures for imaging, aesthetics, and motion smoothness.The masked metrics assess regions covered by the perspective video using ground-truth camera information.
  • Additional Evaluations: The appendix reports model parameter counts, extreme-pose image results, and qualitative out-of-distribution panorama image results.Table 10 fixes FoV at 90° while evaluating pitches of ±60° and rolls of ±15°; Figure 9 shows OOD conditioning images.
  • Image Baselines: CubeDiff assumes a centered 90° FoV input, limiting adaptation to in-the-wild images with arbitrary FoV and camera orientation.Its cubemap representation conditions on the front face while jointly denoising the other five faces.
  • Video Baselines: Video comparisons include Imagine360, Argus, and ViewPoint, with ground-truth camera metadata supplied whenever a baseline requires projection.This evaluation gives metadata-dependent baselines information unavailable to 360Anything.

B More Experimental Results

Additional experiments test robustness to challenging camera inputs, out-of-distribution images, seam formation, and downstream 3D scene reconstruction.

  • Challenging Inputs: 360Anything generates reasonable panorama videos from challenging trajectories where camera estimators produce drifting roll angles or overly small FoV estimates.The end-to-end sequence-concatenation design avoids the explicit projection stage used by Argus.
  • OOD Images and Seam Quality: 360Anything generalizes to text-to-image-generated OOD inputs with high visual quality and correct overall structure.The accompanying boundary-consistency visualization shows a smooth transition after a 180° panorama shift.
  • 3D Scene Reconstruction: Generated 360° room views can support 3D Gaussian Splatting for fly-through exploration of reconstructed scenes.The pipeline first synthesizes the full viewpoint and then optimizes a 3DGS representation.

B.3 Perspective-to-360◦Video Generation

Video experiments show that 360Anything maintains canonical geometry and temporal consistency across difficult motion, supports downstream reconstruction, and benefits from sequence concatenation and real-world trajectory training.

  • Panoramic Video Outpainting: 360Anything keeps panorama videos upright under large elevation changes, while competing methods produce low quality or significantly rotated and distorted outputs.Its canonicalization training objective maintains correct scene geometry across challenging conditioning videos.
  • 3D Reconstruction: Generated panorama videos achieve 0.74-pixel mean reprojection error and 100% image registration, compared with 0.61 pixels and 94.5% for input perspective videos.The authors attribute improved registration to expanded view coverage, broader geometric constraints, and increased visual parallax.
  • Large-Motion Videos: Large object or camera motion still yields temporally consistent videos in a stable canonical pose.The result is presented as evidence of correspondence across the 4D world.
  • OOD Videos: On AI-generated videos from Wan, Sora, Veo, and Runway Gen-4.5, 360Anything produces high-quality panorama videos and preserves out-of-view scene details.Examples include reflections, persistent dust, and objects inferred beyond the conditioning view.
  • Conditioning Ablation: Sequence concatenation outperforms a same-size channel-concatenation variant while adding less than 20% overhead.Because panorama data has 8× more tokens, the input sequence length increases by only 1.125× relative to channel concatenation.
  • Trajectory Ablation: Training with real-world camera trajectories produces stably canonicalized videos, unlike models trained only on simulated trajectories.The simulated-only setup can yield changing gravity directions under large camera motion.

C Limitations and Future Work

The paper identifies limitations from base-model capacity, compute and panorama upsampling, while also discussing object permanence and broader qualitative behavior under difficult inputs.

  • Model Capacity and Bias: Fine-tuning from a pre-trained video diffusion model limits 360Anything’s capacity for complex physics and transfers biases from its training data.Observed artifacts include black borders and undesired tripod or hand appearances at the bottom of videos.
  • Resolution and Temporal Scope: The current video model is limited to 81 frames at 512×1024 resolution, and perspective-video upsamplers can reintroduce ERP seams and distort structure.These constraints arise from panorama resolution and limited compute; longer videos and panorama-specific upsampling remain future directions.
  • Object Permanence: Generating full 360° panoramas is presented as working memory that helps preserve objects outside the current perspective view.Extending this idea to sparse panoramic keyframes for long-term episodic memory remains open because occlusion modeling is unresolved.
  • Qualitative Generalization: The method is qualitatively reported to generalize to challenging perspective-to-360° videos without an explicit projection stage.Figure 17 provides qualitative comparisons, while Figure 16 focuses on failures of camera-estimation-based conditioning.
Loading 2601.16192v2…