PaperlayerView exact revision on arXiv

Source-linked AI summary

DiT360: High-Fidelity Panoramic Image Generation via Hybrid Training

Haoran Feng, Dizhe Zhang, Xiangtai Li, Bo Du, Lu Qi

arXiv:2510.11712v1cs.CV

TL;DR

Panoramic generation lacks sufficient high-quality real-world data, making photorealism and geometric fidelity difficult to maintain. DiT360 uses hybrid training across perspective and panoramic data with image- and token-level mechanisms, achieving better boundary consistency and image fidelity across text-to-panorama, inpainting, and outpainting tasks.

  • Problem

    High-quality real-world panoramic data is scarce, and reliance on simulated data limits photorealism and geometric fidelity.

  • Method

    DiT360 combines perspective and panoramic data through image-level regularization and token-level geometry-aware supervision.

  • Results

    DiT360 outperforms existing methods in boundary consistency and image fidelity across text-to-panorama, inpainting, and outpainting tasks.

  • Takeaways & Limitations

    The framework supports high-resolution, photorealistic panoramic generation and natively enables inpainting and outpainting without additional finetuning.

  • Takeaways & Limitations

    Performance remains constrained by dataset diversity and scale, with suboptimal results for high-resolution faces and intricate scene details.

Abstract

from arXiv · show

In this work, we propose DiT360, a DiT-based framework that performs hybrid training on perspective and panoramic data for panoramic image generation. For the issues of maintaining geometric fidelity and photorealism in generation quality, we attribute the main reason to the lack of large-scale, high-quality, real-world panoramic data, where such a data-centric view differs from prior methods that focus on model design. Basically, DiT360 has several key modules for inter-domain transformation and intra-domain augmentation, applied at both the pre-VAE image level and the post-VAE token level. At the image level, we incorporate cross-domain knowledge through perspective image guidance and panoramic refinement, which enhance perceptual quality while regularizing diversity and photorealism. At the token level, hybrid supervision is applied across multiple modules, which include circular padding for boundary continuity, yaw loss for rotational robustness, and cube loss for distortion awareness. Extensive experiments on text-to-panorama, inpainting, and outpainting tasks demonstrate that our method achieves better boundary consistency and image fidelity across eleven quantitative metrics. Our code is available at https://github.com/Insta360-Research-Team/DiT360.

1 INTRODUCTION

Panoramic generation is hindered by projection distortions and limited high-quality real-world data. DiT360 addresses this gap by combining perspective and panoramic data through multi-level hybrid training, improving realism, fidelity, and boundary consistency across tasks.

  • High-quality real-world panoramic data is scarce, while reliance on simulated data limits perceptual realism and geometric fidelity.
  • DiT360 combines limited synthetic panoramic data with curated perspective images to improve photorealism and geometric fidelity.
  • Its hybrid paradigm applies image-level regularization and token-level geometry-aware supervision across perspective and panoramic domains.
  • Image-level mechanisms refine panoramas and transfer perspective knowledge, while token-level constraints address boundary continuity, rotation, and distortion.
  • DiT360 outperforms existing methods across text-to-panorama, inpainting, and outpainting tasks in boundary consistency, image fidelity, and perceptual quality.

2 RELATED WORK

Prior panoramic generation research includes outpainting and text-to-panorama approaches, while DiT360 presents a hybrid pipeline combining perspective and panoramic training branches.

  • Diffusion models became the dominant paradigm in image generation by reversing a gradual noising process to synthesize high-quality, diverse images.
  • Earlier panoramic generation methods mainly used outpainting to reconstruct 360° views from partial observations but often had limited flexibility and content diversity.
  • DiT360’s pipeline combines perspective and panoramic branches through perspective re-projection, panoramic refinement, and token-level hybrid supervision.

3 METHOD

DiT360 uses hybrid training across perspective and panoramic data, combining image-level regularization with token-level supervision to improve panoramic photorealism and geometric fidelity.

  • 3 METHOD: DiT360 jointly exploits perspective and panoramic data through image-level regularization and token-level supervision.The framework is built on DiT and applies hybrid training across multiple representation levels.
  • 3.2 IMAGE-LEVEL REGULARIZATION: Panoramic refinement converts ERP images to cubemaps, inpaints masked polar regions, and reprojects them to ERP for clearer training panoramas.This regularization reduces blur while retaining the panoramas’ inherent distortion characteristics.
  • 3.2 IMAGE-LEVEL REGULARIZATION: Perspective image guidance reprojects perspective images onto lateral ERP regions and applies masked MSE supervision to transfer photorealistic knowledge.The guidance exposes the model to more diverse scenes and regularizes photorealistic fidelity without contaminating unrelated panoramic regions.
  • 3.3 TOKEN-LEVEL SUPERVISION: Token-level supervision combines circular padding, yaw loss, and cube loss to improve boundary continuity, rotational robustness, distortion awareness, and perceptual quality.The auxiliary losses are applied to noisy post-VAE tokens, while the principal panoramic objective retains Flux’s MSE loss.
  • 3.3 TOKEN-LEVEL SUPERVISION: Circular padding copies boundary columns and their positional encodings so adjacent panorama columns can learn continuity across the horizontal boundary.The first and last column features are concatenated after reshaping noisy latent tokens into a spatial tensor.
  • 3.3 TOKEN-LEVEL SUPERVISION: Cube loss maps sampled and predicted noise to six cube faces for face-wise supervision that transfers perspective priors to ERP distortion modeling.The method applies cube and yaw losses directly in latent noise space, aligning auxiliary supervision with the flow-based scheduler.

4 EXPERIMENTS

DiT360 is evaluated against diverse panoramic-generation baselines using qualitative comparisons and quantitative metrics, followed by component ablations. It delivers strong perceptual realism, geometric fidelity, boundary consistency, and robustness across the reported evaluations.

  • 4.2 MAIN RESULTS AND COMPARISONS: DiT360 produces sharp, detail-preserving panoramas with stronger perceptual realism and geometric fidelity than the compared methods.The qualitative comparison contrasts its results with cluttered details, polar distortions, limited fine-grained fidelity, render-like appearances, and denoising artifacts from baselines.
  • 4.2 MAIN RESULTS AND COMPARISONS: DiT360 ranks first on nearly all reported benchmarks, with consistently strong performance across most metrics.It slightly trails top methods on CLIP Score and Q-Align quality, whose perspective-image design may not fully reflect panorama quality.
  • 4.3 ABLATION STUDY: Circular padding improves boundary consistency and overall image quality by aligning positional encodings at the left and right image edges.The ablation attributes this improvement to enabling the model to learn correct boundary correspondences, with reductions in FID and BRISQUE.
  • 4.3 ABLATION STUDY: Cube loss reduces polar-region artifacts and refines fine-grained details by supervising the cubemap representation for accurate panoramic distortions.Its ablation is associated with improved IS and CS, which are more related to visual semantics.
  • 4.3 ABLATION STUDY: Yaw loss improves global rotation consistency and structural coherence through supervision on rotated tokens.The effect is reflected particularly in FAED, whose autoencoders are pretrained on panoramic images.
  • 4.3 ABLATION STUDY: Perspective image guidance enhances local details, visual diversity, and style-sensitive perceptual quality while mitigating detail-related artifacts.The reported evidence comes from QAquality and QAaesthetic metrics.

5 CONCLUSION

DiT360 combines limited high-quality panoramic data with large-scale perspective images and applies image-level and token-level mechanisms for photorealism and geometric consistency. Across text-to-panorama, inpainting, and outpainting, it demonstrates strong image fidelity, boundary consistency, and visual quality.

  • 5 CONCLUSION: DiT360 demonstrates superior image fidelity, boundary consistency, and visual quality across text-to-panorama, inpainting, and outpainting tasks.The conclusion presents the framework as a strong baseline for future research in 3D scene generation and large-scale open-world environments.
  • 5 CONCLUSION: Hybrid training combines limited high-quality panoramic data with large-scale perspective images to enhance realism and generalization.The framework bridges the two domains through mechanisms operating at multiple representation levels.

B INPAINTING AND OUTPAINTING

DiT360 supports panoramic inpainting and outpainting without additional training through inversion, reference-token extraction, masking, and time-step-adaptive token replacement. Cube loss further reduces polar artifacts and clarifies structures in generated panoramas.

  • B INPAINTING AND OUTPAINTING: DiT360 natively supports inpainting and outpainting without additional training by using inversion, reference tokens, subject masks, and token replacement.Masked or extended-region tokens are replaced with reference-image tokens while preserving the original positional encodings during early denoising.
  • B INPAINTING AND OUTPAINTING: Cube loss produces clearer structures and fewer polar-region artifacts when comparing panoramas and their top and bottom cube faces.Figure 6 compares the same prompt without and with cube-loss supervision.

C EXPERIMENT SETTINGS

The experiments fine-tune DiT360 on Flux with LoRA using hybrid perspective and panoramic datasets, and evaluate it with complementary realism, alignment, and perceptual-quality metrics. Additional qualitative results cover inpainting and outpainting.

  • C EXPERIMENT SETTINGS: Training uses Flux with LoRA, five H20 GPUs, AdamW, a 2 × 10^-5 learning rate, 20 epochs, and guidance scale 1.0 for stable training.The per-GPU batch size is 1 with gradient accumulation of 3.
  • C EXPERIMENT SETTINGS: The hybrid dataset combines 40k curated perspective landscape images with 10,800 Matterport3D panoramas from 90 building-scale scenes.Perspective images are center-cropped and projected onto random panoramic regions, while the panoramic branch follows PanFusion.
  • C EXPERIMENT SETTINGS: Evaluation covers realism, polar distortion, perspective-projection quality, text–image alignment, and perceptual quality using complementary metrics.Reported metrics include FID variants, CLIP Score, Q-Align, BRISQUE, and NIQE.
  • C EXPERIMENT SETTINGS: Additional qualitative results demonstrate the framework’s application to inpainting and outpainting.The results are presented as further examples beyond the primary panorama-generation evaluation.

D FULL COMPARISION

DiT360 produces sharper, more visually authentic panoramas while improving geometric fidelity, boundary continuity, and artifact reduction over baseline methods.

  • DiT360 produces sharper, more visually authentic panoramas than baseline methods while preserving geometric fidelity and boundary continuity.
  • Baseline methods often exhibit visible artifacts and structural inconsistencies that DiT360 effectively handles.
  • Figure 8 presents a qualitative comparison of panorama generation, highlighting representative artifacts with red boxes.

E USER STUDY

A 63-participant user study evaluated text alignment, boundary continuity, realism, and overall quality, with DiT360 receiving the highest preference across all metrics.

  • The study compared DiT360 with representative baselines using preferences from 63 participants.
  • Table 3 reports the user-study results for text-to-panorama generation.
  • DiT360 received the highest preference across text alignment, boundary continuity, realism, and overall quality.

F MORE RESULTS

Additional examples show DiT360 producing high-quality, semantically coherent, and visually detailed panoramic completions across varied scenes.

  • DiT360 consistently produces high-quality, semantically coherent, and visually detailed completions across varied scenes.
  • Figures 9 and 10 provide additional results for panoramic image generation.
  • The paper states that the additional examples further illustrate DiT360’s panoramic image-generation performance.

H LIMITATIONS AND FUTURE WORK

DiT360 remains constrained by the diversity and scale of available datasets, motivating broader data collection and future extensions toward three-dimensional scene generation and understanding.

  • Limited dataset diversity and scale lead to suboptimal results for high-resolution human faces and intricate scene details.
  • Future work targets larger, more diverse high-quality datasets and synthetic-data augmentation to improve generative capabilities and image resolution.
  • The authors identify three-dimensional scene generation and understanding as a long-term research direction.
Loading 2510.11712v1…