PaperlayerView exact revision on arXiv

Source-linked AI summary

Enhancing In-context Panoramic Generation via Geometric-aware Pretraining

Haoran Feng, Ruiyang Zhang, Longyi Zhang, Dizhe Zhang, Lu Qi

arXiv:2607.08765v2cs.CV

TL;DR

Panoramic generation still struggles with geometry-consistent editing because ERP representations introduce latitude-dependent distortions and spherical depth differs from planar depth. Canvas360 combines geometry-aware RGB–depth pretraining, a 1M-scale dataset, and unified in-context fine-tuning, achieving consistent improvements in geometric adherence, seam consistency, and visual fidelity over prior methods.

  • Problem

    ERP panoramas retain latitude-dependent distortions, while existing methods struggle with geometric consistency and depth priors must be formulated for spherical scenes.

  • Method

    Canvas360 combines geometry-aware RGB–depth pretraining, scalable 1M-sample data synthesis, and unified token-level in-context fine-tuning across four panoramic tasks.

  • Results

    Canvas360 achieves the best FAED, IS, QA aesthetic, and NIQE scores, ranks second on several remaining metrics, and improves geometric adherence, seam consistency, and visual fidelity.

  • Takeaways & Limitations

    Canvas360 provides a unified framework with broad task coverage and strong performance across diverse in-context panoramic generation scenarios.

  • Takeaways & Limitations

    Canvas360 can underperform on underrepresented content such as high-resolution faces and text-rich signage, especially in heavily distorted ERP regions.

Abstract

from arXiv · show

In this work, we present Canvas360, a two-stage framework for in-context panoramic generation that combines geometry-aware pretraining with downstream task-specific fine-tuning. To address the lack of large-scale, high-quality training data tailored to in-context panoramic tasks, we propose Canvas360Dataset, a collection of 1M high-quality paired panoramic samples for style transfer, inpainting, outpainting, and editing, enabling effective supervision across diverse in-context generation scenarios. On the modeling side, Canvas360 enhances text-to-panorama generation through parallel depth generation, velocity circular padding, and similarity loss regularization, enabling the model to learn geometry-aware representations, capture object distortion details, and improve geometric consistency and global coherence. Furthermore, empowered by strong panoramic priors, Canvas360 enables a unified in-context panoramic generation framework that supports diverse downstream tasks via token-level concatenation, surpassing prior methods in both task coverage and modeling flexibility. Extensive experiments show that Canvas360 improves panoramic image fidelity, achieving particularly strong performance on the panorama-specific FAED metric and competitive or leading results across the reported quantitative evaluations. More information can be found on our project page: https://zry000.github.io/Canvas360/

1 Introduction

Canvas360 addresses geometric distortion and data scarcity in in-context panoramic generation through geometry-aware pretraining, unified fine-tuning, and a 1M-scale paired dataset. Across five tasks, it improves panorama fidelity and boundary consistency, with leading FAED performance and competitive validation scores.

  • Motivation: ERP panoramas exhibit latitude-dependent distortions that make geometry-consistent editing difficult despite existing distortion-aware methods.Prior approaches use cube-map-based editing or 3D spherical positional embeddings to mitigate this issue.
  • Method: Canvas360 uses two-stage geometry-aware pretraining and unified in-context fine-tuning to improve spatial consistency and geometric fidelity.Its pretraining jointly processes RGB and depth latents with flow-matching objectives, positional offsets, and similarity-loss regularization.
  • Data: Canvas360Dataset contains 100K annotated RGB–depth panoramas and 900K synthesized in-context samples across four downstream tasks.The dataset supports style transfer, outpainting, inpainting, and panorama editing, with the downstream samples generated through a scalable synthesis pipeline.
  • Unified generation: A unified model jointly learns four in-context panorama tasks within one framework, providing broad task coverage and strong generalization.The framework uses token-level concatenation to support diverse downstream in-context generation scenarios.

2 Related Work

Prior in-context image generation uses contextual inputs beyond text across diverse perspective-image tasks, while panoramic methods have evolved from multi-view or cube-map representations toward direct equirectangular and spherical-aware modeling. However, these panoramic approaches remain challenged by view inconsistency, boundary artifacts, and limited data quality.

  • In-context Panoramic Generation: In-context image generation incorporates reference images, depth maps, masks, or edge cues beyond text for style transfer, inpainting, outpainting, editing, and object manipulation.
  • In-context Panoramic Generation: Early panoramic generation methods use multi-view stitching or cube-map representations, but suffer from view inconsistency and boundary artifacts.
  • In-context Panoramic Generation: More recent methods train directly on equirectangular panoramas or introduce spherical-aware convolutions, yet remain constrained by limited data quality.

3 Method

Canvas360 uses geometry-aware pretraining on depth-augmented panoramas, then fine-tunes a unified RGB-only in-context model for four panoramic generation tasks. Its method combines token-wise RGB–depth conditioning, boundary-aware velocity prediction, and similarity regularization, supported by a 1M-sample synthesized dataset.

  • Geometry-aware pretraining: Canvas360 pretrains on depth-augmented panoramic data to strengthen spatial understanding and geometric consistency for in-context generation.Depth provides explicit 3D scene-structure supervision for panoramic image generation.
  • Geometry-aware pretraining: Parallel depth generation appends post-VAE depth tokens to RGB tokens, separates modalities with positional offsets, and computes flow-matching loss independently.The depth coordinates receive a constant first-dimension offset, enabling integration with pretrained models such as FLUX.1-Kontext.
  • Geometry-aware pretraining: Velocity circular padding synchronizes boundary features and target velocities so the model learns spherical adjacency and equivalent 0°/360° panorama boundaries.Flow-matching loss is computed on the padded velocity field to supervise boundary-consistent prediction.
  • Geometry-aware pretraining: Similarity-loss regularization penalizes excessive correlation between predicted RGB and depth velocity fields to encourage modality-specific representations.The loss addresses a degenerate optimum in which the two predicted outputs become overly similar.
  • Unified in-context fine-tuning: Fine-tuning removes depth inputs and unifies style transfer, inpainting, outpainting, and editing through RGB context–target token concatenation with positional offsets.The context is a full panorama for style transfer and editing, and a masked panorama latent for inpainting and outpainting.
  • Dataset synthesis: Canvas360Dataset contains 1M synthesized samples, including 900K downstream in-context samples spanning style transfer, inpainting, outpainting, and editing.The dataset begins with a 100K pilot set sourced from Matterport3D, the web, and panoramic generation models.

4 Experiments

Experiments show that Canvas360 improves panorama-specific fidelity and preserves geometric consistency across text-to-panorama and in-context generation tasks. Ablations attribute these gains to depth-aware modeling, velocity circular padding, and the Canvas360 backbone.

  • Qualitative Comparisons: Canvas360 produces finer-detail qualitative outputs than SMGD and PAR, which often generate cluttered or imprecise results.The comparison covers structural-modification and autoregressive-generation paradigms.
  • Quantitative Comparisons: Canvas360 achieves the best FAED, IS, QA aesthetic, and NIQE scores while ranking second on FID, FIDpole, FIDequ, QA quality, and BRISQUE.These results indicate strong panorama-aware fidelity with competitive performance across complementary metrics.
  • In-context Panoramic Generation: Canvas360 reduces blur and artifacts in inpainting and outpainting compared with panorama-specific baselines and FLUX.1-Fill-dev.FLUX.1-Fill-dev is adequate for small missing regions but produces extensive blur in larger missing areas.
  • In-context Panoramic Generation: Canvas360 applies correct panoramic distortion during editing, unlike existing methods that produce geometry-inconsistent edits.This supports the presence of geometry-aware panoramic priors in the model and data pipeline.
  • Ablation Studies: Parallel depth generation, position offsets, and similarity regularization improve geometric consistency, while velocity circular padding improves boundary continuity and edge alignment.Without these components, the model suffers geometric distortions; spherical depth can improve modeling but joint RGB–depth generation may become unstable.
  • Ablation Studies: Compared with FLUX.1-dev, Canvas360 generates more coherent, panorama-consistent completions and avoids perspective-image biases in inpainting and outpainting.The backbone comparison attributes this improvement to stronger panoramic priors and geometry-aware generation.

5 Conclusion · Appendix

Canvas360 is a two-stage in-context framework that injects geometry-aware priors through RGB–depth pretraining and transfers them to panoramic downstream tasks through unified fine-tuning. Its 1M-scale Canvas360Dataset and geometry-focused modeling components improve geometric adherence, seam consistency, and visual fidelity over prior methods.

  • 5 Conclusion: Canvas360 combines parallel RGB–depth pretraining with unified in-context fine-tuning for panoramic image generation.The framework transfers geometry-aware priors from pretraining to downstream tasks.
  • 5 Conclusion: Large-scale panoramas are paired with predicted depth, whose latents are fused with RGB latents at the token level.
  • 5 Conclusion: A Flow Transformer uses positional offsets, similarity regularization, and velocity circular padding to enforce spherical continuity and improve seam alignment.
  • 5 Conclusion: The framework supports diverse panoramic in-context generation tasks through geometry-aware pretraining and downstream unified fine-tuning.
  • 5 Conclusion: Canvas360Dataset addresses the data bottleneck with a scalable pipeline and 1M-scale coverage of inpainting, outpainting, style transfer, and panorama editing.
  • 5 Conclusion: Experiments show consistent improvements over prior methods in geometric adherence, seam consistency, and visual fidelity.

A Analysis of the Geometry-aware Training Strategies

The analysis evaluates geometry-aware training through predicted depth quality and the geometric structure retained in generated RGB panoramas. Predicted depths align with scene layout and object geometry, while depth supervision improves panorama consistency around seams and boundaries.

  • Geometry Prior Retained After Depth-supervised Training: Depth-supervised training produces more geometrically consistent panoramas, especially near stitching seams and boundary regions.The comparison converts generated panoramas to cubemap faces, retains four side faces, estimates depths, and stitches them back into ERP format.
  • Analysis Overview: The analysis examines geometry-aware training from depth prediction and RGB-panorama geometry perspectives.It evaluates meaningful depth generation and whether depth-supervised training improves generated panorama structure.
  • Visualization of Predicted Depth: Predicted depth maps preserve major scene layouts and object-level geometric structures while aligning with generated panoramic images.The depth branch captures meaningful geometric information that guides panoramic generation rather than producing merely auxiliary outputs.

B More Details on Dataset Construction

Canvas360Dataset is constructed specifically for in-context panoramic generation, combining paired data across multiple generation tasks with geometry-aware supervision. Its construction uses task-specific quality control, manual validation, and truncated pseudo-depth processing to improve data reliability and training stability.

  • Dataset Construction: Canvas360Dataset targets in-context panoramic generation with paired training data, multiple generation tasks, and geometry-aware supervision.Existing panoramic datasets mainly support perception, embodied reasoning, multimodal understanding, or camera-controlled generation rather than in-context panoramic generation.
  • Quality Control: Inpainting and outpainting pairs come from real panoramas, while editing uses task-specific strategies to reduce geometry-inconsistent artifacts.The quality-control procedure addresses artifacts caused by applying non-panoramic image generation and editing models directly to ERP panoramas.
  • Quality Control: 48K of 50K manually inspected samples are clean and valid, indicating limited residual noise in the large-scale dataset.A comparison between models trained on 20K cleaned samples and 20K randomly selected image-editing samples shows similar performance.
  • Pseudo-depth Processing: Pseudo-depth maps from DAP are clipped at 100 for outdoor scenes and 10 for indoor scenes before normalization.Truncation prevents extremely large distant-region depth values from dominating the distribution and destabilizing training.

C Experiment Settings

Canvas360 is evaluated through a diverse metric suite and qualitative comparisons, while implementation uses LoRA-based parameter-efficient fine-tuning on FLUX.1-dev with mixed precision across eight NVIDIA H20 GPUs.

  • Qualitative Comparisons: Qualitative comparisons for panoramic image generation highlight representative artifacts in red boxes.Figure 8 provides full qualitative comparisons.
  • Implementation Details: Canvas360 is built on FLUX.1-dev and fine-tuned with LoRA injected into attention and embedding layers using rank r=64, scaling factor α=64, and dropout 0.10.Only LoRA-injected parameters are optimized with AdamW at a learning rate of 1 × 10−5.
  • Implementation Details: Training uses FP16 mixed precision on 8 NVIDIA H20 GPUs.The implementation optimizes only the LoRA-injected trainable parameters.
  • Evaluation Metrics: Evaluation measures realism with FID, FIDpole, and FIDequ, targeting polar distortion and equatorial perspective quality.FIDpole and FIDequ follow SMGD, while FID may under-reflect panoramic properties because Inception is trained on perspective images.

D Full Comparison

Canvas360 achieves the best left-to-right boundary consistency among compared methods while addressing sharp-detail, global-structure, fine-grained-content, and seam-discontinuity challenges highlighted in qualitative comparisons.

  • Qualitative comparison: Qualitative comparisons mark typical failure patterns, including noisy textures, local distortions, fragile fine-grained content, and seam discontinuities.Structural-heuristic and autoregressive generators often struggle with sharp details and clean global structure, while recent DiT-based methods still exhibit ERP-related fragility and seam discontinuities.
  • Boundary consistency: Canvas360 achieves the lowest LRCE-RGB among all compared methods, indicating the best left-to-right consistency.LRCE-RGB measures discrepancy between the left and right boundary regions of generated RGB panoramas; lower values indicate better boundary consistency.
  • Boundary consistency: The LRCE-RGB result quantitatively supports the qualitative observations in Fig. 8.The evaluation follows PanoFormer by extending LRCE to RGB panoramas through left-right boundary discrepancy measurement.

E Ablations

The ablations show that Canvas360’s panoramic priors improve completion quality, while velocity circular padding produces more seamless panorama boundaries. Parallel depth generation improves FAED, positional offsets improve QAae, and similarity regularization is introduced to address remaining optimization instability.

  • Velocity Circular Padding: Velocity circular padding synchronizes ghost-column features with circular counterparts and assigns continuous longitude indices, producing more seamless panorama boundaries.The resulting coordinate transitions expose horizontal wrap-around boundaries as local transitions rather than discontinuities.
  • Parallel RGB–Depth Generation: FAED decreases from 5.37 to 4.74 when depth supervision is introduced, demonstrating improved panorama-oriented quality.Depth provides geometric cues within the coupled RGB–depth training strategy, while positional offsets and similarity regularization stabilize cross-modal optimization.
  • Parallel RGB–Depth Generation: QAae increases from 3.41 to 4.13 with positional offsets, indicating that separating RGB and depth tokens improves visual and aesthetic properties.Positional offsets alone do not fully resolve optimization instability, as shown by degraded FID-family and no-reference quality metrics.
  • Backbone Design: Canvas360 produces cleaner, more panorama-consistent completion results than FLUX.1-dev, whose inpainting introduces blur and artifacts and whose outpainting is less conditioning-consistent.FLUX.1-dev also exhibits perspective-biased patterns in outpainting, whereas Canvas360 learns stronger panoramic priors that transfer to image completion.

F Full Comparisons on In-context Panoramic Generation · G More Results

Canvas360 is evaluated across style transfer, inpainting, outpainting, and editing, where it generally preserves panoramic geometry, content, and visual coherence. Additional results show high-fidelity panoramas with distortion-consistent details, seam continuity, and geometry-consistent priors across tasks.

  • F Full Comparisons on In-context Panoramic Generation: The evaluation generates 500 results per experiment under the corresponding task-specific settings.These experiments cover in-context panoramic generation tasks.
  • F Full Comparisons on In-context Panoramic Generation: For style transfer, Canvas360 is compared with FLUX.1-Kontext-dev, FLUX.2-dev, and Qwen-Image-Edit using CP, SR, and OV.CP measures content preservation, SR style resemblance, and OV overall vision.
  • F Full Comparisons on In-context Panoramic Generation: Canvas360 achieves the best CP and OV, while FLUX.2-dev attains slightly higher SR.The results indicate stronger content preservation and overall visual quality, with competitive style resemblance.
  • F Full Comparisons on In-context Panoramic Generation: For inpainting and outpainting, Canvas360 is evaluated against the main-paper baselines using LPIPS, FAED, and PSNR.Lower LPIPS and FAED are better, while higher PSNR is better.
  • F Full Comparisons on In-context Panoramic Generation: For editing, Canvas360 is compared with FLUX.1-Kontext-dev, FLUX.2-dev, NanoBanana, SE360, and Omni2 using LPIPS, FAED, and PSNR.SE360 and Omni2 are additional panoramic editing baselines.
  • F Full Comparisons on In-context Panoramic Generation: Canvas360 achieves the best editing performance across all three metrics, with the lowest LPIPS and FAED and the highest PSNR.Qualitatively, it better preserves panoramic geometry and surrounding-content consistency.
  • G More Results: Across additional tasks, Canvas360 produces high-fidelity, visually coherent panoramas with distortion-consistent details, strong seam continuity, and geometry-consistent priors.These results support robust panorama-aware generation that generalizes across diverse in-context scenarios.

H Limitations and Future Work … K Human Preference Study Details

Canvas360’s remaining limitations include imbalanced panoramic training data that can reduce performance on distorted, underrepresented content, while responsible deployment requires safeguards for misuse, privacy, licensing, and attribution. The human preference study used anonymized four-way comparisons across four criteria with aggregate analysis and organizational review.

  • H Limitations and Future Work: Canvas360’s training corpus is imbalanced across scene types, causing underperformance on scarce cases such as high-resolution faces and text-rich signage in distorted ERP regions.Future work will expand and rebalance the dataset to strengthen the panoramic prior in challenging cases.
  • I Broader impacts: Canvas360 supports benign uses including VR/AR authoring, simulation, digital-twin prototyping, immersive scene creation, and 360-degree content design.These applications benefit from improved geometric consistency and visual fidelity in panoramic generation.
  • I Broader impacts: The model may be misused for deceptive scene manipulation, synthetic visual misinformation, or unauthorized content creation, while large-scale curation raises licensing and attribution concerns.These risks are described as similar to those of other generative vision models.
  • J Safeguards for Responsible Release: Dataset construction applies filtering to remove unsafe, sensitive, or low-quality samples and focuses on scene-level rather than identity-centric or face-oriented imagery.The design aims to reduce risks involving privacy, impersonation, and personal-attribute generation.
  • J Safeguards for Responsible Release: Released resources will include usage guidelines discouraging deceptive manipulation, misinformation, and unauthorized generation, alongside recommended provenance tracking, watermarking, attribution, and human review.The safeguards are especially encouraged before deployment in sensitive applications.
  • J Safeguards for Responsible Release: The full-scale dataset will be accompanied by documentation covering data sources, preparation procedures, filtering steps, and intended-use restrictions.The documentation is planned as part of the resource release because the dataset is large.
  • K Human Preference Study Details: The human preference study showed participants four anonymized candidate panoramas per question and asked them to select the image best satisfying text alignment, boundary continuity, panorama awareness, or overall quality.Method names were hidden to reduce potential bias; panorama awareness included spherical distortion and panoramic geometry.
  • K Human Preference Study Details: Internal volunteer participants completed voluntary four-choice comparisons, with responses used only for aggregate statistical analysis and no personally identifiable or sensitive information collected.The study involved minimal risk and was reported and reviewed through the authors’ organizational review process.
Loading 2607.08765v2…