PaperlayerView exact revision on arXiv

Source-linked AI summary

SpheRoPE: Zero-Shot Optimization-Free 360 Panorama Generation with Spherical RoPE

Or Hirschorn, Aaron Olender, Eli Alshan, Ianir Ideses, Lior Fritz, Sagie Benaim

arXiv:2606.32033v1cs.CV

TL;DR

Existing diffusion models struggle to enforce ERP’s horizontal periodicity and polar convergence when generating 360° content. SpheRoPE injects spherical positional priors and semantic geometry guidance at inference time, achieving competitive performance against fine-tuned baselines while outperforming them in panoramic coherence and human preference.

  • Problem

    Standard diffusion models struggle to satisfy ERP’s horizontal periodicity and polar convergence constraints, limiting reliable generation of immersive 360° images and videos.

  • Method

    SpheRoPE combines spherical RoPE with Semantic Distortion CFG to impose spherical topology on pre-trained diffusion transformers without training or optimization.

  • Results

    The framework achieves competitive performance against fine-tuned baselines while consistently outperforming them in panoramic coherence and human preference across image and video benchmarks.

  • Takeaways & Limitations

    Inference-time geometric grounding preserves foundation-model versatility while supporting zero-shot 360° image and video generation across diverse backbones and modalities.

  • Takeaways & Limitations

    The framework assumes ERP’s horizontal periodicity, with the left and right edges representing the same meridian.

Abstract

from arXiv · show

We present a zero-shot, training-free and optimization-free framework for generating 360 panoramic images and videos by directly injecting spherical priors into pre-trained diffusion transformers. Existing methods either rely on costly fine-tuning on scarce panoramic data that limits generalization, or leverage multi-step optimization that incurs prohibitive inference latency. We observe that contemporary generative models natively exhibit some panoramic priors from large-scale training. However, these emergent capabilities are insufficient, as the models fundamentally fail to satisfy the rigorous topological constraints imposed by equirectangular projection (ERP). We introduce a zero-shot and optimization-free approach that resolves these constraints at inference time. Spherical RoPE replaces standard rotary position embeddings: low-frequency channels are re-parameterized as 3D Cartesian coordinates to natively encode the spherical manifold, while high-frequency channels are harmonically quantized to enforce exact periodicity. Coupled with complementary Semantic Distortion classifier-free guidance (CFG) that explicitly steers geometry, we avoid retraining and inherit the full creative breadth of state-of-the-art models. Our approach generalizes across diverse backbones and 360 generation modalities. We demonstrate this across text-to-panorama using Flux.1, Flux.2, and LTX-Video backbones, achieving competitive performance against baselines, all while remaining training-free. Project page: https://orhir.github.io/SpheRoPE

1 Introduction

SpheRoPE addresses the geometric failures of planar diffusion transformers in 360° equirectangular generation with a zero-shot, training-free, optimization-free inference framework. It combines spherical spatial priors and semantic distortion guidance, generalizing across image and video backbones while achieving competitive benchmark performance and stronger panoramic coherence and human preference.

  • Motivation and problem: 360° panoramas use equirectangular projection, but planar RoPE and attention assume a Euclidean grid, causing boundary seams and polar geometric incoherence.These limitations arise because ERP maps spherical imagery onto a 2D rectangular plane.
  • Related work: Existing solutions rely on specialized panoramic training, including full fine-tuning, LoRA adaptation, or spherical-manifold architectures, and are constrained by scarce panoramic data.The introduction identifies training-based adaptation as one of two prevailing paradigms for addressing the ERP-generation gap.
  • Method overview: SpheRoPE realigns pre-trained diffusion-transformer inductive biases with spherical geometry through zero-shot, training-free, optimization-free inference-time modifications.The framework leverages panoramic priors already present in foundational models while correcting their inability to enforce rigorous ERP constraints.
  • Method overview: Spherical RoPE induces spherical topological invariants through spectral decomposition, while Semantic Distortion CFG uses an anchored geometric prompt to steer denoising toward valid ERP projections.The three-way guidance scheme extends standard CFG and is designed to preserve semantic detail while improving geometric validity.
  • Experiments and scope: The framework generalizes across diffusion transformers, demonstrated with Flux for static environments and LTX 2.3 for 360° video, without altering model weights.Evaluation covers ODI-SR, VBench, LLM-based perceptual assessment, and user studies, with competitive benchmark performance and consistent gains in panoramic coherence and human preference.

2 Related Work

Related work spans training-based and optimization paradigms for text-driven 360 panorama generation, alongside transformer positional-encoding adaptations for spatial, temporal, and spherical geometry. Prior spherical RoPE work maps spatial domains to a 3D sphere but requires fine-tuning to accommodate the changed positional structure.

  • Text-driven 360 panorama generation: Text-driven 360 panorama generation has developed along training-based and optimization paradigms following early VQGAN-CLIP and latent-diffusion models.
  • Text-driven 360 panorama generation: Training-based panorama methods fine-tune on panoramic datasets using DreamBooth or LoRA, with architectural extensions targeting spatial consistency.These extensions include MultiDiffusion stitching, dual-branch networks, specialized attention, and spherical convolutions.
  • Positional encoding adaptation in transformers: Prior RoPE adaptations rescale frequencies, modify DiT spatial or temporal axes, adjust embeddings across denoising timesteps, and address non-Euclidean spaces.
  • Positional encoding adaptation in transformers: IaaW uses spherical RoPE by mapping the entire spatial domain to a 3D sphere, requiring fine-tuning to accommodate the changed positional structure.

3 Method

SpheRoPE generates ERP panoramas from pretrained diffusion transformers without fine-tuning, architectural changes, or parameter modification. It combines spherical position encoding that enforces ERP topology with geometric classifier-free guidance for prompt-faithful generation.

  • 3.1 Problem Formulation: The framework targets semantically faithful ERP images or video frames that satisfy horizontal periodicity and polar convergence without modifying model parameters.ERP uses width W = 2H and imposes continuity across the ±π meridian plus column-independent values at both poles.
  • 3.2 Spherical RoPE: Standard linear RoPE violates both ERP constraints by assigning different embeddings to equivalent boundary columns and distinct embeddings to columns at the poles.The method replaces only width-axis RoPE channels while preserving the height and temporal axes identically to the original model.
  • 3.2 Spherical RoPE: Spherical RoPE partitions the spectrum at the first non-quantizable frequency, assigning spherical Cartesian encoding to low-frequency channels and cyclic harmonic encoding to high-frequency channels.Quantization uses the fundamental frequency ωfund = 2π/Wtokens, requiring at least one cycle and near-integer frequency ratio; low-frequency channels instead encode Cartesian X and Y coordinates.
  • 3.2 Spherical RoPE: The low-frequency Cartesian encoding guarantees horizontal wrap-around and polar convergence by tracing a closed longitude circle and making polar coordinates independent of column index.As θ → ±π/2, X and Y converge to R; as longitude wraps from −π to π, X(r, 0) = X(r, W) and Y(r, 0) = Y(r, W).
  • 3.3 Semantic Distortion CFG: Semantic Distortion CFG adds an anchored geometric prompt and combines user-conditioned, unconditional, and geometric noise predictions with independently controlled semantic and geometric scales.The formulation recovers standard CFG when γ = 0 while providing control over prompt fidelity and geometric validity.

4 Experiments

Experiments evaluate SpheRoPE on text-to-360 image and video generation, combining qualitative comparisons, benchmark metrics, user preferences, and component ablations. Across these evaluations, the zero-shot method produces coherent panoramas, supports image conditioning, and performs competitively or favorably against trained and optimization-based baselines.

  • Text-to-360 Image: SpheRoPE generates seamless, globally coherent panoramas and handles out-of-distribution stylized prompts better than training-based and patch-based optimization baselines.Training-based methods show architectural bias, while SphereDiff’s patch-based synthesis lacks global coherence.
  • Plug-and-Play Conditioning: The inference-only framework preserves foundation-model conditioning, enabling image-conditioned 360 generation that maintains reference identity and style across the sphere.This avoids retraining adapters when adding supported conditioning modalities.
  • Text-to-360 Image: SpheRoPE achieves the best panorama-level FAED score on ODI-SR without task-specific training and remains competitive on DS, KID, IS, and local perspective-view realism.It matches PAR on KID and IS while competing with DiT360 on crop-level metrics.
  • Text-to-360 Video: On SphereDiff-20, SpheRoPE leads all reported metrics, and on Stress-20 it dominates temporal metrics and imaging quality despite DynamicScaler’s higher CLIP Mean.DynamicScaler uses multi-pass stitching, has the worst temporal coherence, and is an order of magnitude slower.
  • User Preference Study: In 320 blind pairwise judgments from 18 annotators, users consistently prefer SpheRoPE over all baselines for overall quality and text alignment.The method also holds a clear overall-quality advantage over DiT360.
  • Component Contributions: Ablations show that Spherical RoPE resolves hard boundary discontinuities and improves DS, while the vanilla baseline lacks global structure despite panoramic prompt guidance.Table 4 evaluates Spherical RoPE and Semantic Distortion CFG on FLUX.2.

5 Conclusion · A Preliminaries

The paper presents a zero-shot, optimization-free framework for seamless 360° image and video synthesis that enforces spherical topology with Spherical RoPE and Semantic Distortion CFG. Its preliminaries define diffusion-model denoising, classifier-free guidance, and rotary position embeddings as foundations for the method.

  • 5 Conclusion: The framework synthesizes seamless 360° images and videos with pre-trained diffusion models without fine-tuning.It is described as zero-shot, optimization-free, and training-free, while preserving foundation-model versatility.
  • 5 Conclusion: Spherical RoPE and Semantic Distortion CFG enforce spherical topological invariants within the latent space.The conclusion identifies these two components as the mechanism for natively enforcing the required spherical structure.
  • A Preliminaries: Diffusion models generate samples by iteratively reversing Gaussian noising from xT ∼ N(0, I) toward a clean sample.A neural network ϵθ predicts noise at each timestep of the gradual noising process.
  • A Preliminaries: Standard classifier-free guidance interpolates between unconditional and text-conditioned predictions using ˆϵ = ϵuncond + wsem · (ϵcond −ϵuncond).Here, wsem > 1 amplifies the influence of the text condition.
  • A Preliminaries: Standard CFG follows a single semantic guidance direction and encodes no geometric priors about the output domain.The same guidance principle applies to flow-matching models in velocity or x0-prediction space.
  • A Preliminaries: RoPE provides relative spatial-position encoding for diffusion transformers, which otherwise require positional mechanisms because transformer blocks are permutation-equivariant.The preliminaries contrast RoPE with earlier fixed sinusoidal and learned absolute embeddings.
  • A Preliminaries: Each RoPE frequency channel applies a 2D rotation matrix R(αi) to corresponding query and key dimensions, with θbase determining the frequency series.Dw denotes the number of width frequency channels.
  • A Preliminaries: The complete RoPE transformation applies block-diagonal rotations to query q and key k before their attention inner product is computed.This describes how the positional rotations enter the attention operation.

B Additional Experiments · B.1 Implementation Details

The appendix details the implementation settings, evaluation protocol, baseline handling, and circular latent encoding used for the experiments. Image generation uses fixed ERP and spherical-encoding configurations, while comparisons rely primarily on official implementations and hardware-based timings.

  • B.1 Implementation Details: ERP panoramas are generated at 1024 × 2048 resolution with 50 denoising steps on NVIDIA H100 GPUs.The harmonic quantization tolerance is ε = 0.06.
  • B.1 Implementation Details: The spherical Cartesian encoding sets R = Wtokens/2, or R = Wspan/2 for LTX 2.3, with only the width axis modified.RoPE dimensionality follows each backbone’s native structure; Wspan denotes the coordinate-normalized width extent.
  • B.1 Implementation Details: Assets are used under their original licenses, and the experiments are restricted to non-commercial academic research.Listed licenses include Black Forest Labs’ Non-Commercial License, Lightricks’ Community License, and Apache 2.0.
  • B.1 Implementation Details: Baselines use official codebases and released pretrained weights, except StitchDiffusion uses a diffusers reimplementation with official LoRA weights.Methods depending on Stable Diffusion 2.1-base use the community mirror Manojb/stable-diffusion-2-1-base because the original weights were removed from HuggingFace.
  • B.1 Implementation Details: GPT-4o scores 14 perspective views per scene on a 1–5 scale across panoramic and image criteria using a 20-prompt benchmark.Inference time per scene is additionally reported in seconds on an NVIDIA H100.
  • B.1 Implementation Details: The reported evaluation criteria include distortion, end continuity, image quality, aesthetic appearance, and time per scene.These criteria organize the panoramic and image assessments alongside runtime measurement.
  • B.1 Implementation Details: Scores for 360 LoRA, Text2Light, PanFusion, and DynamicScaler come from, while all other scores and timings are measured on NVIDIA H100 hardware.All methods are timed for fair comparison except 360 LoRA and DynamicScaler, whose reimplementations lack publicly available inference code.
  • B.1 Implementation Details: Circular padding symmetrically extends VAE-decoder tensors before decoding, then cropping preserves periodic boundaries and eliminates latent-space seams.This procedure makes convolutional receptive fields perceive a continuous spherical manifold.

B.2 Evaluation Metrics

The evaluation combines universal image-quality metrics with panorama-specific measures to assess both distributional quality and ERP geometry. Universal metrics are computed on undistorted perspective crops, while the paper notes that distributional metrics structurally favor training-based methods.

  • Universal metrics: Universal metrics include FID, KID, IS, and CS, measuring distributional distance, quality, diversity, and text-image alignment.FID and KID use Inception-v3 features; IS uses KL divergence between conditional and marginal class distributions; CS uses cosine similarity between CLIP embeddings.
  • Universal metrics: Universal metrics are computed on gnomonic perspective crops from ERP panoramas to reduce penalties for geometrically correct panoramic features.The crops provide undistorted patches consistent with the perspective-image training distribution of the feature extractors.
  • Metric limitations: FID, KID, and FAED inherently favor training-based methods because their fine-tuning data overlaps with the evaluation distribution.The zero-shot approach instead samples from the foundation model’s broader unconstrained distribution, creating a structural disadvantage on these metrics despite visually competitive results.
  • Panorama-specific metrics: Panorama-specific metrics include FAED for distortion-aware perceptual and geometric quality and DS for seam artifacts at the horizontal wrap boundary.FAED uses an autoencoder trained on 360° panoramas, while DS applies kernel-based edge detection at the ±π meridian.
  • Panorama-specific metrics: Lower DS indicates better perceived continuity across the panorama’s horizontal seam.DS directly measures continuity across the horizontal wrap boundary at the ±π meridian.

B.3 Additional Quantitative Results · B.4 Additional Qualitative Results

The additional results evaluate panoramas with GPT-4o across distortion, end continuity, image quality, and aesthetics, while qualitative analysis shows that pretrained models capture ERP characteristics but fail to close the sphere topologically. SpheRoPE preserves seamless periodicity and improves key evaluation outcomes while running over 20× faster than SphereDiff.

  • B.3 Additional Quantitative Results: GPT-4o evaluates 14 perspective views on a 1-5 scale across distortion, end continuity, image quality, and aesthetic appearance.The protocol follows SphereDiff and uses its exact prompt.
  • B.4 Additional Qualitative Results: Horizontal panorama shifts reveal a noticeable vertical seam in the base model, whereas SpheRoPE maintains seamless periodicity.The comparison is shown in Figure 5(a).
  • B.4 Additional Qualitative Results: Perspective crops show that the base model sometimes fails to model ERP distortions properly, motivating SpheRoPE’s CFG enhancement.This comparison is shown in Figure 5(b).
  • B.3 Additional Quantitative Results: SpheRoPE achieves the best end continuity and competitive distortion performance.These results are reported in the additional quantitative evaluation.
  • B.3 Additional Quantitative Results: SpheRoPE substantially outperforms SphereDiff on image quality and aesthetics while running over 20× faster.The speed comparison is stated alongside the evaluation results.
  • B.4 Additional Qualitative Results: Flux.2 inherently captures ERP characteristics including stretched poles and the wide aspect ratios of 360◦ scenes.The qualitative analysis attributes these capabilities to pretrained diffusion transformers.
  • B.4 Additional Qualitative Results: Base-model panoramas exhibit disjointed boundaries and polar discontinuities because they lack topological awareness to mathematically close the sphere.Horizontal shifts reveal vertical seams, while perspective crops expose failures to model ERP distortions.

B.5 More Ablation Studies · C Geometric Prompts

Ablations show that spherical RoPE, spectral partitioning, Semantic Distortion CFG, and circular latent encoding are jointly important for seamless, geometrically valid panoramas. Geometric prompts balance ERP constraints without over-constraining image generation, while video prompts additionally enforce temporal stability.

  • B.5 More Ablation Studies: Without spherical RoPE, panoramas become perspective-like, with hard seam discontinuities, pole artifacts, and no polar convergence.Without Semantic Distortion CFG, boundaries remain seamless but the content misses ERP-specific pole stretching and horizon curvature.
  • B.5 More Ablation Studies: The dual-path RoPE assigns cyclic linear encoding to preserve local textures and spherical encoding to anchor the panorama’s coherent global layout.Cyclic linear encoding alone causes artifacts, while spherical encoding alone disrupts local distance metrics and produces blurry or aliased details.
  • B.5 More Ablation Studies: ε ≥0.10 produces an identical channel split, whereas ε=0.01 splits too early and routes too many channels into the spherical path.The tolerance determines whether frequencies are routed through cyclic linear or spherical Cartesian encoding based on harmonic quantization and minimum-cycle requirements.
  • B.5 More Ablation Studies: Semantic Distortion CFG exhibits a trade-off: γ = 3, 4 favor multi-view IS and CS, while γ = 8, 10 favor FAED, DS, perspective FID, and KID.Across the tested range, visual quality and layout remain stable, with lower γ producing slightly sharper, more varied texture and higher γ smoother, more regularized content.
  • B.5 More Ablation Studies: Early-weighted Semantic Distortion CFG schedules enforce valid ERP geometry, whereas late-weighted schedules yield flat perspective-like panoramas with pole artifacts.This indicates that panoramic geometry is established during the initial denoising steps.
  • B.5 More Ablation Studies: The default radius scale s=2.0 preserves global geometry and local detail by matching the positional-value range learned by the pretrained RoPE.Extreme scale choices move RoPE frequencies outside their learned regime and produce failure modes.
  • C Geometric Prompts: Geometric prompts require balance: minimal wording produces flat panoramas with pole artifacts, verbose wording causes extreme top-down curvature, and the default yields natural ERP geometry.The default geometric prompt describes a unified scene, true equirectangular projection, spherical geometry, continuous horizontal wrapping, and zero parallax error.
  • C Geometric Prompts: The video geometric prompt is longer because it enforces a static camera, pixel-locked scene geometry across frames, and only minimal atmospheric motion.These constraints prevent camera movement and structural warping that would violate ERP requirements.

C.2 Stress-20 Benchmark Construction

Stress-20 extends SphereDiff-20 with 20 diverse prompts designed to stress-test temporal coherence in challenging 360° video dynamics. The prompts were finalized before evaluation, and cross-set rankings show the method remains competitive without uniformly favoring it.

  • Prompt construction: The generation instructions emphasized full-sphere motion, independent subject trajectories, dynamic lighting, particle effects, and difficult temporal-coherence scenarios across distinct environments.Each prompt was specified to contain 30–60 words and avoid repeating or paraphrasing reference prompts.
  • Evaluation protocol: The 20 prompts were used verbatim, with no filtering, reordering, or modification, and were finalized before evaluating any method.No prompts were added, removed, or edited after observing method outputs.
  • Cross-set ranking agreement: The method ranks first on all 6 VBench metrics for SphereDiff-20 and on 5/6 metrics for Stress-20, ranking second only on Stress-20 CLIP Mean behind DynamicScaler.This ranking pattern indicates the self-designed benchmark does not uniformly inflate the method’s scores.

D Limitations and Future Work

The method can fail on strong perspective prompts, repeat structures, and depends on RoPE-based backbones exposed to panoramic content. Future work includes spatially aware audio-video generation and extending RoPE reshaping beyond spherical geometry.

  • Failure Cases: Strong perspective prompts may produce conventional perspective images instead of true equirectangular panoramas, while structural elements may occasionally repeat across the 360° field of view.These representative failure modes are illustrated in Figure 14.
  • Limitations: The method requires backbones using Rotary Position Embeddings because Spherical RoPE directly modifies width-axis rotation angles.Architectures with learned absolute embeddings or additive sincos embeddings would require a different adaptation strategy.
  • Limitations: The pre-trained model must have been exposed to panoramic or ERP-like content during training for the approach to apply.The supplied passage states this as a second architectural assumption.
  • Future work: Future work will spatialize audio to match 360° visual content, replacing LTX 2.3’s natively generated mono audio with omnidirectional audio-video generation.This would enable fully immersive audiovisual experiences.
  • Future work: RoPE reshaping could extend beyond spheres to domains including cylindrical projections and hyperbolic spaces.The passage frames this as a broader extension of encoding non-Euclidean geometry.
Loading 2606.32033v1…