PaperlayerView exact revision on arXiv

Source-linked AI summary

Linear Fusion MultiDiffusion for Fast Training-Free Spherical Panorama Generation

Akio Hayakawa, Yusuke Mukuta, Tatsuya Harada

arXiv:2609.01997v1cs.CVcs.LG

TL;DR

Panorama generation is constrained by limited-view pretrained models, scarce panorama-domain data, and costly direct-sampling approaches. LF-MultiDiffusion uses arbitrary linear mappings with regularized least-squares aggregation and a Krylov solver, achieving improved quality and consistency with 15.36× faster inference than the best-performing training-free baseline. Its decoded ERP images can remain slightly blurrier than native perspective outputs.

  • Problem

    Panorama-specific training data has limited domain coverage, while most pretrained image generators target narrow fields of view.

  • Method

    LF-MultiDiffusion reformulates latent aggregation under arbitrary linear projections as regularized least squares solved by a matrix-free Krylov method during denoising.

  • Results

    15.36× faster inference than the best-performing training-free baseline accompanies better visual quality, text alignment, and geometric consistency.

  • Takeaways & Limitations

    Denser panorama-to-perspective mappings reduce view evaluations during denoising while maintaining more stable panoramic generation.

  • Takeaways & Limitations

    Decoded ERP images can remain slightly blurry because latent-space compression and ERP distortion weaken local details.

Abstract

from arXiv · show

We propose LF-MultiDiffusion, a training-free panorama generation method that extends MultiDiffusion to support linear projections between target and reference image spaces. Our key idea is to reformulate latent aggregation as a regularized least-squares problem and solve it efficiently with a Krylov-based iterative solver inside the denoising loop. This formulation enables denser and more natural mappings than prior training-free methods, yielding more stable generation with far fewer perspective views. As a result, LF-MultiDiffusion reduces the number of image generator evaluations during denoising and significantly improves inference efficiency. Experiments show that LF-MultiDiffusion achieves better visual quality, text alignment, and panoramic consistency than the strongest training-free baseline, while providing a 15.36$\times$ speedup. Our project page is available at: https://ahykw.github.io/lfmd.

1 Introduction

LF-MultiDiffusion addresses the limited field of view and dataset-domain constraints of panorama generation by extending MultiDiffusion with arbitrary linear projections. Its regularized, iterative formulation supports denser mappings, improves quality and consistency, and achieves substantially faster inference.

  • Motivation: Training-free approaches are attractive because panoramic datasets have scarce domain coverage, while general image models support broader visual domains.Panorama-specific models may not cover artistic or cartoon-style content that general image models can represent.
  • Contribution: LF-MultiDiffusion extends MultiDiffusion with arbitrary linear projections between target and reference image spaces.The method is designed to remove the direct pixel-sampling constraint of prior approaches.
  • Contribution: Latent aggregation is formulated as a regularized least-squares problem and solved with a Krylov-based iterative solver inside the denoising loop.The iterative update adds only 20% of the denoising-step cost.
  • Efficiency: Denser and more natural mappings stabilize generation with far fewer perspective views than prior training-free methods.This reduces the number of neural function evaluations required during denoising.
  • Results: 15.36× speedup over the best-performing baseline accompanies better visual quality, text alignment, and geometric consistency.The comparison is reported against existing training-free methods.

2 Related work

Prior panorama-generation methods either rely on limited-domain paired datasets or adapt general image generators at inference time. Training-free approaches preserve pretrained-model flexibility, but existing MultiDiffusion-based methods use direct sampling that requires dense views and high computation.

  • General image generation: Diffusion and flow-matching models dominate text-to-image generation but are typically trained for fixed aspect ratios or narrow resolution ranges.Generation outside those regimes can degrade layout and composition.
  • MultiDiffusion: MultiDiffusion variants enable arbitrary aspect ratios and higher resolutions without additional training, but are primarily designed for planar or wide-canvas imagery.These methods extend pretrained text-to-image models through region-wise denoising and fusion.
  • Training-based panorama generation: Panorama-specific diffusion methods rely on text-panorama paired datasets that are difficult to collect at scale and have limited domain coverage.Models trained mainly on indoor panoramas may generalize poorly to outdoor scenes.
  • Training-free panorama generation: Training-free panorama methods such as DynamicScaler and SphereDiff use MultiDiffusion with direct pixel sampling between panoramas and perspective views.SphereDiff represents latents on the sphere to reduce pole distortion.
  • Training-free 3D generation: Training-free text-to-3D methods progressively expand scenes from generated views, which can introduce loop inconsistency and error accumulation.These methods optimize representations including point clouds, 3D Gaussian splatting, and Gaussian surfels.

3 Method

LF-MultiDiffusion extends MultiDiffusion from direct pixel sampling to arbitrary linear projections, formulating latent aggregation as regularized least squares solved iteratively during denoising. This supports denser mappings, reduces view evaluations, and combines efficient optimization with view-wise refinement to recover local detail.

  • MultiDiffusion: MultiDiffusion generates larger target images by extracting reference views, denoising them with a pretrained model, and aggregating the results.The target panorama typically has greater spatial extent than the reference image, and each view receives a corresponding condition.
  • Limitations of direct sampling: Direct pixel sampling leaves panorama pixels uncovered, forcing prior training-free methods to use densely overlapping views and incur high denoising cost.The diffusion model must be evaluated for every perspective view at every diffusion step; SphereDiff requires more than 30 minutes per panorama in the cited experiment.
  • Limitations of direct sampling: Bilinear mappings cover the panorama but naive pixel-wise averaging violates consistency with the pretrained model and severely degrades intermediate latents.The cited example reports that bilinear mapping leaves no pixels uncovered, while naive averaging produces a blurry result.
  • LF-MultiDiffusion: LF-MultiDiffusion replaces direct-sampling operators with arbitrary linear projections and casts latent aggregation as a regularized least-squares problem.For projection S_t,i, the method compares the projected target latent with the denoised reference prediction Φ(S_t,iJ_t | y_t,i).
  • LF-MultiDiffusion: A Krylov-based iterative solver avoids explicit matrix inversion, while matrix-free operator applications prevent projection matrices from being stored explicitly.The formulation uses a quadratic regularizer; the weighted first-order operator can yield ridge- or Laplacian-type regularization depending on D and W.

4 Experiments

Experiments show that LF-MultiDiffusion achieves strong panorama quality, alignment, consistency, and efficiency with substantially fewer reference views than training-free baselines. Ablations identify LSMR, discrete-Laplacian regularization, and N = 14 views as effective choices.

  • Comparison with baselines: LF-MultiDiffusion outperforms DynamicScaler and SphereDiff across all reported metrics while supporting 2048 × 4096 panoramas.Training-based methods struggle more with out-of-domain prompts; the comparison uses FLUX for training-free methods.
  • Comparison with baselines: 15.36× speedup over SphereDiff and 3.58× over DynamicScaler result from reducing costly perspective-view generator evaluations.The method supports dense projections without the direct-sampling constraint, lowering computational cost while improving quality.
  • User study: LF-MultiDiffusion achieves the highest subjective scores across image quality, geometric naturalness, and continuity in the user study.Figure 6 likewise reports natural views across directions and higher overall fidelity for LF-MultiDiffusion.
  • Solver and regularizer ablations: LSMR performs better than PCG across metrics, with improvement from 10 to 30 iterations but diminishing returns at 100.The solver comparison evaluates PCG and LSMR at 10, 30, and 100 iterations.
  • Solver and regularizer ablations: Ridge and discrete-Laplacian regularizers perform comparably, while discrete Laplacian with λ = 10^-4 provides consistently higher QAlign and CLIP scores.This configuration is selected as the default.
  • Number of reference perspective views: N = 14 provides the best overall trade-off among quality, consistency, and runtime, whereas N = 6 harms distortion and continuity.More than 14 views slightly improve some metrics, degrade others, and increase runtime.

5 Conclusion

LF-MultiDiffusion extends MultiDiffusion to arbitrary linear projections between perspective views and ERP panoramas through regularized least-squares optimization and a matrix-free Krylov solver. The method improves generation quality, text alignment, and geometric consistency while achieving about 15.36× faster inference than the strongest training-free baseline.

  • Contribution: LF-MultiDiffusion extends MultiDiffusion to arbitrary linear projections between perspective views and ERP panoramas.The formulation removes the direct-sampling constraint and supports denser mappings.
  • Contribution: A regularized least-squares update solved with a matrix-free Krylov method reduces the number of view evaluations during denoising.The method enables denser and more natural mappings between panorama and perspective views.
  • Conclusion: About 15.36× faster inference accompanies improvements in generation quality, text alignment, and geometric consistency.These are reported relative to the best-performing training-free baseline.
  • Future work: The formulation suggests extending optimization-based synthesis to other non-planar domains, surface topologies, and mesh textures.The paper specifically mentions torus-like representations and direct texture optimization on mesh surfaces.

A Details of the user study

The user study compares DynamicScaler, SphereDiff, and LF-MultiDiffusion using panorama-video pairs rated on image quality, distortion, and continuity. LF-MultiDiffusion receives the highest image-quality ratings and comparable or slightly better geometric ratings than SphereDiff, with faster inference.

  • Study setup: The study samples four prompt sets and compares three methods using panorama-video pairs showing multiple viewing directions.Each panorama is accompanied by a 10-second video.
  • Study setup: Each participant rates 12 panorama-video pairs on image quality, distortion, and continuity using a 5-point Likert scale.The scale ranges from 1 (Poor) to 5 (Excellent).
  • Study setup: 132 ratings per aspect were collected from 11 participants, with 95% confidence intervals reported using the standard error.Table 2 summarizes the main-paper results.
  • Results: LF-MultiDiffusion achieves the highest image-quality ratings while matching or slightly exceeding SphereDiff on distortion and continuity with significantly faster inference.The study evaluates subjective visual quality and seamlessness across the three methods.
  • Solver implementation: PCG and LSMR solve the regularized least-squares update using matrix-free operators, with LSMR formulated as an equivalent augmented least-squares problem.PCG instead uses a regularized normal equation and a diagonal preconditioner.
  • Solver implementation: The PCG algorithm initializes the panorama feature and residual, iteratively updates the search direction, and returns the optimized panorama feature.Iterations stop on convergence or after the configured iteration limit.

C Details of the VLM-based evaluation

The VLM-based evaluation uses Qwen2.5-VL to assess panoramic distortion and continuity under a protocol adapted from SphereDiff. The prompt is modified to emphasize geometric criteria rather than image content or artistic style.

  • Evaluation protocol: Qwen2.5-VL evaluates panoramic Distortion and Continuity using the VLM-based procedure introduced in SphereDiff.The evaluation uses an open-source vision-language model.
  • Evaluation protocol: The evaluation prompt explicitly instructs the model to ignore image content and artistic style when judging geometric criteria.The change addresses lower scores caused by content or style under the original prompt.

D Experiments with SANA

The SANA experiments evaluate LF-MultiDiffusion with a computationally efficient text-to-image backbone, using the LSMR solver within the denoising process. The supplied passages identify the solver inputs, iterative updates, and convergence check, while Table A1 concerns evaluation prompts for distortion and continuity.

  • SANA evaluation: SANA is evaluated as an alternative base text-to-image model for LF-MultiDiffusion.SANA has a highly compressed latent space and a linear diffusion transformer.
  • LSMR solver: The LSMR solver takes an initial panorama feature, stacked denoised targets, projection operators, an iteration limit, and a tolerance.These are the algorithm's required inputs.
  • LSMR solver: Each LSMR iteration updates the panorama feature and residual, then stops when the normalized residual falls below the tolerance.The procedure returns the optimized panorama feature after convergence or the iteration limit.
  • Evaluation: Table A1 uses Qwen2.5-VL to assess distortion and continuity in generated panoramas.Its prompt is adapted from SphereDiff to isolate geometric consistency from image content and style.

E Panoramic Video Generation

LF-MultiDiffusion is adapted for text-to-panoramic video generation with LTX-Video and compared with training-based and training-free baselines. Additional experiments examine spherical latents and forward or backward projections, while the supplied results report a faster runtime than SphereDiff with mostly better metrics.

  • Video generation: LF-MultiDiffusion is adapted to text-to-panoramic video generation using LTX-Video as the pre-trained text-to-video generator.The comparisons include 360DVD, ViewPoint, and SphereDiff.
  • Video generation: 25m 14s to 2m 44s: LF-MultiDiffusion reduces runtime over SphereDiff while achieving better scores on most metrics except comparable QAlign.The SphereDiff-versus-LF-MultiDiffusion runtime comparison is direct because both use the same output size.
  • Latent and projection variants: The variant experiments compare ERP and spherical latent representations with backward bilinear and forward splatting projections.The backward projection is the main-paper design, while the forward variant scatters 3D-point features to neighboring perspective pixels.
  • Latent and projection variants: The forward projection is also evaluated with the ERP latent representation.This extends the projection comparison beyond the spherical latent setting.

F.3 Performance comparison of latent and projection variants

The latent and projection study tests alternative representations and projection directions, then selects ERP with backward projection as the default based on runtime and overall performance. The study also finds that larger spherical point sets and higher ERP resolution can further improve results, while stopping-time ablations support both core aggregation and post-refinement.

  • Comparison setup: Table A4 reports panorama generation results across different latent and projection designs.The table organizes the comparison underlying the variant analysis.
  • Projection variants: Forward projection is slower than backward projection because it projects all Ns points into every perspective view at each update.It also slightly degrades all evaluation scores, possibly because denser overlap causes excessive averaging and smoother representations.
  • Latent variants: Spherical latents improve the Aesthetic score while maintaining comparable performance on other metrics, so ERP with backward projection is adopted by default.The choice considers both runtime and overall performance.
  • Generalization: LF-MultiDiffusion generalizes across ERP and spherical latent representations as well as different projection operators.Larger Ns in spherical latents and higher ERP panorama resolution can further improve results.
  • Stopping-time sensitivity: TM = 0 produces disconnected panoramas, whereas TM = 28 degrades image quality and text alignment without view-wise post refinement.Intermediate values perform better overall, and TM = 23 is selected for balanced performance.

H Additional generated samples

Additional samples illustrate panoramas generated by LF-MultiDiffusion. The accompanying prompts describe the upper, horizontal, and lower scene regions and are automatically generated from scene labels.

  • Additional samples: Figure A1 presents additional panoramas generated by LF-MultiDiffusion.The samples use three prompts per scene following the SphereDiff setup.
  • Prompt construction: Prompts describe the upper, horizontal, and lower parts of each scene and are automatically generated from scene labels using Qwen3.5-2B.The supplementary material also provides a comparison demo with panoramas and perspective-view videos.
Loading 2609.01997v1…