SyncDiffusion: Coherent Montage via Synchronized Joint Diffusions

1 Introduction

• Joint diffusion averages overlapping noisy features to make seamless montages, but can mix content and styles across views because it does not enforce coherence.Earlier sequential inpainting can also struggle to extend images seamlessly and repeat patterns.Source: §1 · p. 2
• SyncDiffusion guides reverse diffusion with gradients from perceptual similarity across windows, computed on predicted denoised images at each step.It combines this guidance with latent-feature averaging for local seamlessness and global coherence, without retraining or fine-tuning.Source: §1 · p. 2
• 66.35% vs. 33.65%: user-study participants preferred SyncDiffusion over the previous method for coherence.The experiments also report maintained fidelity and prompt compatibility, with a coherence–diversity trade-off and better FID and KID scores than the baseline.Source: §1 · p. 2
• SyncDiffusion is demonstrated in layout-guided generation, conditional image generation, and 360-degree panorama generation.The experiments use Stable Diffusion 2.0 for text-guided panorama generation.Source: §1 · p. 2

2.1 Diffusion Models

• Diffusion models sample from an initial Gaussian distribution through iterative denoising and have been applied to images, audio, video, and 3D objects.DDIM reduces reverse-process sampling time relative to DDPM, while latent diffusion performs the process in latent space.Source: §2.1 · p. 3

2.2 Few-Shot or Zero-Shot Adaptation of Diffusion Models

• Prior work adapts pretrained diffusion models through few-shot fine-tuning or zero-shot conditioning, and this paper proposes guided joint diffusion for image montages.ControlNet and Custom Diffusion use few-shot tuning; SDEdit and RePaint illustrate zero-shot applications.Source: §2.2 · p. 3

2.3 Montage Generation via Diffusion Models

• Inpainting-based and joint-diffusion montage methods can create continuous images, but prior joint approaches do not enforce global coherence across panoramas or textures.SyncDiffusion is presented as a module that can integrate with any joint-diffusion process to pursue global semantic coherence.Source: §2.3 · p. 3

3.1 Diffusion Models

• DDPM models data generation as a Markov chain and learns reverse-process parameters by minimizing the negative evidence lower bound.Its forward process adds Gaussian noise with increasing noise scale, with a starting distribution that approaches unit Gaussian noise.Source: §3.1 · pp. 3–4
• DDIM predicts denoised data to form reverse transitions and can sample a subset of timesteps, reducing reverse-process computation.The same simplified objective permits use of a pretrained DDPM without retraining.Source: §3.1 · pp. 4–5

3.2 Joint Diffusion

• For fixed-size image models, MultiDiffusion uses multiple windows and averages their colors or latent features at each reverse-diffusion step.The windows cover the panorama, and their noisy images are combined in panorama space before the result is cropped back to each window.Source: §3.2 · p. 5
• Figure 2 compares MultiDiffusion on the left with SyncDiffusion on the right for a twilight mountain-range prompt; the caption describes SyncDiffusion panoramas as more coherent.The caption notes that MultiDiffusion can combine different scenes and blend them awkwardly.Source: Figure 2 · p. 5

4 SyncDiffusion

• MultiDiffusion aligns colors or features in overlapping regions but not content or style, and only adjacent views influence each other.This limits global coherence between distant windows.Source: §4 · p. 5
• Noisy-image LPIPS scores are indistinguishable, while predicted-denoised-image scores align with final outputs and provide guidance for coherence updates.The update is applied to the noisy images for all windows during each denoising step.Source: Figure 3 · p. 6, §4 · pp. 6–7
• SyncDiffusion updates noisy images by backpropagating a perceptual-similarity loss across windows, using LPIPS or Style Loss.The module can be integrated into an existing joint-diffusion framework.Source: §4 · p. 6
• SyncDiffusion produces seamless panoramas with improved coherence over MultiDiffusion [3], which can blend distinct skies and backgrounds.Blended Latent Diffusion [1] tends to show visible seams and repetitive patterns.Source: Figure 4 · p. 7

5.1 Text-Guided Panorama Generation

• SYNCDIFFUSION produces visually and semantically more coherent panoramas than Blended Latent Diffusion and MultiDiffusion in the qualitative comparisons.Blended Latent Diffusion often shows visible seams and repetitive patterns; MultiDiffusion is seamless but can mix different scenes.Source: §5.1 · p. 8
• At w = 20, Intra-LPIPS and Intra-Style-L are about 3/4 and 1/6 of Stable Diffusion reference scores, while Mean-CLIP-S and Mean-GIQA remain comparable.The comparison reports the reference set as SD (Stable Diffusion).Source: §5.1 · p. 8, Figure 5 · p. 9
• As w increases, FID and KID rise slightly, indicating a coherence–diversity trade-off, but remain much lower than Blended Latent Diffusion.Blended Latent Diffusion also has low Mean-GIQA and very high FID and KID, indicating degraded fidelity.Source: §5.1 · p. 8, Figure 5 · p. 9

5.2 Additional Applications of SYNCDIFFUSION

• In layout-guided generation, SYNCDIFFUSION produces a natural, globally coherent background where MultiDiffusion produces incoherence around the house and bear.This is demonstrated in Figure 6-(A).Source: §5.2 · p. 9
• With ControlNet [51], SYNCDIFFUSION generates coherent panoramas that reflect the supplied Canny edge map.The integration extends conditional image generation to arbitrary resolutions.Source: §5.2 · p. 9
• With MVDiffusion [47], SYNCDIFFUSION improves 360-degree panorama coherence; views from different angles better depict the same room.Vanilla MVDiffusion views appear to show two different rooms.Source: §5.2 · pp. 9–10
• Realistic results rely on appropriate input prompts, and SYNCDIFFUSION adds computational overhead from a neural-network forward pass and gradient descent.Source: §5.2 · p. 10

Appendix

• Appendix materials cover quantitative evaluation, panorama resolutions, perceptual-loss comparisons, and an ablation substituting Eq. 13 for Eq. 14.These topics are listed as supplementary results.Source: Appendix · p. 10
• Additional appendix sections analyze computation time and describe the user-study details.The document also points to further qualitative comparisons.Source: Appendix · p. 10
• Figure 7’s failure cases emphasize that realistic panorama generation requires a suitable input prompt.The caption does not specify which prompts are suitable.Source: Figure 7 · p. 10

6 Conclusion

• SYNCDIFFUSION is a synchronization module designed to generate coherent montages through joint diffusions.It updates noisy images during the reverse process using gradient descent on perceptual similarity loss computed from denoised-image predictions.Source: §6 · p. 10
• Generating textures for 3D models is identified as a possibility for future investigation.The conclusion does not report a texture-generation evaluation.Source: §6 · p. 10
• The paper notes potential harms from generative images, including deepfakes, copyrighted-material resemblance, bias or discrimination, and harmful outputs.It calls for research on manipulated-content detection and societal barriers to protect intellectual property.Source: §6 · p. 10

Appendix

• The appendix roadmap lists further qualitative comparisons, quantitative evaluations, resolution tests, loss-function comparisons, an ablation, computation-time analysis, and user-study details.The quantitative evaluation varies gradient-descent weights.Source: Appendix · p. 14

6.1 More Qualitative Results with Various Prompts

• Qualitative examples include horizontal panoramas at 512 × 3072 and vertical panoramas at 2048 × 512.The subsection presents images for varied prompts.Source: §6.1 · p. 14
• Further prompts depict a New York skyline, a La La Land-style beach illustration, a waterfall, railway, shooting stars, rock concert, brick-wall vines, and an alley with shops.The figures label comparisons as MD [3] and SYNCDIFFUSION.Source: §6.1 · pp. 14–16
• Shown prompts include beaches, grassland animals, a sunset castle, a beachside street, coral reef, and a lake beneath northern lights.These examples span natural and built scenes.Source: §6.1 · pp. 16–17

6.2 Details About Quantitative Evaluation

• Intra-LPIPS falls from 0.69 (w = 0) to 0.56 (w = 20), while Intra-Style-L falls from 2.98 (w = 0) to 1.39 (w = 20).The passage describes these changes as significant improvement in global coherence as the gradient-descent weight increases.Source: §6.2 · p. 18
• As w increases, panorama images become more globally coherent, with the left and right sides growing more coherent than in MultiDiffusion.Figure 8 provides the qualitative comparison across weights.Source: Figure 8 · p. 18

6.3 Quantitative Evaluation on Different Resolutions

• Across 512 × 3072, 512 × 2048, and 512 × 1024 panoramas, SyncDiffusion maintains a coherence advantage over MultiDiffusion, while the FID and KID gaps increase with resolution.The authors hypothesize that longer panoramas make it harder to find images that can be merged into one coherent panorama.Source: §6.3 · p. 18, Table 3 · p. 19
• Longer globally coherent panoramas can reduce generation diversity, negatively affecting FID and KID.Source: §6.3 · p. 19

6.4 Results of SYNCDIFFUSION with Style Loss

• With Style Loss, SyncDiffusion improves coherence over MultiDiffusion on Intra-LPIPS and Intra-Style-L, but worsens Mean-GIQA, FID, and KID.Intra-Style-L decreases significantly with Style Loss guidance.Source: §6.4 · p. 19

6.5 Ablation on Predicting the Foreseen Denoised Observation

• Using decoded noisy images instead of foreseen denoised observations yields only a negligible Intra-LPIPS reduction versus MultiDiffusion, unlike the original formulation.The authors attribute the difference to noisy-image perceptual loss providing no meaningful guidance, while comparing foreseen denoised observations can guide coherence.Source: §6.5 · p. 19

6.6 Analysis on the Computation Time

• At f = 10, Intra-LPIPS decreases from 0.69 to 0.62 and Intra-Style-L from 2.98 to 2.14; applying gradient descent at fixed intervals reduces computation time versus f = 50.The study also tests f = 5; applying gradient descent at every step gives the highest reported coherence, with Intra-LPIPS of 0.56 and Intra-Style-L of 1.39.Source: §6.6 · pp. 19–20
• Less than 70% of the f = 10 computation time is needed when applying gradient descent only for the initial three steps, with analogous coherence and better fidelity and diversity than f = 10.The comparison fixes the gradient descent weight at w = 20.Source: §6.6 · p. 21

• The study collected 20 responses for each of 200 questions presenting random panorama pairs, with image order shuffled.Source: §6.7 · p. 21
• Of 100 participants, 86, 90, and 84 passed all vigilance tasks for coherence, image quality, and prompt compatibility, respectively.The tasks distinguished the study outputs from concatenations of Stable Diffusion images generated without joint diffusion.Source: §6.7 · p. 21
• Participants were US-based Amazon Mechanical Turk Masters; average completion time was 248.21 seconds, with $0.76 compensation per person.The paper reports this as $11.02 per hour, above the US federal minimum wage.Source: §6.7 · p. 21

• Supplementary qualitative results include a city skyline at night and a misty forest.Source: §6.8 · pp. 23, 25
• The listed prompts also include snowy mountain peaks with skiers and mountain ranges at twilight.Source: §6.8 · pp. 26–27
• A further prompt depicts a cartoon panorama of spring-summer nature.Source: §6.8 · p. 27