PaperlayerView exact revision on arXiv

Source-linked AI summary

StreetDiff: Multi-view Street Scenes Generation via Cross-view Consistent Multi-view Stable Diffusion with Structure Prompts

Qi Zhang, Yanyifan Wang, Weiyuan Zhang, Hui Huang

arXiv:2609.09890v1cs.CV

TL;DR

StreetDiff targets cross-view inconsistency in complex urban street generation, where existing methods can produce duplication, distortion, and layout errors. It combines a panorama–perspective framework, spherical alignment through PAM, and the Street360 dataset; experiments report improved visual fidelity and consistency, including over 77% user preference across three criteria.

  • Problem

    Existing multi-view diffusion methods struggle to maintain cross-view consistency in structurally complex urban scenes, while available datasets emphasize indoor or simple natural environments.

  • Method

    StreetDiff combines global panoramic and local perspective generation with structure prompts and a Panorama Alignment Module that establishes geometry-aware spherical correspondences during denoising.

  • Results

    Over 77% of users selected StreetDiff as best for Style Consistency, Reasonableness, and Multi-view Consistency in a study of 45 image sets and 100 questionnaires.

  • Takeaways & Limitations

    StreetDiff and Street360 provide a framework and dataset for realistic, structurally coherent urban multi-view street scene generation and evaluation.

Abstract

from arXiv · show

Multi-view diffusion models have shown strong performance in scenes with strong geometric priors and sparse semantics, such as indoor rooms or simple outdoor environments (e.g., fields, courtyards). However, they often fail to maintain cross-view consistency under camera rotation, especially in structurally complex urban environments. Without explicit modeling of spherical correspondence across views, existing approaches tend to produce object duplication, structural distortion, and layout inconsistency. To address this limitation, we propose StreetDiff, a multi-view diffusion framework that explicitly enforces cross-view alignment during denoising. StreetDiff introduces a Panorama--Perspective Synergy design to decouple global layout reasoning from local detail synthesis, and incorporates a Panorama Alignment Module (PAM) that establishes spherical-projection-based attention constraints across views. By injecting structured alignment constraints without modifying the diffusion backbone, our framework achieves robust cross-view coherence in challenging urban street scene generation tasks. In addition, we construct Street360, a large-scale HDR multi-view urban panorama dataset. Extensive experiments demonstrate that StreetDiff significantly improves structural consistency and visual fidelity compared to prior multi-view diffusion generation methods.

I. INTRODUCTION

StreetDiff addresses the difficulty of generating coherent urban street panoramas by introducing the Street360 dataset and a panorama–perspective framework with geometry-aware alignment. Experiments report improved generation quality and multi-view consistency over prior methods.

  • I. INTRODUCTION: Existing multi-view and panorama methods remain limited for complex outdoor street scenes because datasets and models emphasize indoor or simple natural environments.These settings do not adequately represent urban layouts and dynamic elements.
  • I. INTRODUCTION: Street360 is a large-scale HDR multi-view urban street panorama dataset with aligned text prompts and dense structural annotations.It is designed for realistic urban street scene generation and evaluation.
  • I. INTRODUCTION: StreetDiff combines a global panoramic representation with a geometry-aware Panorama Alignment Module to enforce spherical cross-view consistency during denoising.The framework is presented as a panorama–perspective synergy design.
  • I. INTRODUCTION: The method uses structure prompts to guide street image generation and establishes geometry-aware correspondences between panoramic and perspective representations.Supported prompts include segmentation maps, contour maps, and user sketches.
  • I. INTRODUCTION: StreetDiff is reported to achieve the best street scene generation performance, surpassing previous models in generation quality and multi-view consistency.The introduction presents this as the result of extensive experiments.

III. STREETDIFF MODEL

StreetDiff organizes textual and structural conditions in a Global–Local Prompt Space, then supplies them to coordinated panoramic and multi-view generation branches. The design supports both global scene context and view-specific structure.

  • III. STREETDIFF MODEL: Global text is formed by concatenating local texts, while multi-view structure prompts are obtained by segmenting the panoramic structure prompt.This preserves shared content across the panoramic and perspective conditions.
  • III. STREETDIFF MODEL: Local and global conditions are inserted into the multi-view and panoramic branches, respectively, while both derive their semantic information from the same scene content.The prompts differ in whether descriptions are supplied separately or as one merged description.
  • III. STREETDIFF MODEL: StreetDiff generates eight perspective views with 90° horizontal fields of view and 45° overlap, which are subsequently stitched into a panorama.The views are produced within the Panorama–Perspective Synergy Framework.
  • III. STREETDIFF MODEL: The Global–Local Prompt Space integrates local and global text descriptions with corresponding structural prompts.It bridges panoramic images and their multi-view representations at different semantic granularities.
  • III. STREETDIFF MODEL: The model supports segmentation maps, contour maps, and user-input sketches as structural prompts.Segmentation maps are divided into multi-view prompts using the same viewpoint cropping used for text prompts.

B. Panorama–Perspective Synergy Framework

The Panorama–Perspective Synergy Framework reverses the usual bottom-up panorama synthesis strategy by using a panoramic representation as a global prior for perspective-view generation. Its branches divide global layout reasoning from local texture synthesis.

  • B. Panorama–Perspective Synergy Framework: StreetDiff uses a panoramic module and a multi-view module adapted from pretrained Stable Diffusion U-Nets, collaborating during diffusion denoising.The resulting latent representation is decoded into multi-view outputs.
  • B. Panorama–Perspective Synergy Framework: Unlike PanFusion’s bottom-up aggregation, StreetDiff uses a global panoramic representation as a top-down prior to guide consistent perspective views.This explicitly constrains multi-view generation during denoising.
  • B. Panorama–Perspective Synergy Framework: Panorama generation results are obtained by stitching perspective images, enabling qualitative comparison of distorted lines, unrealistic items, and panoramic inconsistencies.The comparison reports that StreetDiff addresses these issues more effectively than MVDiffusion and PanFusion.
  • B. Panorama–Perspective Synergy Framework: The multi-view branch operates in pinhole-camera space to synthesize 512 × 512 local textures and view-dependent details.Structural cues from the panoramic module are injected into U-Net features.
  • B. Panorama–Perspective Synergy Framework: The panoramic branch captures holistic layout, long-range structural continuity, and global spatial priors that are difficult to model from perspective views alone.Its output serves as a structural prior rather than a final image.

C. Model Training

StreetDiff trains panoramic and multi-view branches in stages, then uses spherical projection and PAM-based cross-attention to align their features across views. The training balances panoramic and per-view denoising objectives while preserving local detail and global geometric consistency.

  • Panorama Alignment Module: PAM maps perspective pixels into panorama coordinates through camera intrinsics, rotation, spherical angles, and projection-consistent correspondence anchors.The mapping supports cross-branch attention without dense feature warping.
  • Panorama Alignment Module: PAM performs local-neighborhood cross-attention around projected source positions, using query, key, value, and Gaussian-bias features.The neighborhood schedule expands from small high-resolution windows to full-map bottleneck attention.
  • Panorama Alignment Module: The attention output is fused residually with input features through a zero-initialized 1 × 1 convolution.This design preserves the diffusion backbone's generative capacity while improving panorama–perspective consistency.
  • Training Strategy: StreetDiff uses three training stages: street-scene priors, multi-view consistency, and joint PAM refinement of panoramic and perspective branches.Stage 1 learns both normal-perspective and panoramic distributions; Stages 2–3 progressively add view alignment and spherical consistency.
  • Training Objectives: The panoramic and multi-view modules use separate denoising losses, while Stage 3 combines them as λL* + (1/N) Σ_i L_i.λ balances the panoramic loss and N is the number of views.

E. Structural Information Fusion

StreetDiff integrates structural information through latent addition, channel concatenation, and related conditioning choices for panoramic and viewpoint inputs. These mechanisms provide structural cues to guide spatial layout and generation.

  • Structural Information Fusion: Structural information is encoded into latent features and directly added to latent noise under the “Add” strategy.This is one of three methods used to inject structural cues into the Panorama–Perspective Synergy Framework.
  • Structural Information Fusion: The “Concatenation” strategy appends viewpoint structural information to multi-view noise channels and panoramic structural information to panoramic noise channels.The conditioning is applied separately to viewpoint and panoramic modules.

F. Consistency-Reward Fine-Tuning

After three-stage training, StreetDiff applies consistency-reward fine-tuning with GRPO to optimize agreement among generated views while preserving prompt alignment. The reward combines geometric agreement in overlapping regions with a CLIP-based semantic score.

  • Consistency-Reward Fine-Tuning: Consistency-reward fine-tuning samples groups of multi-view outputs and optimizes the final StreetDiff model with GRPO.The GRPO objective includes a KL regularizer toward the frozen Stage-3 model.
  • Reward Design: The reward is R = λ_geoR_geo + λ_semR_sem, combining geometric agreement in 45° overlapping regions with CLIP prompt-alignment scores.R_geo evaluates adjacent-view consistency, while R_sem measures semantic preservation.
  • Reward Design: PAM determines where cross-view information is aligned, whereas CRFT directly optimizes how consistent the generated views are.This separates geometric correspondence from post-training consistency optimization.

A. Dataset Generation and Experiment Setting

Street360 supplies high-resolution HDR urban panoramas for street-scene generation, while experiments compare StreetDiff with established methods under matched structural prompting and multiple evaluation protocols.

  • Dataset Generation: Each panorama is split into six skybox images and eight perspective images extracted from four non-polar views.The splitting strategy follows MVDiffusion’s view-splitting procedure.
  • Experiment Setting: StreetDiff is compared with MVDiffusion, PanFusion, SD+LoRA, and other panorama or multi-view generation baselines using the same structure prompts.For segmentation and contour conditioning, ControlNet is additionally trained to support fair comparison.
  • Experiment Setting: Evaluation combines FID, IS, CLIP Score, overlapping PSNR, and a user study assessing generated panoramas.Overlapping PSNR measures multi-view consistency, while participants compare outputs from different methods.

B. Street Generation Results

Across structural prompts and human evaluation, StreetDiff produces strong image quality and multi-view coherence, with especially favorable results for realistic, consistent urban panoramas.

  • Quantitative Results: With segmentation and contour prompts, StreetDiff achieves the best reported results among compared methods across the evaluated image-quality metrics.It outperforms SD+LoRA, PanFusion, and MVDiffusion on FID and leads on IS and CLIP Score; contour prompting remains strongest among the compared methods.
  • Qualitative Results: StreetDiff generates smooth transitions, coherent semantics, and realistic panoramas with minimal structural distortions.The qualitative comparison highlights preserved local details and global context while avoiding distorted lines, unnatural objects, and panoramic inconsistencies.
  • User Study Results: Over 77% of users selected StreetDiff as producing the best images across style consistency, reasonableness, and multi-view consistency.The study used 45 generated panorama sets and collected 100 valid questionnaires.

C. Ablation Study

Ablations indicate that StreetDiff’s dual-branch design, prompt integration, panoramic guidance, alignment strategy, staged training, and consistency-focused fine-tuning each affect generation quality or cross-view coherence.

  • Model architecture: The panoramic branch offers global consistency but weaker authenticity, whereas the multi-view branch improves resolution-aligned image quality but has poorer adjacent-view consistency.The full framework combines these complementary properties.
  • Prompt insertion: Cross-attention integration of ControlNet features achieves the best results among the tested structural-prompt insertion strategies.The alternatives concatenate structural information with noise or add ControlNet outputs directly to corresponding U-Net layers.
  • Street generation with user sketches: StreetDiff generates high-quality images that follow user sketch layouts even though sketches are used only at test time.Training uses contour maps as the structural prompts.
  • Only using text prompts: Without structure prompts, StreetDiff achieves the best overall performance on Street360 among the compared methods.The authors attribute this result to panoramic features guiding multi-view generation and improving cross-view consistency and visual details.
  • Training Strategy Ablation: The staged three-part training strategy outperforms end-to-end training in visual quality, cross-view consistency, convergence behavior, and overall fidelity.The stages progressively learn panoramic priors, intra-view consistency, and joint branch alignment.
  • Consistency-reward fine-tuning: Explicit consistency optimization improves cross-view consistency, while removing the KL term increases overlap agreement but sharply degrades FID through over-smoothed facades.This is identified as reward hacking, so the method retains a moderate β.

V. DISCUSSION AND CONCLUSION

StreetDiff combines structure prompts and a Panorama–Perspective Synergy Framework with the Street360 dataset to generate realistic, consistent urban multi-view scenes, including user-guided sketches.

  • V. DISCUSSION AND CONCLUSION: StreetDiff generates highly consistent and realistic multi-view street-scene images using structure prompts and a Panorama–Perspective Synergy Framework.The framework is designed to address object-structure challenges in urban street-scene generation.
  • V. DISCUSSION AND CONCLUSION: Street360 provides a large multi-view street-scene dataset for studying urban street-scene generation.The dataset and model are presented as resources for extending multi-view generation to more complicated scenarios.
  • V. DISCUSSION AND CONCLUSION: Beyond segmentation and contour maps, StreetDiff accepts user-input sketches and generates images aligned with the input.The paper identifies semantic bounding boxes and bird’s-eye-view representations as future structural-guidance directions.
Loading 2609.09890v1…