PaperlayerView exact revision on arXiv

Source-linked AI summary

PanoWorld: A Generative Spatial World Model for Consistent Whole-House Panorama Synthesis

Jinrang Jia, Zhenjia Li, Yijiang Hu, Yifeng Shi

arXiv:2605.17916v2cs.CV

TL;DR

Whole-house VR synthesis must preserve both photorealistic panorama quality and spatial coherence across rooms and viewpoints. PanoWorld combines node-based autoregressive panoramas with a floorplan shell and progressive 3DGS memory, preserving high-frequency texture while improving cross-node layout and material consistency.

  • Problem

    Whole-house panorama synthesis must preserve geometry, furniture layout, materials, and fine details across rooms and viewpoints, beyond single-view realism.

  • Method

    PanoWorld autoregressively generates node-based 360-degree panoramas using a floorplan-derived shell, room-aware panoramic LRM, and topology-aware progressive 3DGS cache.

  • Results

    PanoWorld preserves high-frequency texture quality while improving cross-node layout and material consistency across multi-room indoor tours.

  • Takeaways & Limitations

    Separating shell geometry and cache-rendered memory from appearance generation supports rapid global restyling of whole-house panorama tours.

  • Takeaways & Limitations

    Performance can weaken with imperfect floorplan-to-shell geometry, missing doorway topology, sparse panorama nodes, and challenging dynamic or reflective scene elements.

Abstract

from arXiv · show

Generating a consistent whole-house VR tour from a floorplan and style reference requires both photorealistic panoramas and cross-view spatial coherence. Pure 2D generators produce appealing single panoramas but re-imagine geometry and materials when the viewpoint changes, whereas monolithic 3D generation becomes expensive and loses fine texture at multi-room scale. We introduce PanoWorld, a generative spatial world model that treats whole-house synthesis as autoregressive generation of node-based 360-degree panoramas, matching the discrete navigation used by real VR tour products. PanoWorld uses a floorplan-derived 3D shell as a global geometric proxy and a dynamic 3D Gaussian Splatting cache as renderable spatial memory. A feed-forward panoramic LRM designed for metric-scale multi-room 360-degree inputs lifts generated panoramas into local 3DGS updates, while Room-aware Group Attention suppresses cross-room feature interference. A topology-aware progressive caching strategy fuses these local updates without repeatedly reconstructing the full history. By decoupling shell-based geometry guidance from cache-rendered visual memory, PanoWorld preserves high-frequency 2D synthesis quality while improving cross-node layout and material consistency. The project link is https://jjrcn.github.io/PanoWorld-project-home/

1. Introduction

PanoWorld addresses whole-house panorama synthesis as node-based autoregressive generation, combining floorplan-derived geometric guidance with renderable 3D visual memory. Its room-aware reconstruction and caching design targets cross-node consistency while retaining photorealistic texture quality.

  • Problem: Whole-house synthesis must preserve geometry, furniture layout, material identity, and fine details across overlapping views spanning rooms, doorways, corridors, and long-range visibility.The challenge extends beyond single-view realism because viewpoint changes must remain spatially coherent throughout the tour.
  • Limitations: 2D diffusion models produce rich, realistic panoramas but lack persistent spatial memory, while global 3D representations are costly and can lose fine texture at multi-room scale.Without persistent memory, repeated views may regenerate doorways, walls, or sofas with different shapes, positions, or materials.
  • Formulation: PanoWorld formulates whole-house synthesis as node-based generation aligned with commercial VR tours, producing high-resolution panorama nodes supported by lightweight renderable 3D memory.Users inspect one panorama node and jump to another nearby node rather than navigating a continuous 6-DoF environment.
  • Method: The method converts a floorplan into a coarse 3D shell that supplies global coordinates, room boundaries, doorway connectivity, and viewpoint visibility for geometric guidance.The shell is rendered at target and auxiliary viewpoints and is not intended to be the final visual asset.
  • Method: A feed-forward panoramic LRM reconstructs metric-scale multi-room 360-degree inputs, while Room-aware Group Attention limits cross-room interference through dense within-room and restricted boundary communication.The module is presented as the first LRM-style approach for whole-house multi-room reconstruction from multi-view panoramas in a single feed-forward pass.
  • Design principle: PanoWorld separates geometric guidance from appearance memory: the floorplan shell constrains large-scale structure, while the 3DGS cache preserves colors, materials, and high-frequency details across overlapping views.This decoupling is designed to retain photorealistic texture quality without losing cross-node consistency.

2. Related Work

Prior work advances panoramic synthesis, feed-forward image-to-3D reconstruction, and indoor scene synthesis using structural priors. However, these directions generally focus on single panoramas, shared compact targets, or interior layouts rather than whole-house tours with room-level topology.

  • Panoramic Image Synthesis: Recent diffusion systems improve 360-degree panorama quality and seam consistency through outpainting, correspondence-aware generation, environment expansion, and projection-aware synthesis.These methods mainly target one panorama or a synchronized view set.
  • Large Reconstruction Models: Large Reconstruction Models rapidly lift images into 3D representations, including NeRF, mesh, multi-view, and Gaussian representations.Representative systems include LRM, Instant3D, TripoSR, pixelSplat, GS-LRM, LGM, and M-LRM.
  • Large Reconstruction Models: Most LRM-style systems target objects or compact scenes whose input views describe a shared target, unlike whole-house panoramas with room-level topology.Views from different rooms may belong to different spatial contexts.
  • Indoor Scene Synthesis: Indoor scene synthesis uses structural priors such as scene graphs and floorplans for 3D arrangement, textured meshes, and autoregressive furniture-layout generation.Representative approaches include Graph-to-3D, Plan2Scene, ATISS, and SceneFormer, alongside diffusion models such as HouseDiffusion and DiffuScene.

3. Method

PanoWorld autoregressively generates node-based 360-degree panoramas from a floorplan and style condition while maintaining a dynamic 3DGS cache as spatial memory. It combines shell-based geometric guidance, topology-selected panoramic reconstruction, room-aware attention, and local cache updates to preserve whole-house consistency.

  • Overall formulation: PanoWorld represents a VR tour as a topological node graph and autoregressively synthesizes panoramas while maintaining a renderable 3DGS cache.Panoramas are the primary deliverable; the cache provides memory and guidance rather than a perfect continuous 6-DoF asset.
  • Geometric guidance: The floorplan becomes a coarse 3D shell whose rendered geometric proxies constrain walls, openings, floors, ceilings, and room layout.The shell pipeline is assumed to provide room labels and doorway connectivity, while the proxy supplies stable low-frequency structural constraints.
  • Panoramic LRM: A panoramic LRM predicts local Gaussian primitives from topology-selected multi-view 360-degree contexts in one feed-forward pass, avoiding full-history reconstruction at every node.Each panorama is encoded equirectangularly, and the decoder predicts Gaussian parameters in the global coordinate frame.
  • Room-aware attention: Room-aware Group Attention permits dense same-room interaction and restricts cross-room communication to topological boundaries, reducing interference between physically separated regions.Unrelated cross-room attention logits are masked, while doorway-connected nodes can exchange information.
  • Progressive caching: PanoWorld decouples shell-derived geometry from cache-rendered appearance and updates spatial memory through bounded local topology-aware increments.The bounded context keeps per-node reconstruction cost approximately constant while the cache grows into whole-house memory; nearby panoramas carry style and appearance context.

4. Experiments

PanoWorld is evaluated on large-scale synthetic and real panorama data using dedicated whole-house synthesis and reconstruction benchmarks. Across quantitative, qualitative, and ablation studies, it shows strong multi-node consistency, reconstruction quality, and benefits from visual memory, nearby-view conditioning, and topology-aware attention.

  • Datasets and evaluation: Training uses 3D-FRONT and RealSee3D for 3D/depth supervision, while 2.5M private 2D panoramas improve only visual synthesis quality.The data include approximately 200K rendered 3D-FRONT panoramas, 299,073 RealSee3D panoramas, and 10K RealSee3D house scenes.
  • Datasets and evaluation: The private evaluation set contains seven real floorplans, three styles per floorplan, and 126 panoramas sampled from 42 viewpoints.Each viewpoint includes shell-rendered placeholder images and depth maps for evaluating image quality, style consistency, and cross-node consistency.
  • Panorama synthesis comparison: 22.1365 Overlap PSNR is achieved by PanoWorld, exceeding OmniRoam by 5.75 dB, while Nano Banana 2 leads HPSv3 at 9.5483 and CLIP-I Style at 0.7940.Overlap PSNR measures cross-node consistency; HPSv3 measures single-node aesthetics, and CLIP-I Style measures image-reference style consistency.
  • Whole-house reconstruction: PanoWorld obtains the best panoramic-LRM reconstruction quality with both 8-panorama and 12-panorama inputs.The 12-panorama result is slightly lower because broader spatial coverage introduces more cross-room visibility changes and makes global fusion more challenging.
  • Qualitative results: Qualitative results show sharper local textures, coherent wall-door geometry, and preserved cross-room geometry and material identity across viewpoints and target styles.Competing methods exhibit blur, structural drift, or cross-room feature interference under multi-room inputs.
  • Ablation studies: Ablations identify visual memory, nearby-view conditioning, CPRoPE, and especially Room-Aware Group Attention as important for multi-room consistency and reconstruction.Removing RAGA causes the largest degradation, while removing visual memory or nearby-view conditioning tests their contributions to spatial memory and local appearance context.

5. Discussion and Limitations

PanoWorld combines 2D texture fidelity with spatially disciplined 3D memory in node-based indoor panorama tours and enables rapid global restyling. Its limitations primarily arise from imperfect geometry and sparse observation.

  • Strengths: PanoWorld matches the node-based structure of real indoor panorama tours while combining 2D texture fidelity with renderable 3D memory.The approach separates shell geometry and cache-rendered memory from the final appearance generator.
  • Strengths: Separating shell geometry and cache-rendered memory from the appearance generator supports rapid global restyling.
  • Limitations: The main limitations stem from imperfect geometry and sparse observation, including floorplan-to-shell errors, missing doorway topology, and overly large panorama spacing.

6. Conclusion

PanoWorld is a generative spatial world model for consistent whole-house panorama synthesis. It combines autoregressive node generation, room-aware panoramic lifting, topology-aware 3DGS caching, and decoupled geometry-appearance guidance to target high-fidelity, consistent multi-room tours.

  • Core contribution: PanoWorld generates whole-house panoramas autoregressively over navigation nodes.This formulation matches the node-based structure of multi-room indoor tours.
  • Core contribution: The model combines a room-aware panoramic LRM with topology-aware progressive 3DGS caching.These components support local panoramic processing and progressive spatial memory across the tour.
  • Core contribution: Decoupled geometry-appearance guidance targets high-fidelity furnished panoramas while preserving cross-node layout and material consistency.The stated goal is consistent visual synthesis across multi-room indoor tours.

Panorama Synthesis

Panorama synthesis depends on circular panoramic position encoding to maintain horizontal consistency and avoid seam artifacts.

  • Removing circular panoramic encoding causes left-right inconsistency and seam artifacts in generated panoramas.

7. Visualization without Panoramic Position Encoding

Without Panoramic Position Encoding, the generator treats the panorama’s left and right boundaries as distant rather than adjacent rays. This causes seam inconsistencies in structures and textures, including broken walls, discontinuous furniture, and mismatched lighting.

  • Boundary Modeling: Removing circular horizontal encoding makes the generator treat the panorama’s left and right boundaries as distant image regions.The boundaries should represent adjacent rays, but the variant lacks this circular relationship.
  • Seam Consistency: The missing adjacency often produces inconsistent structures or textures across the panorama seam.Reported failures include broken wall patterns and discontinuous furniture edges.
  • Seam Consistency: Mismatched lighting at the panorama boundary is another qualitative failure of the variant without Panoramic Position Encoding.The failure cases show that seam problems affect both appearance and structural continuity.

8. Baseline Adaptation Details

This section details how competing baselines are adapted for whole-house panorama generation. Image-editing baselines generate target panoramas independently, while OmniRoam is adapted through progressive video generation along navigation paths.

  • Pano2room: Pano2room estimates depth from perspective sub-images, reprojects results panoramically, and uses iterative render-then-estimate refinement to extend scenes.The decomposition increases algorithmic complexity and compromises depth consistency across viewing directions.
  • Image-editing baselines: Nano Banana 2 and Seedream-4.5-Edit are adapted as image-editing baselines without persistent 3D memory, generating each target panorama independently from geometry and style inputs.The following adaptations specify each baseline’s model, prompt, and input simplification.
  • Nano Banana 2: Nano Banana 2 uses Gemini-3.1-flash-image-preview with a geometry-control image, a style reference, and a fixed descriptive prompt.The prompt conditions spatial structure and furniture layout on the first image and style, materials, colors, and textures on the second.
  • Seedream-4.5-Edit: Seedream-4.5-Edit converts the shell image into a line drawing and uses a simplified fixed prompt to improve adherence to panoramic spatial structure.Directly using the shell-rendered image with a complex prompt often fails because the model weakly handles equirectangular distortion and these conditions.
  • OmniRoam: OmniRoam is adapted via progressive video generation, using a reprojected frame near each target node as the visual condition for the next panoramic segment.Repeating this process simulates a multi-node panorama tour along the planned navigation path.

9. Training Data Visualization

Figure 11 visualizes PanoWorld’s training data from 3D-FRONT and RealSee3D, spanning rendered panoramas, depth or shell-proxy images, and room-level BEV maps. The visualizations show how synthetic CAD-derived and reconstructed real indoor scenes support metric-scale multi-room panoramic training.

  • Training Data Sources: Figure 11 presents representative training examples from 3D-FRONT [7] and RealSee3D.The examples cover both synthetic CAD-derived scenes and reconstructed real indoor scenes.
  • Training Data Representations: The visualized modalities include rendered panoramas, depth or shell-proxy images, and room-level BEV maps.Together, these representations describe the inputs used to construct training views.
  • Spatial Organization: BEV maps illustrate floorplan topology, room partitions, doorway connectivity, sampled camera nodes, and local room groups.These elements expose the spatial structure used to construct the training views.
  • Training Scope: Both data sources support metric-scale multi-room panoramic training while highlighting differences between synthetic and reconstructed indoor scenes.The visualizations clarify the complementary scene origins represented in the training data.

10. Additional Experimental Details

This section details the evaluation setup for reproducibility, including panorama resolution, node sampling, and overlap-mask construction. It also defines cross-node consistency measurement by projecting densely sampled co-visible shell regions into panorama nodes and computing PSNR.

  • The supplementary implementation details cover panorama resolution, node sampling rules, and overlap-mask construction for cross-view PSNR.
  • Cross-node consistency is evaluated on manually selected co-visible 1m × 1m surface patches defined on the 3D shell asset.Selected regions include walls, floors, paintings, and other wall-mounted decorations.
  • Each patch is sampled every 1cm along two surface axes, producing 100 × 100 = 10,000 3D points for projection.
  • Sampled points are projected into the initial and evaluated panorama nodes using camera extrinsics and equirectangular projection, after which pixel MSE and PSNR are computed over valid co-visible samples.
  • Final overlap PSNR averages across all selected regions and evaluated nodes, emphasizing planar regions that limit self-occlusion and expose appearance drift.The evaluation targets inconsistencies such as a white wall becoming patterned wallpaper or acquiring different material details across nodes.
Loading 2605.17916v2…