PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniX: From Unified Panoramic Generation and Perception to Graphics-Ready 3D Scenes

Yukun Huang, Jiwen Yu, Yanning Zhou, Jianan Wang, Xintao Wang, Pengfei Wan, Xihui Liu

arXiv:2510.26800v2cs.CVcs.GRcs.LG

TL;DR

Panorama-based 2D lifting can produce diverse, realistic 3D environments, but existing approaches emphasize appearance and neglect intrinsic properties needed for graphics pipelines. OmniX repurposes pretrained 2D flow-matching models for panorama generation, perception, and completion, using circular synchronization and modality-specific adapters. Experiments demonstrate effectiveness across panorama generation and perception, and the framework enables PBR-ready scene construction, though distance accuracy, metallic generalization, and efficiency remain limitations.

  • Problem

    Existing panorama-based 2D lifting emphasizes appearance and often relies on depth estimation without textures or PBR materials, limiting integration into modern graphics pipelines.

    Source: §1 · p. 2

  • Method

    OmniX repurposes pretrained 2D flow-matching models for panorama generation, perception, and completion, with circular synchronization and modality-specific adapters for seamless multimodal modeling.

    Source: §1 · p. 2

  • Results

    Experiments demonstrate OmniX's effectiveness across panorama generation and perception, and the framework enables automatic construction of immersive, PBR-ready 3D scenes from images.

    Source: §5 · p. 15

  • Takeaways & Limitations

    OmniX provides a unified framework for panorama generation, perception, and completion, with PanoX covering indoor and outdoor scenes and multiple visual modalities.

    Source: §5 · p. 15

    OmniX inherits slow training and inference from pretrained 2D flow-matching models; Euclidean-distance predictions remain insufficiently accurate, metallic prediction generalizes poorly, and 2D image priors may have limited benefit for PBR material estimation.

    Source: §5 · p. 15

Abstract

from arXiv · show

There are two prevalent ways for automatic 3D scene construction: procedural generation and 2D lifting. Among these, panorama-based 2D lifting has emerged as a promising technique, leveraging powerful 2D generative priors to produce immersive, realistic, and diverse 3D environments. In this work, we advance this technique to generate graphics-ready 3D scenes suitable for physically based rendering (PBR), relighting, and simulation. Our key insight is to repurpose 2D generative models for panorama perception of geometry, textures, and PBR materials. Unlike existing 2D lifting approaches that emphasize appearance generation and neglect the perception of intrinsic properties, we present OmniX, a versatile and unified framework for panorama generation, perception, and completion. Built upon cross-modal adapter structure and cyclic spatial operators, OmniX effectively repurposes pre-trained 2D flow matching priors for joint modeling of multimodal, seamless equirectangular representations. Furthermore, we construct a large-scale synthetic panorama dataset comprising high-quality multimodal panoramas from diverse indoor and outdoor scenes. Extensive experiments demonstrate the effectiveness and generality of OmniX as a unified framework for panorama generation and perception across geometry, lighting, and semantics, enabling graphics-ready 3D scene generation and opening new possibilities for immersive and physically realistic virtual world creation.

1 Introduction

Building complex 3D scenes requires substantial artist and engineer effort, while panorama-based 2D lifting often emphasizes appearance and neglects intrinsic scene properties. OmniX unifies panorama generation, perception, and completion, supported by multimodal synthetic data and designs for seamless panoramas and cross-modal inputs.

Motivation

  • Complex 3D scene construction demands significant artist and engineer effort, limiting scene-data scale and hindering native 3D generative models.Digital replication can also support entertainment, interactive experiences, and simulation learning for agents or robots.Source: §1 · p. 2

Problem

  • Panorama-based 2D lifting improves cross-view consistency, but existing methods emphasize appearance and often rely on off-the-shelf depth estimation without textures or PBR materials.This limits integration of generated scenes into modern graphics pipelines.Source: §1 · p. 2

OmniX

  • OmniX repurposes pretrained 2D flow-matching models for panorama generation, perception, and completion, including image-to-panorama, panorama-to-X, and mask-guided tasks.Its unified formulation extends the 2D generative paradigm to these panorama tasks.Source: §1 · p. 2

Technical designs

  • Circular synchronization addresses discontinuous equirectangular seams, while modality-specific adapters integrate cross-modal inputs using pretrained 2D generative priors.The introduction identifies circular synchronization and cross-modal adapters as the framework's key designs.Source: §1 · pp. 2–3

Dataset

  • PanoX is a synthetic panorama dataset covering indoor and outdoor scenes, created to address scarce high-quality panoramas with dense geometric and material annotations.The dataset includes varied visual modalities and supports panorama perception.Source: §1 · pp. 2–3

Results

  • Experiments demonstrate OmniX's effectiveness in panorama generation and perception, and the framework enables automatic construction of immersive, PBR-ready 3D scenes from images.The stated scene-generation scope is graphics-ready rendering.Source: §1 · p. 3

2 Related Work

Prior inverse-rendering work uses generative models to estimate intrinsic scene properties, while panoramic inverse rendering remains underexplored. In 3D scene generation, procedural methods use rules and assets, whereas 2D lifting emphasizes appearance and often neglects intrinsic properties.

2.1 Inverse Rendering

  • Generative-model inverse-rendering work includes text-prompted PBR map generation by IntrinsiX and joint inverse and forward rendering by DiffusionRenderer.DiffusionRenderer combines G-buffer estimation with photorealistic image generation, trained on synthetic and real data.Source: §2.1 · p. 3
  • Panoramic inverse rendering remains underexplored; PhyIR recovers geometry, complex SVBRDFs, and spatially coherent illumination from indoor panoramas but is limited to indoor scenes.The paper contrasts this scope with its use of 2D generative priors across indoor and outdoor environments.Source: §2.1 · p. 3

2.2 3D Scene Generation

  • Procedural scene-generation methods use predefined rules or constraints and are scalable, but often lack diversity and realism because of their rule-based nature.Examples include CityEngine's grammar-based urban layouts and InfiniGen's terrain, material, and creature generators.Source: §2.2 · p. 3
  • 2D lifting bridges 2D inputs and 3D representations: image-based approaches reconstruct 3D scenes from single or sequential images using outpainting or depth estimation, while video-based methods leverage temporal information for coherent dynamic scenes.ImmerseGAN and MVDiffusion generate panoramas for scene synthesis.Source: §2.2 · p. 3
  • These 2D lifting methods emphasize appearance and use off-the-shelf depth estimators for geometry, neglecting albedos, normals, and PBR materials.This is the stated distinction from intrinsic-property estimation.Source: §2.2 · p. 4

3 Method

The method pairs a dense synthetic panorama dataset with a unified flow-matching framework, then applies generated panoramas and intrinsic maps to reconstruct and iteratively complete PBR-ready scenes.

3.1 Overview

  • The pipeline first constructs PanoX, then applies pretrained 2D flow matching to panorama generation, perception, and completion, and demonstrates PBR-ready scene generation.The overview presents dataset construction, OmniX, and PBR-ready scene generation as the method's three stages.Source: §3.1 · p. 4

3.2 PanoX: A Synthetic Panorama Dataset with Dense Annotations

  • PanoX contains 60,000 multimodal panoramas from eight indoor and outdoor environments and is described as the first panorama dataset with dense geometry and material annotations.The dataset includes distance, world normal, albedo, roughness, and metallic maps.Source: §3.2 · pp. 4–5

3.3 OmniX: A Unified Framework for Panorama Generation, Perception, and Completion

  • OmniX extends pretrained 2D flow matching to spatially aligned conditions; circular synchronization makes boundary consistency intrinsic, while Separate-Adapter assigns modality-specific LoRAs.Separate-Adapter is reported to outperform the other adapter structures on cross-modal panorama perception and to accommodate new input modalities with minimal impact on model weight distribution.Source: §3.3 · pp. 5–8

3.4 Application: PBR-Ready 3D Scene Generation

  • The PBR-ready scene pipeline generates multimodal panoramas, projects distance pixels into a mesh and assigns material maps, then fills occluded regions through OmniX-Fill.OmniX-Fill generates new regions while preserving previously reconstructed content.Source: §3.4 · pp. 8–9

4 Experiment

Experiments evaluate OmniX on panorama generation and perception, test its adapter and pretrained-prior designs, and demonstrate graphics workflows with generated 3D scenes.

4.1 Implementation Details

  • The implementation trains 12 Flux.1-dev-based adapter models for panorama generation, perception, and their masked variants using shared optimization settings.Training uses AdamW at learning rate 1e-4, 512 × 1024 images, batch size 1 per GPU, and four NVIDIA L40S GPUs.Source: §4.1 · p. 9

4.2 Results on Panorama Generation

  • Panorama generation uses text–image conditioning in a perspective-to-panorama setting, with evaluation on Laval Indoor and SUN360.The reported metrics include FID, KID, CLIP-FID, FAED, and CLIP Score; the first three are computed on perspective crops and FAED directly on ERP outputs.Source: §4.2 · pp. 9–10
  • OmniX achieves state-of-the-art panorama-generation performance on Laval Indoor and SUN360, with the best reported metrics differing between datasets.On Laval Indoor, it records FID 7.4, CLIP-FID 2.5, FAED 5.2, and CLIP Score 28.47; on SUN360, it records KID 0.66, CLIP-FID 6.5, and CLIP Score 27.54.Source: §4.2 · p. 10, Table 2 · p. 10

4.3 Results on Panorama Perception

  • Perception evaluation uses PanoX and Structured3D, but only PanoX trains roughness and metallic models because Structured3D lacks PBR materials.The reported metrics vary by modality, and invalid ground-truth values are excluded from calculations.Source: §4.3 · p. 11
  • OmniX achieves consistent state-of-the-art performance for albedo, roughness, and metallic prediction, and the second-highest depth accuracy alongside the highest normal accuracy.The geometry comparison notes that OmniX uses substantially less data than MoGe; the in-the-wild results are reported as satisfactory generalization.Source: §4.3 · pp. 11–12, Table 3 · p. 11, Table 4 · p. 13

4.4 Ablation Analysis and Discussion

  • The Separate-Adapter design performs best among evaluated adapter structures, while initializing from 2D generative priors consistently improves perception performance.The reported advantage of separate adapters is attributed to modality-specific adaptation that leverages pretrained priors without disrupting their original weight distribution; prior-based initialization also helps modalities with substantially different pixel-value distributions.Source: §4.4 · pp. 13–14, Table 5 · p. 13
  • Circular synchronization produces nearly seamless transitions across horizontal wrap-around, unlike the discontinuities or residual artifacts of the compared alternatives.The qualitative comparison includes the baseline, circular blending, and latent rotation; the displayed images are rolled by half their width to expose seam continuity.Source: §4.4 · p. 14, Figure 10 · p. 14

4.5 Applications

  • OmniX-generated 3D scenes are demonstrated in Blender through free exploration, PBR-based relighting, and physical simulation.The demonstrations move the camera, animate a point light around the scene, and introduce an elastic ball with initial horizontal velocity.Source: §4.5 · p. 14, Figure 11 · p. 15

5 Conclusion and Limitations

OmniX repurposes pretrained 2D flow-matching models for panorama generation, perception, and completion, with PanoX providing indoor/outdoor data and dense geometric and material annotations. The authors report effectiveness across these tasks and identify limitations in efficiency, distance prediction, metallic generalization, and PBR material estimation.

Conclusion

  • OmniX repurposes pretrained 2D flow-matching models for panorama generation, perception, and completion, and supports automatic creation of immersive, PBR-ready 3D scenes.The framework casts RGB-to-modality perception and masked-modality completion into a 2D generative paradigm, using cross-modal adapters.Source: §5 · p. 15
  • PanoX spans indoor and outdoor environments and multiple visual modalities, providing dense geometric and material annotations for panorama perception.The conclusion presents it as a benchmark addressing panorama-data scarcity.Source: §5 · p. 15

Limitations

  • The method inherits slow training and inference from pretrained 2D flow-matching models, and Euclidean-distance predictions remain insufficiently accurate for smooth reconstructed surfaces.The authors say bumpy surfaces affect subsequent PBR rendering.Source: §5 · p. 15
  • OmniX-Pano2Metallic generalizes poorly, partly because panoramic PBR material training data are scarce; the authors also question the benefit of 2D image priors for PBR material estimation.They attribute the latter concern to the significant differences between neural and PBR rendering.Source: §5 · p. 15

A. Additional Details of OmniX

The appendix describes three panorama-task settings for OmniX: generation or completion, RGB-conditioned perception, and guided perception. These settings use masked panorama, RGB, and optional target-modality inputs to generate complete panoramas or modalities.

Task settings

  • The framework is built around modality-specific LoRAs and circular synchronization operators for different panorama tasks.Only panorama generation uses text prompts; other tasks use an empty prompt.Source: A. Additional Details of OmniX · p. 20
  • For panorama generation and completion, a masked panorama conditions generation of the complete panorama.Image-to-panorama generation is included by projecting a single-view image onto an otherwise empty panorama.Source: A. Additional Details of OmniX · p. 20
  • For RGB-to-modality perception, an RGB reference conditions generation of targets such as distance, normal, albedo, roughness, and metallic.A camera ray map can optionally provide an additional condition.Source: A. Additional Details of OmniX · p. 20
  • Guided perception conditions generation of a complete target modality on both an RGB reference and a masked target.The text identifies this setting as useful for progressive completion when building 3D scenes.Source: A. Additional Details of OmniX · p. 20

B. Results on Panorama Completion

OmniX-Fill is evaluated on panorama completion using masked panoramas constructed with an occlusion-aware pipeline. The reported comparisons find stronger quantitative and qualitative completion results than Flux-Fill, including reconstruction of fine structures in masked regions.

Evaluation setup

  • Completion training and evaluation use masked panoramas, masks, and ground truths produced with the occlusion-aware data-construction pipeline.The data sources otherwise match those used for panorama generation and perception.Source: B. Results on Panorama Completion · p. 20
  • Completion is evaluated with FID, KID, CLIP-FID, and FAED, using perspective crops for the first three and ERP outputs for FAED.The stated metrics match those used for panorama generation.Source: B. Results on Panorama Completion · p. 21

Quantitative evaluation

  • OmniX-Fill consistently outperforms Flux-Fill across all reported panorama-completion metrics.The comparison concerns viewpoint changes and occlusion-induced masking; the supplied table caption identifies Flux.1-Fill as the baseline.Source: Table S1 · p. 20, B. Results on Panorama Completion · p. 21

Qualitative evaluation

  • OmniX-Fill produces visually plausible, structurally consistent completions and preserves details such as vegetation and terrain contours.The qualitative comparison says Flux-Fill struggles with fine structures and often produces artifacts and inconsistencies; additional demonstrations include guided perception and complex masking scenarios.Source: B. Results on Panorama Completion · p. 21, Figure S1 · p. 21

C. Additional Results on Panorama Perception

OmniX generalizes to in-the-wild panoramas from the Internet, with its modality-specific adapters leveraging pretrained 2D flow-matching priors for panorama perception.

C. Additional Results on Panorama Perception

D. Additional Results on Panorama Generation

OmniX generates diverse, high-quality panoramas from single-view images and completes masked panorama regions with locally coherent results; additional tests report generalization to unseen panoramas.

D. Additional Results on Panorama Generation

  • OmniX generates high-quality, diverse panoramas from single-view images, inferring missing spatial content across the full 360° field of view.The reported results describe visually coherent details even from limited single-view observations.Source: D. Additional Results on Panorama… · pp. 21, 25, Figure S4 · p. 24
  • Given masked inputs, OmniX-Fill generates accurate, locally coherent results in masked panorama areas.Source: Figure S2 · p. 22
  • Additional in-the-wild perception results report generalization to unseen panoramas; guided perception requires RGB references to align predicted modalities with context.Source: Figure S2 · p. 22, Figure S3 · p. 23

E. Additional Ablation Analysis and Discussion

The extended ablations examine camera-ray conditioning, PBR material input arrangements, and joint distance-normal modeling across in-domain and out-of-domain evaluation sets.

E. Additional Ablation Analysis and Discussion

Related papers

Loading 2510.26800v2…