PaperlayerView exact revision on arXiv

Source-linked AI summary

PixWorld: Unifying 3D Scene Generation and Reconstruction in Pixel Space

Sensen Gao, Zhaoqing Wang, Qihang Cao, Dongdong Yu, Changhu Wang, Jia-Wang Bian

arXiv:2607.05373v1cs.CV

TL;DR

3D scene reconstruction and generation have largely followed separate paradigms, while latent-space unification introduces encoding-related limitations. PixWorld unifies both tasks through pixel-space diffusion and geometry-aware supervision, outperforming prior latent-space generation methods and matching state-of-the-art reconstruction methods.

  • Problem

    3D scene reconstruction and generation remain separate research lines, while latent-space unification operates on encoded features and requires pretrained autoencoders.

  • Method

    PixWorld uses end-to-end pixel-space diffusion with differentiable rendering and geometry perception loss to jointly generate and reconstruct 3D Gaussian scenes.

  • Results

    PixWorld achieves superior performance on 3D scene generation and reconstruction, matching state-of-the-art reconstruction methods and outperforming prior latent-space generation methods.

  • Takeaways & Limitations

    The results support pixel-space diffusion as a unified paradigm for 3D scene modeling across generation and reconstruction.

  • Takeaways & Limitations

    Evaluation focuses on scene-level datasets, leaving generalization to more diverse outdoor and object-centric scenes insufficiently characterized.

Abstract

from arXiv · show

3D reconstruction and generation are commonly tackled by separate paradigms: pixel-based regression for reconstruction, and latent diffusion for generation. Recent works attempt to unify them in latent space, but with notable drawbacks: the diffusion objective is defined on latent features rather than the underlying 3D representation, and both branches suffer from information loss introduced by latent encoding, while requiring a pretrained Variational Autoencoder (VAE) or Representation Autoencoder (RAE). In this paper, we reformulate these two tasks under a unified pixel-space diffusion paradigm and introduce PixWorld, a single model that jointly addresses 3D reconstruction and generation. By supervising diffusion directly on rendered images, PixWorld removes the above limitations and aligns optimization with 3D scene fidelity. Beyond photometric and perceptual supervision that operates at the 2D image level and lacks 3D geometric awareness, we further introduce a geometry perception loss that aligns rendered views with their ground truth in the geometry-aware feature space of a pretrained 3D foundation model, providing 3D structural supervision. PixWorld consistently outperforms prior latent-space generation methods and matches state-of-the-art reconstruction methods, demonstrating the superiority of a unified pixel-space approach.

1 INTRODUCTION

PixWorld unifies 3D scene generation and reconstruction in a single pixel-space diffusion framework, supervising a pixel-aligned 3D Gaussian representation through rendered views without a VAE or RAE. It also adds geometry-aware supervision from a pretrained 3D foundation model to improve structural fidelity beyond image-level losses.

  • Motivation: Reconstruction and generation have traditionally followed separate paradigms: direct 3D regression from multi-view images versus latent-space scene synthesis.Reconstruction recovers scenes from real captures, while generation synthesizes plausible scenes from limited or imagined conditions.
  • PixWorld: PixWorld jointly handles 3D reconstruction and generation with pixel-space diffusion in a single framework.The framework supervises a pixel-aligned 3D Gaussian representation directly through multi-view differentiable rendering.
  • PixWorld: Removing the latent stage avoids information loss from latent encoding and the additional training cost of a VAE or RAE.This directly aligns the diffusion signal with 3D scene fidelity, which is especially important for reconstruction.
  • Geometry supervision: PixWorld introduces a geometry perception loss that aligns rendered views with ground truth in the geometry-aware feature space of a pretrained 3D foundation model.The loss provides 3D structural supervision beyond 2D photometric and perceptual losses by encouraging shared underlying geometry.
  • Results: A single PixWorld model achieves superior performance on both 3D scene generation and reconstruction, supporting pixel-space diffusion as a unified paradigm.The experiments establish the framework for unified 3D scene modeling.

2 RELATED WORK

Prior 3D scene generation spans iterative optimization and multi-view reconstruction pipelines, while newer methods diffuse in compressed latent spaces. PixWorld instead extends pixel-space diffusion to 3D, removing intermediate VAE/RAE representations and directly supervising predicted 3D scenes through rendered views.

  • Diffusion-based Iterative 3D Scene Generation: Iterative 3D generation methods use pretrained 2D generative models as priors and optimize 3DGS or NeRF representations with Score Distillation Sampling.These methods align rendered views with a pretrained diffusion model’s prior.
  • Multi-View Reconstruction-Based 3D Scene Generation: Multi-view generation pipelines first synthesize images or videos with pretrained 2D diffusion models, then reconstruct 3D scenes or incrementally outpaint them.This separates multi-view visual synthesis from subsequent 3D reconstruction or scene expansion.
  • 3D Scene Generation in Latent Spaces: Recent 3D generation methods move diffusion into compressed latent spaces, typically using pretrained image or video VAEs and a latent-to-3D decoder targeting 3DGS.The diffusion backbone is fine-tuned in the latent space produced by the frozen VAE.
  • 3D Scene Reconstruction: 3D reconstruction recovers representations from captured views, progressing from classical multi-view geometry and per-scene optimization toward feed-forward mappings from multi-view inputs.NeRF and 3D Gaussian Splatting popularized per-scene optimization for high-fidelity novel-view synthesis.
  • Pixel-Space Diffusion: PixWorld performs diffusion directly in pixel space, removes the intermediate VAE/RAE used by latent-space 3D methods, and supervises 3D representations through differentiable rendering of predicted 3DGS.This approach connects the generative objective directly to the underlying 3D representation and rendered outputs.

3 METHODOLOGY

PixWorld unifies 3D reconstruction and generation through pixel-space diffusion over posed multi-view images, predicting a pixel-aligned 3D Gaussian scene that is rendered for supervision. Its objective combines flow-matching rendering and depth losses with geometry-aware feature supervision from a frozen 3D foundation model.

  • Framework: PixWorld jointly reconstructs and generates 3D scenes by processing clean and noisy posed views with a two-stream diffusion transformer and decoding them into pixel-aligned 3D Gaussians.The predicted Gaussian representation is rendered back into images for training supervision.
  • Pixel-space diffusion: Pixel-space diffusion keeps the diffusion variable and rendered output in RGB space, enabling direct supervision on rendered images rather than encoded latent features.This avoids decoupling the diffusion objective from the rendered output and underlying 3D representation.
  • Task formulation: The view partition guarantees at least one clean observation; with no noisy views, PixWorld performs multi-view reconstruction, otherwise it predicts noisy views conditioned on clean views.An optional text prompt can further specify the scene.
  • 3D Gaussian decoding and rendering: The model predicts per-view depth and Gaussian attributes, unprojects pixels using depth and camera parameters, aggregates them into a scene representation, and differentiably renders all views.Clean views use direct MSE supervision, while noisy views use flow-matching supervision; novel views are additionally supervised with MSE and LPIPS.
  • Geometry perception loss: PixWorld adds a geometry perception loss that compares rendered and reference multi-view features from a frozen 3D foundation model, complementing photometric supervision with cross-view structural constraints.The full objective is L = Lrender + λdepth Ldepth + λgeo Lgeo.

4 EXPERIMENTS

PixWorld is evaluated across reconstruction and generation protocols designed to test wide-baseline reasoning, novel-view fidelity, perceptual quality, and camera control. Ablations show that geometry perception loss is crucial for 3D-aware reconstruction.

  • Evaluation Protocols: PixWorld is evaluated on RealEstate10K and DL3DV-10K using 200 wide-camera-range test scenes per dataset, plus the official 2,000-scene WorldScore split.All scene-generation baselines are conditioned on camera poses.
  • Evaluation Metrics: Generation evaluation covers novel-view synthesis fidelity, VBench generation quality, and camera-control precision measured by π3-estimated AUC at 30°, 15°, and 5°.NVS uses PSNR, SSIM, and LPIPS; VBench includes subject, background, image-quality, and aesthetic-quality scores.
  • Baseline Comparisons: Gen3R is omitted from NVS comparisons because it supports only point-cloud reconstruction, while YoNoSplat is represented by its stronger pose-conditioned results.The reported NVS experiments use four-view and eight-view inputs.
  • Baseline Comparisons: PixWorld is compared with LVSM, GF, Gen3C, FlashWorld, and Gen3R under single- and two-image conditioning on RealEstate10K and DL3DV-10K.Reported metrics include NVS quality, VBench-style generation quality, and pose accuracy via π3-estimated AUC.
  • Ablation Study: Removing geometry perception loss reduces PSNR by 1.13 dB (19.12 →17.99), SSIM by 0.105 (0.717→0.612), and AUC@5 by 0.080 (0.642→0.562).The controlled RealEstate10K one-view ablation uses a 10K-sequence subset and identical 30K-step training; VBench-style scores barely change.
  • Ablation Study: The geometry perception loss supplies 3D structural supervision unavailable from purely 2D objectives, validating it as a key PixWorld component.Its removal consistently degrades the three reconstruction-oriented metric groups, while VBench-style scores barely shift.

5 CONCLUSION · A OVERVIEW OF THE APPENDIX

PixWorld unifies 3D scene generation and reconstruction through end-to-end pixel-space diffusion in one model, while the appendix documents its architecture, training procedures, and disaggregated evaluation.

  • 5 CONCLUSION: PixWorld is an end-to-end pixel-space diffusion framework that jointly handles 3D scene generation and reconstruction in a single model.It partitions multi-view inputs into clean and noisy subsets.
  • 5 CONCLUSION: The model produces a pixel-aligned 3D Gaussian representation in one forward pass.This unifies the two tasks within a shared inference process.
  • 5 CONCLUSION: Eliminating the intermediate VAE/RAE stage avoids latent-autoencoder information loss and extra training cost.The framework therefore does not require an intermediate latent encoding stage.
  • 5 CONCLUSION: The diffusion objective directly supervises the 3D representation through differentiable rendering.This connects pixel-space supervision to the represented 3D scene.
  • A OVERVIEW OF THE APPENDIX: Appendix B details the two-stream MMDiT denoiser architecture and its per-component parameter budget.It also describes the data, batching, and optimization recipes used to train PixWorld from scratch.
  • A OVERVIEW OF THE APPENDIX: Appendix C reports disaggregated quantitative results for 1-view and 2-view generation.The results separate configurations averaged in the main paper and distinguish behavior in harder versus easier settings.

B IMPLEMENTATION DETAILS

PixWorld uses a 24-layer, dual-stream DiT denoiser and trains on posed multi-view scenes supplemented by 10M single images. Optimization combines rendering, depth, LPIPS, and geometry losses with timestep-based gating and a frozen π3 geometry critic.

  • Architecture: PixWorld’s denoiser is a 24-layer DiT with hidden width d=1024, 16 attention heads, SwiGLU feed-forward layers, RMSNorm, and adaLN-Zero timestep conditioning.Following SD3-style MMDiT, each block maintains clean and noisy parallel streams with independent normalization, projection, MLP, and modulation weights.
  • Data and batching: RealEstate10K and DL3DV-10K provide ∼67K posed multi-view scenes, while a 10M-image BLIP-3o branch strengthens the 2D appearance prior.Each scene contributes N ∈{4, . . . , 8} views partitioned into conditioned and novel subsets, with sampling biased toward small conditioned sets.
  • Optimization: PixWorld optimizes rendering, depth, and geometry objectives with λdepth=1.0 and λlpips=λgeo=0.1, gating perceptual and geometric losses at t>tth=0.3.The frozen geometry critic is π3 with gradients stopped on the reference branch, and training starts from scratch using AdamW.

C DETAILED RESULTS ON 1-VIEW AND 2-VIEW GENERATION

This section disaggregates generation results by input configuration for the 1-view and 2-view settings. The configurations distinguish long-horizon forward extrapolation, local bidirectional consistency, interpolation, and stronger-parallax extrapolation.

  • 1-view generation: The 1-view results average two complementary configurations: First Frame and Bidirectional.First Frame generates a purely forward trajectory over a long, fully extrapolative horizon, whereas Bidirectional generates two shorter, roughly symmetric horizons around a randomly selected middle frame.
  • 1-view generation: First Frame tests long-horizon forward extrapolation from the clip’s first frame.The model generates a purely forward trajectory along a long, fully extrapolative horizon.
  • 1-view generation: Bidirectional tests local consistency by generating two shorter, roughly symmetric horizons around a randomly chosen middle frame.The input view is a randomly chosen middle frame, and generation proceeds toward both ends of the clip.
  • 2-view generation: The 2-view results average Interpolation and Extrapolation configurations.Interpolation places two anchors around the target trajectory, while Extrapolation places both anchors at one end and requires generation beyond their span under stronger parallax.

D ADDITIONAL VISUALIZATIONS

PixWorld’s additional visualizations show geometrically coherent reconstruction and generation across varied view selections, input counts, and camera trajectories. Qualitative comparisons further indicate faithful geometry under single-image extrapolation and benefits from the Geometry Perception loss.

  • Qualitative Results: Fig. 6 visualizes input and generated views, camera trajectories, RGB renderings, and predicted depth maps across varied view selections.The examples cover varied input counts and trajectory shapes.
  • Qualitative Results: PixWorld produces geometrically coherent scenes with faithful geometry and rendered appearance, even under heavy extrapolation from a single conditioning image.The joint depth and 3D-Gaussian prediction captures scene geometry faithfully.
  • Ablation Visualization: The full model with Geometry Perception loss yields sharper later-view renderings and more accurate pose control than the version without it.Fig. 5 qualitatively compares PixWorld with and without the Geometry Perception loss.

E INFERENCE SPEED COMPARISON · F RESPONSIBLE CONSIDERATIONS

PixWorld generates a scene in 15 seconds on a single NVIDIA A100-SXM4-80G GPU, approaching FlashWorld’s 10-second runtime. However, the comparison is not strictly apples-to-apples because PixWorld uses a lower output resolution than video-diffusion baselines.

  • E INFERENCE SPEED COMPARISON: 15 seconds per scene places PixWorld in the same order of magnitude as FlashWorld at 10 s.Inference speed is measured as wall-clock time per scene on a single NVIDIA A100-SXM4-80G GPU.
  • E INFERENCE SPEED COMPARISON: PixWorld’s speed comparison is not strictly apples-to-apples because it runs at a lower output resolution than video-diffusion baselines.The lower resolution reduces per-step compute.
  • E INFERENCE SPEED COMPARISON: The benchmark reports wall-clock time per scene, key frames per scene, and number of function evaluations (NFE).These measurements are summarized in Table 9.

F.1 LIMITATIONS.

PixWorld’s evaluation and training setup leave open questions about generalization and efficiency. The study focuses on scene-level datasets, while finite resolution and compute budgets constrain pixel-space diffusion with differentiable rendering.

  • Evaluation scope: Evaluation is limited to widely used scene-level datasets, leaving generalization to more diverse outdoor and object-centric scenes insufficiently characterized.The paper specifically identifies RealEstate10K and DL3DV-10K as its experimental datasets.
  • Training constraints: Finite resolution and compute budgets constrain training for pixel-space diffusion with differentiable rendering.The passage identifies these constraints as an open direction for improvement.

F.2 BROADER IMPACTS. · F.3 LLM USAGE.

PixWorld offers potential benefits across 3D reconstruction, content creation, robotics, simulation, and VR/AR, while raising privacy, misuse, and safety concerns that require responsible deployment. The work did not use LLMs as an important or non-standard component of its core method.

  • F.2 BROADER IMPACTS.: PixWorld may benefit efficient 3D reconstruction, 3D content creation, robotic perception, simulation, and VR/AR applications.These applications are identified as potential beneficiaries of the system.
  • F.2 BROADER IMPACTS.: The Geometry Perception loss improves camera pose control and substantially mitigates blurriness compared with the w/o Geom. variant.The ablation generates 7 subsequent frames from one input image and quantitatively evaluates pose accuracy against ground-truth poses.
  • F.2 BROADER IMPACTS.: Improved 3D scene reconstruction and generation may raise concerns about privacy-sensitive scene capture and misuse of synthetic or reconstructed 3D content.The paper specifically identifies these risks as broader-impact concerns.
  • F.2 BROADER IMPACTS.: Deployment in safety-critical settings may be unreliable, motivating responsible data usage, careful evaluation, and human oversight.The paper recommends these safeguards for real-world applications.
  • F.3 LLM USAGE.: The work does not use LLMs as an important, original, or non-standard component of the core method.Any LLM-based tools were limited to writing, editing, or formatting assistance and did not affect the methodology, experiments, or scientific conclusions.
  • F.3 LLM USAGE.: Additional visualizations show reconstruction and generation under varying view selections, including camera trajectories, input and generated views, and predicted depth maps.The visualizations are presented in Figure 6.
  • F.3 LLM USAGE.: Generated-scene visualizations include RGB renderings and predicted depth maps, with the first view serving as the input.These examples are presented in Figure 7.
Loading 2607.05373v1…