PaperlayerView exact revision on arXiv

Source-linked AI summary

OneWorld: Taming Scene Generation with 3D Unified Representation Autoencoder

Sensen Gao, Zhaoqing Wang, Qihang Cao, Dongdong Yu, Changhu Wang, Tongliang Liu, Mingming Gong, Jiawang Bian

arXiv:2603.16099v1cs.CV

TL;DR

Existing 2D-latent approaches make cross-view appearance and geometric consistency difficult. OneWorld diffuses in a unified 3D representation space using 3D-URAE, cross-view correspondence preservation, and Manifold-Drift Forcing, achieving high-quality scenes with superior cross-view consistency to state-of-the-art 2D-based methods.

  • Problem

    Existing 3D scene generation methods primarily use 2D image or video latent spaces, which provide limited explicit 3D coupling across views and can yield inconsistent geometry.

  • Method

    OneWorld performs diffusion in a unified 3D space whose latents jointly encode geometry, appearance, and semantics, while correspondence regularization and Manifold-Drift Forcing stabilize cross-view generation.

  • Results

    OneWorld generates high-quality 3D scenes with superior cross-view consistency compared with state-of-the-art 2D-based methods.

  • Takeaways & Limitations

    The findings support generative modeling in 3D foundation-model representation space as a promising direction for scalable, unified 3D world generation.

  • Takeaways & Limitations

    Limited training-data scale and diversity and relatively low-resolution training and decoding may reduce generalization and fine-grained rendering quality.

Abstract

from arXiv · show

Existing diffusion-based 3D scene generation methods primarily operate in 2D image/video latent spaces, which makes maintaining cross-view appearance and geometric consistency inherently challenging. To bridge this gap, we present OneWorld, a framework that performs diffusion directly within a coherent 3D representation space. Central to our approach is the 3D Unified Representation Autoencoder (3D-URAE); it leverages pretrained 3D foundation models and augments their geometry-centric nature by injecting appearance and distilling semantics into a unified 3D latent space. Furthermore, we introduce token-level Cross-View-Correspondence (CVC) consistency loss to explicitly enforce structural alignment across views, and propose Manifold-Drift Forcing (MDF) to mitigate train-inference exposure bias and shape a robust 3D manifold by mixing drifted and original representations. Comprehensive experiments demonstrate that OneWorld generates high-quality 3D scenes with superior cross-view consistency compared to state-of-the-art 2D-based methods. Our code will be available at https://github.com/SensenGao/OneWorld.

1 Introduction

OneWorld addresses the geometric and appearance inconsistencies of 2D-latent 3D scene generation by diffusing directly in a coherent unified 3D representation space. Its 3D-URAE, CVC loss, and MDF jointly support unified scene encoding, cross-view structural consistency, and robust sampling.

  • Motivation: 2D-prior methods produce plausible scenes but suffer expensive per-scene optimization and inconsistent geometry because they lack strong explicit 3D modeling.They use score distillation or generate multi-view images/videos followed by 3D reconstruction.
  • Motivation: Gen3R limits 3D representation capacity through video-latent alignment, separately generates geometry and appearance, and models geometry mainly across small-baseline views.These constraints hinder unified 3D representations with coherent geometry and high-fidelity appearance.
  • OneWorld: OneWorld performs diffusion in a coherent 3D representation space using 3D-URAE to jointly encode geometry, appearance, and semantics.3D-URAE builds on pretrained 3D foundation models, injects appearance, and distills semantics into a unified 3D latent space.
  • Cross-view consistency: The token-level CVC consistency loss preserves conditioning-induced correspondence patterns in target-view tokens during denoising to improve cross-view structural consistency.It explicitly preserves correspondence rather than only minimizing mean error between 3D representations.
  • Sampling robustness: MDF trains the 3DGS decoder with mixed diffusion-sampled and original representations to counter train–inference sampling drift and shape a robust 3D manifold.This addresses accumulated inference errors and improves appearance consistency and cross-view coherence.

2 Related Work

Prior 3D scene generation methods either iteratively optimize 3D representations using pretrained 2D diffusion priors or reconstruct 3D scenes from synthesized multi-view media. OneWorld instead performs diffusion in a unified 3D representation space, addressing limitations associated with 2D latent generation and compressed representations.

  • Diffusion-based Iterative 3D Scene Generation: Iterative methods use Score Distillation Sampling to optimize 3DGS or NeRF by aligning rendered views with pretrained 2D diffusion distributions.These approaches leverage pretrained 2D generative models as strong priors.
  • Multi-View Reconstruction-Based 3D Scene Generation: Multi-view reconstruction methods synthesize images or videos first, then reconstruct 3D scenes through multi-view synthesis or incremental outpainting.They improve generation efficiency by avoiding per-scene optimization.
  • 3D Scene Generation in 2D Latent Spaces: 2D-latent approaches generate multi-view or video latents with frozen image or video VAEs and train a decoder, usually implemented as 3DGS, for 3D representations.This differs from multi-view reconstruction methods that generate multi-view images or videos directly.
  • OneWorld: OneWorld constructs a unified 3D space through appearance injection and semantic distillation, while cross-view correspondence preserves geometric token alignment during diffusion training.Manifold-drift forcing mixes ground-truth and sampled 3D features to make the 3D decoder more robust.
  • Representation Autoencoders: VAEs compress data into low-dimensional latent spaces for latent diffusion, but this compression inevitably causes information loss.Representation autoencoder studies instead freeze pretrained representation encoders and train decoders to reconstruct images from high-dimensional semantic features.

3 Method

OneWorld performs scene generation by diffusing directly in a unified, geometry-aware 3D representation space produced by 3D-URAE. It further preserves cross-view structure with correspondence-aware training and improves inference robustness by training the decoder on manifold-drifted latents.

  • 3.1 3D-URAE: 3D-URAE converts a feed-forward 3D foundation model into a unified latent representation encoding geometry, appearance, and semantics.It is trained with differentiable 3DGS rendering and semantic distillation losses.
  • 3.1 3D-URAE: The geometry encoder jointly processes patchified visual tokens and camera parameters, then decodes geometry tokens into 3D Gaussian Splatting representations and depth maps.Camera parameters enable camera-controllable generation and direct use of dataset-provided coordinate systems.
  • 3.1 3D-URAE: Appearance injection restores fine-grained image details discarded by semantic patchification, while semantic distillation organizes geometry-dominant tokens into a latent manifold easier for diffusion to model.The two branches respectively augment geometry conditioning with appearance-preserving tokens and transfer semantic structure into geometry tokens.
  • 3.2 Diffusion Model: A conditional diffusion model generates target-view token grids in the unified 3D space, using x0-prediction while optimizing an equivalent v-space objective.The target tokens are flattened before applying the forward perturbation process.
  • 3.2 Diffusion Model: The Cross-View Correspondence loss aligns nearest-neighbor matching patterns between target and conditioning tokens, retaining matches only when cosine-similarity confidence exceeds τ = 0.9.It complements token-space proximity by explicitly preserving cross-view structural correspondence.
  • 3.3 Manifold-Drift Forcing: Manifold-Drift Forcing mixes diffusion-predicted and ground-truth unified latents across sampling steps, then trains the 3D decoder with rendering supervision on available views.This addresses train–inference exposure bias and teaches the decoder to tolerate sampling-induced drift for more stable inference reconstruction.

4 Experiment

OneWorld is trained and evaluated as a unified 3D representation model on calibrated multi-view benchmarks and reference-free world-generation protocols. It achieves leading novel-view quality and consistency, while ablations show complementary benefits from CVC and MDF.

  • Implementation: Training uses approximately 70K scenes from RealEstate10K and DL3DV-10K, with 3D-URAE sampling eight input views and supervising four novel views.The datasets cover diverse real-world environments and use official or scene-level train/test splits.
  • Implementation: The diffusion model operates in 3D-URAE latent space, uses v-prediction with CVC consistency, and conditions on one-view latents plus camera parameters.Training applies τ=0.9, λcvc=0.2, classifier-free text dropping at rate 0.5, batch size 256, and 100K steps.
  • 1-view NVS on calibrated benchmarks: On calibrated one-view NVS, OneWorld achieves the best overall results on RealEstate10K, reaching 21.57 PSNR, 0.735 SSIM, and 0.231 LPIPS.It also attains I.Q. 0.604, I2V Subj. 0.993, and surpasses current SOTA baselines in producing coherent 3D scenes.
  • WorldScore-style reference-free evaluation: Under WorldScore-Indoor, OneWorld records the best 3D Consistency at 84.98, Photometric Consistency at 81.67, and Style Consistency at 76.74.Object Control, Content Alignment, and Subjective Quality remain competitive at 48.92, 46.88, and the reported final score, respectively.
  • Ablation: Removing CVC lowers PSNR from 21.57 to 19.10, SSIM from 0.735 to 0.682, and raises LPIPS from 0.231 to 0.284.Removing MDF causes milder degradation, lowering PSNR from 21.57 to 20.59 and SSIM from 0.735 to 0.714, while increasing LPIPS from 0.231 to 0.256.

5 Conclusion

OneWorld generates 3D scenes directly in a pretrained 3D foundation model’s representation space, unifying geometry, appearance, and semantics while enforcing cross-view structure and stabilizing sampling. The appendix provides complementary analyses of training settings, prediction spaces, and feed-forward 3D Gaussian Splatting reconstruction.

  • 5 Conclusion: OneWorld generates 3D scenes directly in a pretrained 3D foundation model’s representation space, addressing structural inconsistency and inefficiency in per-view 2D latent pipelines.Its unified 3D representation combines geometry, appearance, and semantics, while explicitly enforcing cross-view structure and stabilizing sampling.
  • 5 Conclusion: The appendix analyzes key hyperparameters in 3D-URAE reconstruction and unified-space diffusion, compares x0-prediction with v-prediction, and evaluates feed-forward 3D Gaussian Splatting reconstruction.The reconstruction comparison uses the same 8-view setting, while prediction-space analysis examines convergence in the high-dimensional 3D-URAE token space.

A Training hyperparameter settings

The paper reports training hyperparameter settings for 3D-URAE reconstruction and unified-space diffusion, keeping the core configuration constant while varying ablation-specific hyperparameters. Performance is evaluated using reconstruction and generation metrics.

  • Training configuration: Training settings cover both 3D-URAE reconstruction and unified-space diffusion.The reported settings appear in Tab. 5 and Tab. 6.
  • Training configuration: The core configuration remains constant across experiments, including optimizer, learning rate, batch size, token sizes, and rendering-loss weights.Only ablation-specific hyperparameters are explored unless otherwise specified.
  • Evaluation: Changes are evaluated through 3D-URAE reconstruction and OneWorld generation performance using PSNR, SSIM, LPIPS, and semantic similarity.These metrics assess the resulting reconstruction and generation performance.

A.1 3D-URAE reconstruction hyperparameters

The 3D-URAE reconstruction study varies the shared semantic margin m and semantic weight λsem, selecting m = 0.05 and λsem = 0.10 as defaults for strong reconstruction and semantic similarity. At λsem = 0.10, m = 0.05 improves reconstruction over m = 0.00, whereas m = 0.10 sharply reduces semantic similarity and mildly degrades reconstruction.

  • Hyperparameter setup: 3D-URAE uses rendering and semantic distillation losses, with shared margins m1 = m2 = m and semantic weight λsem as the varied hyperparameters.The semantic similarity measure is feature similarity to distilled DINOv2 targets.
  • Hyperparameter setup: The default m = 0.05 and λsem = 0.10 provide strong reconstruction and high semantic similarity.These settings are selected based on reconstruction quality and similarity to distilled DINOv2 targets.
  • Margin ablation: With λsem = 0.10, increasing m from 0.00 to 0.05 raises PSNR from 26.67 to 28.19 and SSIM from 0.918 to 0.932, while lowering LPIPS from 0.122 to 0.102.Semantic similarity changes only slightly from 0.986 to 0.984.
  • Margin ablation: At m = 0.10, semantic similarity drops from 0.984 to 0.917, while SSIM decreases from 0.932 to 0.930.The passage also reports that reconstruction quality degrades mildly at this larger margin.

A.2 Diffusion and generation hyperparameters

The diffusion model combines velocity and cross-view correspondence losses, with τ and λcvc controlling CVC matching and weighting. In 10K-step ablations, τ = 0.90 is the default and improves reported reconstruction and perceptual metrics over τ = 0.80.

  • Diffusion objective and defaults: The conditional diffusion objective is Ldiff = Lv + λcvcLcvc, with τ = 0.90 and λcvc = 0.20 used as defaults.All hyperparameter-exploration experiments are trained for 10K steps.
  • CVC threshold: In RealEstate10K 1-view NVS, τ = 0.90 improves PSNR to 19.10, SSIM to 0.682, and LPIPS to 0.284 over τ = 0.80.The τ = 0.80 setting reports PSNR 17.62, SSIM 0.645, and LPIPS 0.316, with λcvc fixed at 0.2.
  • CVC threshold: At τ = 0.90, the model achieves its best reported perceptual scores: I2V Subj. 0.989, I2V BG 0.990, and I.Q. 0.566.These scores are reported alongside the τ ablation results with λcvc fixed at 0.20.

B Prediction Space Comparison

OneWorld adopts x0-prediction because clean 3D-URAE tokens lie near a low-dimensional manifold, whereas noise and velocity targets span the high-dimensional ambient space. Consistent with this rationale, x0-prediction fits scenes faster and with higher fidelity than v-prediction under the same optimization budget.

  • Prediction target motivation: 3D-URAE tokens occupy a high-dimensional ambient feature space, while clean data x0 concentrates near a low-dimensional manifold capturing underlying structure.The manifold assumption motivates predicting clean data rather than heavily compressed latent quantities.
  • Prediction target motivation: Noise ϵ and velocity v spread across the full ambient space, making them substantially harder to predict and potentially catastrophically difficult as dimensionality grows.Velocity is a linear combination involving ϵ and does not lie on the clean-data manifold.
  • Empirical comparison: Compared to v-prediction, x0-prediction converges faster and reaches higher-fidelity structure and appearance within 2000 overfitting iterations.Figure 8 visualizes fitting results at different training steps for a single training scene.
  • Prediction target motivation: The optimal prediction target shifts toward x0 rather than v when prediction targets are reconsidered through dimensionality.This theoretical perspective complements the manifold-based explanation.
  • Empirical comparison: x0-prediction fits a single scene faster and with fewer residual artifacts than v-prediction under identical training steps.The comparison uses single-scene overfitting with the same optimization budget.

C Feed-forward 3D Gaussian Splatting Reconstruction

The section evaluates feed-forward 3D Gaussian Splatting reconstruction on RealEstate10K and DL3DV under an 8-view setting, comparing 3D-URAE with representative prior methods. 3D-URAE consistently improves novel-view rendering quality, achieving the strongest reported PSNR and SSIM while substantially reducing LPIPS on RealEstate10K.

  • Evaluation setup: The evaluation uses RealEstate10K and DL3DV with representative feed-forward 3DGS baselines, including PixelSplat, MVSplat, DepthSplat, and AnySplat.DepthSplat is identified as the strongest prior method in the comparison table.
  • RealEstate10K: On RealEstate10K, 3D-URAE reaches PSNR 28.19 and SSIM 0.932, surpassing DepthSplat’s PSNR 27.02 and SSIM 0.879.These results use the same 8-view reconstruction setting.
  • RealEstate10K: On RealEstate10K, 3D-URAE reduces LPIPS to 0.102, compared with baseline values ranging from 0.169 to 0.177.Lower LPIPS indicates the reported improvement in perceptual similarity.
  • DL3DV: On DL3DV, 3D-URAE achieves PSNR 24.68, SSIM 0.816, and LPIPS 0.140.The passage reports PSNR and SSIM as the best values on DL3DV.

D Proof Sketch for Manifold-Drift Forcing · E More Visualization · F Discussion

The proof sketch explains how train–inference distribution mismatch causes off-manifold drift, how coupled multi-view denoising amplifies it, and why manifold-drift forcing improves decoder robustness. Additional visualizations show OneWorld’s 3DGS structures alongside novel-view RGB renderings and depth maps across diverse scenes and viewpoints.

  • D Proof Sketch for Manifold-Drift Forcing: Training and inference expose the denoiser to different input distributions, causing accumulated rollout error and increasing the generated latent’s distance from the valid 3D-consistent manifold.Training uses forward-noised X0 ∈ M, whereas inference uses model-produced rollout states.
  • D Proof Sketch for Manifold-Drift Forcing: The proof bounds rollout error recursively using a Lipschitz one-step map and an oracle chain whose clean output remains on the valid manifold.The recursion is unrolled to bound δ0 and define the manifold distance d(˜X0, M).
  • D Proof Sketch for Manifold-Drift Forcing: Cross-view coupling amplifies drift because perturbations propagate through coupled denoising dynamics and the shared 3D decoder projects artifacts into multiple views.With cross-view coupling, the Lipschitz constant increases relative to independent generation, where ρt = 0 and Lt ≤ κt.
  • D Proof Sketch for Manifold-Drift Forcing: Manifold-drift forcing trains the decoder on interpolated latents between drifted and original representations, addressing off-manifold inference states.The method mixes drifted and original latents using α sampled uniformly from [0, 1] over diffusion times t sampled uniformly from [T1, T2].
  • D Proof Sketch for Manifold-Drift Forcing: This training increases decoder robustness to diffusion-produced latents and encourages the decoder to project them back toward valid 3D geometry.The decoder is trained mainly on manifold latents but must handle off-manifold latents produced during inference.
  • E More Visualization: OneWorld’s additional visualizations present 3D Gaussian Splatting structures with corresponding novel-view RGB renderings and depth maps.Figures 9 and 10 show structure, RGB, and depth visualizations, while Figure 11 adds further rendered examples.

F.1 Limitation

OneWorld is limited by training data scale and diversity, as well as relatively low-resolution training and decoding. These constraints may reduce robustness on rare or extreme scenes and limit fine-grained rendering quality.

  • F.1 Limitation: Limited training-data scale and diversity may reduce robustness on rare scene types, extreme viewpoints, and uncommon appearance distributions.The planned remedy is scaling training to larger and more diverse multi-view corpora.
  • F.1 Limitation: Relatively low-resolution training and decoding may limit fine-grained texture fidelity and thin-structure rendering quality.Future work plans to adopt higher-resolution training and decoding.
  • F.1 Limitation: Future work targets larger, more diverse multi-view corpora and higher-resolution training and decoding to address these limitations.These changes are intended to improve generalization and rendering quality.
Loading 2603.16099v1…