PaperlayerView exact revision on arXiv

Source-linked AI summary

SceneAE: Scene Generation and Reconstruction in Queryable 3D-Anchored Latent Space

Shuofeng Sun, Wenzhao Zheng, Yunpeng Zhang, Yi Wu, Linqing Zhao, Haibin Yan

arXiv:2610.19708v1cs.CVcs.AI

TL;DR

Viewpoint-aligned world-model latents describe particular observations, limiting direct novel-view querying, while camera conditioning on appearance latents can make camera control difficult. SceneAE uses spatially grounded 3D latents for reconstruction and geometry-first content generation. It reports strong generation quality, geometric consistency, and camera controllability at about half the computational cost of dense latents.

  • Problem

    Viewpoint-aligned latents describe individual observations and limit direct novel-view querying, while injecting camera conditions into appearance latents can entangle camera and appearance.

    Source: §1 · p. 2

  • Method

    SceneAE anchors content features at explicit 3D positions for query-view decoding and generates camera-conditioned geometry before synthesizing corresponding content.

    Source: §1 · p. 2

  • Results

    SceneAE achieves strong generation quality, superior geometric consistency and camera controllability, at about half the computational cost of dense latents.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The results support using a 3D scene representation as the latent space for both reconstruction and generation.

    Source: §5 · pp. 9–10

    The representation assumes persistent 3D locations, which is effective for static scenes but restrictive for moving or deformable objects.

    Source: §C.2 · p. 19

Abstract

from arXiv · show

Recent video world models have increasingly adopted 3D memory mechanisms to integrate historical observations into a lightweight and persistent scene representation. This naturally raises a complementary question: if 3D representations can serve as an effective mechanism for memorizing observed scenes, can they also be directly used as the latent space for reconstructing and generating scenes? Motivated by this idea, we represent a scene as a compact and unstructured 3D-anchored latent space (\textbf{SceneAE}). Each latent combines an explicit 3D position with an implicit content feature and is spatially grounded to describe its local scene region, forming a sparse latent point cloud. Unlike conventional video-VAE latents that describe how a scene appears from particular viewpoints, our representation directly describes the scene itself, making it naturally applicable to both reconstruction and generation. For reconstruction, once the scene has been generated, our persistent 3D representation can be projected and decoded into RGB-D observations by different target cameras, without the need to regenerate a video for each new camera trajectory. For generation, we leverage the geometry-content separation to first generate camera-controlled 3D geometry and then synthesize content conditioned on it, reducing camera-appearance entanglement and improving camera controllability. Experiments show that this 3D latent achieves superior generation quality, computational cost, and camera controllability.

1 INTRODUCTION

Camera-controllable world models commonly use viewpoint-aligned latents, which limit novel-view querying and can entangle camera conditions with appearance. SceneAE instead grounds latents in 3D, supporting scene reconstruction and geometry-first generation; experiments report stronger reconstruction and generation quality, lower computational cost, and more faithful camera following.

1 INTRODUCTION

  • Existing viewpoint-aligned latent grids describe individual observations, producing trajectory-specific videos that cannot be directly queried from novel viewpoints.Overlapping views also introduce redundancy, while appearance-centric representations make geometric consistency harder to learn.Source: §1 · p. 2
  • SceneAE anchors latent features at explicit 3D positions to represent local scene regions in a shared coordinate system, reusing each physical region across viewpoints.The projected latents can be decoded into per-pixel RGB-D for target cameras within the field of view.Source: §1 · p. 2
  • Camera conditioning on view-aligned appearance latents requires the model to infer geometry and appearance together, making camera control difficult.SceneAE instead generates future 3D anchors from the target camera trajectory before conditioning content generation on those positions.Source: §1 · p. 2
  • SceneAE achieves superior reconstruction and generation quality with a more compact latent space and lower computational cost, while reconstructing coherent RGB-D across query viewpoints and following target trajectories more faithfully.The comparison is reported across multiple real-world datasets.Source: §1 · p. 2

2 RELATED WORK

Prior work spans dense appearance latents, geometry-aware generation, 3D reconstruction, and compact unstructured tokens. These approaches provide useful capabilities but retain limitations in spatial grounding, viewpoint redundancy, computational cost, or generative use.

Scene Latent Space

  • Conventional image and video VAEs encode appearance in dense grids, while GLD generates dense geometry features at the cost of a high-dimensional latent space.Both remain viewpoint-aligned and redundantly encode overlapping regions.Source: §2 · p. 2
  • SceneTok supports query-view decoding with compact unstructured tokens, but lacks spatial grounding and relies on costly token-pixel interactions to learn region correspondence.The passage identifies increased optimization difficulty and cost.Source: §2 · p. 3

Camera-Controllable Scene Generation

  • Camera-controllable generation methods incorporate camera poses, ray embeddings, or geometric features; other approaches model image, depth, and pose latents, projective relations, or geometry features.GEN3C and Mirage instead maintain persistent 3D memories for conditioning or querying historical content.Source: §2 · p. 3

3D Scene Reconstruction

  • NeRF and 3D Gaussian Splatting support novel-view rendering but typically require per-scene optimization and are not designed as generative latent spaces.Feed-forward alternatives predict pixel-aligned 3D Gaussians or construct implicit scene representations for arbitrary-view reconstruction.Source: §2 · p. 3

3 METHOD

SceneAE encodes a scene as sparse 3D-anchored latents whose positions identify local regions and whose features describe their content. It decodes those latents into RGB-D views and generates future scenes by first predicting geometry from camera conditions, then content conditioned on that geometry.

3.1 3D LATENT REPRESENTATION

  • Each latent pairs an explicit 3D position with an implicit feature describing its local region, forming a compact, view-independent representation.The scene is encoded from back-projected points, with farthest-point sampling selecting anchor positions.Source: §3.1 · p. 3
  • Projected DA3 features from visible input views initialize each latent query, which deformable cross-attention and self-attention refine.Aggregation selects views by visibility and occlusion.Source: §3.1 · p. 3
  • For decoding, latents are splatted onto a downsampled target-view grid, then a deep decoder refines and upsamples features into RGB-D observations.Splatting initializes low-resolution features and establishes 3D-to-2D correspondences rather than rendering at full resolution; RGB and depth decoders attend to nearby latents.Source: §3.1 · pp. 3–4

3.2 GEOMETRY-CONTENT DECOUPLED DIT

  • A lightweight Point DiT predicts future 3D positions from the reference point cloud and target camera poses, using camera ray tokens to constrain the generated layout.The model trains along a flow path from the first-frame point cloud to the target point cloud.Source: §3.2 · p. 5
  • A second-stage Content DiT generates features on the predicted geometry, and the two stages are trained sequentially because content generation with 3D RoPE requires relatively accurate positions.The Content DiT uses 3D RoPE for geometric association and camera control.Source: §3.2 · pp. 5–6

4 EXPERIMENTS

Experiments evaluate scene generation, reconstruction, and latent-encoder/DiT design. SceneAE reports strong generation and reconstruction results alongside lower computational cost and improved camera control.

4.1 TRAINING DETAILS

  • Training uses DL3DV, RE10K, and ScanNet, with generation evaluated in one-view and two-view settings.Each sample uses nine views; training runs for 100k iterations on eight A800 GPUs.Source: §4.1 · p. 6, §4.2 · p. 6

4.2 3D SCENE GENERATION

  • SceneAE achieves the best generation quality while using half the GFLOPs and inference time of GLD.The authors attribute quality to the splat-based decoder and geometry-content-decoupled DiT, and lower cost to latent sparsity.Source: §4.2 · p. 7
  • For depth generation, SceneAE substantially outperforms existing approaches on DL3DV Eval and RealEstate10K test.Each method is evaluated against depth from its own geometry model: VGGT for Gen3R and DA3 for SceneAE and GLD.Source: §4.2 · p. 7
  • SceneAE shows stronger trajectory following and round-trip consistency comparable to explicit 3D memory.Trajectory following uses AUC@30 and ATE from VGGT-estimated poses; the round-trip evaluation compares corresponding forward and return RGB and depth views.Source: §4.2 · pp. 7–8, Table 3 · p. 7, Table 4 · p. 7

4.3 3D SCENE RECONSTRUCTION

  • Input reconstruction improves over SceneTok while reducing GFLOPs from 27,096 to 13,171 and batch latency from 4,020 ms to 1,623 ms; SceneAE also outperforms SceneTok and LVSM on novel-view synthesis.Novel-view synthesis uses nine context frames to render 20 other viewpoints and is evaluated on DL3DV and RE10K; Wan-VAE and GLD do not support that setting.Source: §4.3 · p. 8, Table 5 · p. 8
  • On DL3DV depth reconstruction, SceneAE reduces Chamfer Distance to 0.20 from 0.63 for Gen3R and 0.79 for Aether, and surpasses Gen3R and DA3-Base in cross-view depth consistency.Figure 6 shows recovered object structures, while DA3 and Gen3R may bend or distort objects.Source: §4.3 · p. 8, Table 6 · p. 9, Table 7 · p. 9

4.4 ABLATION STUDY

  • The encoder-decoder ablation describes low-resolution splat initialization followed by feature refinement, and restricts each pixel to attention over nearby splatted latents.The authors report that original-resolution rendering performs worse because the latents are sparser than pixel-aligned Gaussians; Jitter perturbs latent coordinates during decoding.Source: §4.4 · p. 9, Table 8 · p. 9
  • Among the tested DiT designs, decoupled geometry-then-content generation performs best, while applying camera conditioning only to geometry outperforms conditioning both geometry and content.The authors state that explicit geometric priors facilitate content synthesis and that camera-appearance entanglement harms generation quality.Source: §4.4 · p. 9, Table 9 · p. 9

5 CONCLUSION

SceneAE is a sparse, scene-centric 3D latent representation for reconstruction and generation. Its 3D anchors support RGB-D decoding from arbitrary cameras, while a geometry-content-decoupled DiT generates camera-conditioned geometry before content.

5 CONCLUSION

  • Each latent pairs an explicit 3D anchor with an implicit content feature, encoding each physical region once for sharing across viewpoints.A splat-based decoder renders the representation into RGB-D observations from arbitrary cameras without regenerating a video for each trajectory.Source: §5 · pp. 9–10
  • The paper presents SceneAE as one 3D latent representation for both scene reconstruction and generation.The authors hope the work encourages use of 3D representations as latent spaces for video world models.Source: §5 · pp. 9–10
  • The Geometry-Content Decoupled DiT generates the 3D anchor layout from the camera trajectory before synthesizing content, creating an explicit camera-to-geometry-to-content control path.The conclusion reports strong generation quality, geometric consistency, and camera controllability at about half the computational cost of dense latents.Source: §5 · p. 10

A IMPLEMENTATION DETAILS

The encoder anchors content features at fixed 3D positions and aggregates visual evidence through complementary attention paths. The decoder reconstructs RGB-D views, while scene generation separates camera-controlled geometry generation from geometry-guided content generation and position correction.

A.1 LATENT ENCODER

  • The encoder keeps each latent's 3D position fixed while associating it with a 64-dimensional content feature, using depth-consistent and multi-view attention paths to balance texture reliability and geometric coverage.Explicit coordinates also guide image feature sampling and provide spatial relationships for latent self-attention.Source: §A.1 · p. 13

A.2 LATENT DECODER

  • The decoder uses local latent interactions to reconstruct RGB and uses projected latent depths as a prior that its depth decoder corrects.The RGB head upsamples in two stages, by factors of 7 and 2.Source: §A.2 · pp. 13–14

A.3 CAMERA-CONTROLLED POINT DIFFUSION

  • Point Diffusion predicts target 3D positions from the reference scene and camera trajectory, using ray-cell cues to distinguish geometrically constrained regions from unobserved ones.Projected point queries interact with nearby ray tokens across future views, directly constraining point positions and coverage.Source: §A.3 · pp. 14–15

A.4 GEOMETRIC-GUIDED CONTENT DIFFUSION

  • Content Diffusion generates features from the predicted layout and corrects local position errors, with geometry correction delayed until content is relatively accurate.Independent content and geometry noise levels prevent content noise from serving as an implicit signal of geometric quality.Source: §A.4 · p. 15

B.1 LATENT DECODER RECONSTRUCTION

The decoder reconstructs RGB and depth observations from the encoded 3D latents. The visualized reconstructions are high-fidelity despite the representation's sparse, unstructured latents.

B.1 LATENT DECODER RECONSTRUCTION

  • The decoder reconstructs high-fidelity RGB and depth observations from sparse, unstructured 3D latents.Figure 7 visualizes the RGB and depth reconstructions.Source: §B.1 · p. 15

B.2 ABLATION STUDY OF LATENT ENCODER-DECODER.

The encoder's dual-path attention combines reliable local texture evidence with geometric information from multiple views. Vision-encoder comparisons favor DA3-Base, which matches VGGT at lower computational cost.

Dual Path Deformable Cross Attention

  • Combining the Main and Multi-View paths achieves superior RGB and depth generation, while either path alone favors its respective strength: RGB-related metrics for Main and depth metrics for Multi-View.Table 10 reports the dual-path ablation on DL3DV.Source: §B.2 · pp. 15–16, Table 10 · p. 16

Different Vision Encoders

  • DA3-Base matches VGGT at lower computational cost, while VAE performs worst on depth prediction and subsequent depth reconstruction.The comparison also includes DINOV2; DINOV2 and VAE require an added learnable depth prediction head because they do not directly output depth.Source: §B.2 · p. 16, Table 11 · p. 16

B.3 3D SCENE GENERATION

Scene generation tests report superior performance on out-of-distribution datasets and gains from reference-based point noise and geometry correction. The discussion contrasts compact, controllable scene latents with feed-forward 3D Gaussians' reconstruction advantages.

B.3 3D SCENE GENERATION

  • The 1-view and 2-view visualizations show generation both with and without camera control.Figures 8–9 depict the 1-view setting; Figures 10–11 depict the 2-view setting.Source: §B.3 · pp. 16–17, Figure 8 · p. 17, Figure 9 · p. 17
  • On out-of-distribution TartanAir and ScanNet++, the method achieves superior generation performance, indicating generalization beyond the training distribution.The comparison is reported for 1-view appearance generation using 2D and 3D metrics.Source: §B.3 · p. 16, Table 12 · p. 18
  • Reference Noise achieves higher performance than Gaussian Noise, and geometry correction improves performance with almost no additional overhead.The noise-scheme and correction ablations are reported in Tables 13 and 14, respectively.Source: §B.3 · pp. 16–17, Table 13 · p. 18, Table 14 · p. 18

C.1 DIFFERENCE WITH FEED-FOWARD 3D GAUSSIAN SPLATTING

  • Compared with feed-forward 3D Gaussians, the compact scene-level latents reduce representation size and generative-learning burden, but rely on a learned decoder and do not match Gaussian rendering speed or fine-detail fidelity.The stated trade-off is efficient, high-fidelity reconstruction with 3DGS versus compact, controllable scene generation with this representation.Source: §C.1 · p. 18

C.2 LIMITATIONS

SceneAE’s sparse 3D latents have lower reconstruction sharpness than conventional video-VAE latents, and the current representation is designed for static scenes.

Rendering Fidelity

  • SceneAE has lower reconstruction sharpness than conventional video VAEs because it compresses each scene into sparse 3D latents decoded into views.Dense video-VAE latents preserve spatially aligned appearance information at every view.Source: §C.2 · p. 18
  • The sparse representation may smooth high-frequency textures and lose detail in thin structures, object boundaries, and view-dependent appearance.More spatially adaptive anchors and a higher-capacity rendering decoder are proposed as directions for closing the fidelity gap.Source: §C.2 · p. 19

Dynamic-scene Modeling

  • Persistent 3D anchors work for static scenes but constrain moving or deformable objects, whose observations across time can produce ambiguous aggregation, duplicated geometry, or inconsistent appearance.The generative model completes spatial scenes without explicitly modeling motion or temporal evolution.Source: §C.2 · p. 19

Related papers

Loading 2610.19708v1…