PaperlayerView exact revision on arXiv

Source-linked AI summary

CGGS: Consistency-Augmented Geometric Gaussian Splatting for Ego-Centric 3D Scene Generation

Zhenyu Sun, Xiaohan Zhang, Qi Liu, Huan Wang

arXiv:2607.03819v3cs.GRcs.AI

TL;DR

Ego-centric 3D generation struggles with limited view overlap, cross-view inconsistency, semantic misalignment, and geometric distortion. CGGS combines consistency-augmented multi-view diffusion, flow-based layout estimation, and hierarchical Gaussian refinement, with experiments showing coherent, accurate text-driven 3D scenes.

  • Problem

    Ego-centric generation lacks holistic cross-view coherence and semantic alignment, while multi-view synthesis struggles with inter-view consistency despite distortion-free, detailed perspectives.

  • Method

    CGGS generates consistent text-aligned ego-centric 2D priors, converts them into coarse 3D layouts using flow-depth estimation, and refines Gaussian scenes with MID and hierarchical optimization.

  • Results

    Generation results show strong cross-view consistency, text alignment, and vivid 3D scenes, while ablations report improved semantic coherence and geometric reconstruction from the full design.

  • Takeaways & Limitations

    CGGS supports more diverse and realistic 3D content creation and virtual environment exploration within the evaluated ego-centric text-to-3D setting.

Abstract

from arXiv · show

Challenges remain in ego-centric 3D scene generation due to limited view overlap and the dominant influence of individual perspectives on scene interpretation. These factors hinder the creation of viewpoint-consistent and semantically aligned visual content, as well as the construction of accurate geometric structures. In this paper, we propose CGGS, a text-to-3D framework aiming to enhance 3D-content-awareness and address geometric distortions in ego-centric scene generation. Firstly, the Ego-centric Generator is proposed by fine-tuning a Multi-View Latent Diffusion Model with consistency-augmented loss to generate consistent, high-fidelity 2D content aligned with textual descriptions. Then, Layout Decorator leverages optical flow and point-track correspondence to estimate depth, therefore producing dense point clouds as coarse layouts from the ego-centric 2D priors. Building on this initialization, Geometric Refiner is proposed to enhance 3D Gaussian reconstruction via an entropy-based Mutual Information Depth Loss (MID) combined with a hierarchical optimization scheme for improving visual quality and geometric structure. Comprehensive experiments demonstrate that CGGS outperforms previous methods in generating coherent and accurate text-driven 3D scenes.

I. INTRODUCTION · II. RELATED WORK

CGGS addresses ego-centric text-to-3D generation by combining consistency-enhanced multi-view 2D priors, flow-based dense layout initialization, and MID-supervised hierarchical 3D Gaussian refinement. It is motivated by the trade-off between panoramic continuity and multi-view geometric fidelity, while building on prior diffusion-, progressive-expansion-, and 3DGS-based methods.

  • I. INTRODUCTION: Panoramic representations provide global continuity but introduce severe polar distortions that violate pinhole-camera assumptions in 3DGS and SfM pipelines.Equirectangular projection causes radial stretching and bending near the image boundaries.
  • I. INTRODUCTION: Multi-view representations avoid projection distortion and preserve local high-frequency details, but lack a unified canvas and therefore struggle with inter-view consistency.The paper frames panoramic and multi-view generation as complementary but problematic 2D-prior choices for ego-centric reconstruction.
  • I. INTRODUCTION: CGGS learns ego-centric 3D representations from text-aligned 2D images through hierarchical 3D Gaussian optimization using Matterport3D, RealEstate-10k, and CO3Dv2.The stated goal is domain-free, realistic 3D generation from textual descriptions.
  • I. INTRODUCTION: The Ego-centric Generator fine-tunes a Multi-View Latent Diffusion Model with Consistency-Augmented Loss to improve semantic faithfulness and cross-view consistency.It produces ego-centric 2D priors aligned with textual descriptions.
  • I. INTRODUCTION: The Layout Decorator uses optical flow and point-track correspondences in a Flow-Depth Estimator to convert ego-centric 2D priors into a dense coarse 3D layout.This initialization addresses inefficiencies and failure modes of direct SfM on ego-centric views.
  • I. INTRODUCTION: The Geometric Refiner applies hierarchical 3D Gaussian optimization with Mutual Information Depth Loss to sharpen structural details, enforce cross-view consistency, and improve geometric precision.The refinement operates on the initial 3D layouts produced by the Layout Decorator.
  • II. RELATED WORK: Diffusion models became a strong alternative to GANs for prompt-conditioned 2D generation, supporting customized Latent Diffusion Models and classifier-free guidance for improved textual fidelity.Earlier GANs struggled with textual prompt interpretation and dataset-specific biases.
  • II. RELATED WORK: Prior text-to-3D and scene-level methods use CLIP-NeRF optimization, meshes, diffusion-based distillation, progressive image expansion, monocular depth, panoramas, and 3DGS, but remain vulnerable to artifacts, depth-alignment errors, scarce overlap, and viewpoint bias.CGGS differs by constructing dense point clouds from ego-centric 2D priors before MID-supervised hierarchical Gaussian refinement.

III. PRELIMINARY

The preliminary section introduces latent diffusion generation and correspondence-aware attention for multi-view consistency. It also notes that distribution alignment alone is insufficient to balance multi-view consistency with text-image alignment.

  • Latent Diffusion Models: Latent diffusion models compress data into latent representations, denoise them iteratively from random noise, and decode the final latent into an output image.An autoencoder maps x to z = E(x) and approximately reconstructs x via ˆx = D(z).
  • Correspondence-Aware Attention: CAA blocks are integrated into a pretrained Stable Diffusion UNet while other modules remain frozen to preserve the original model’s functionality.The CAA blocks are trained with a dedicated loss.
  • Multi-View Alignment: Simply optimizing KL divergence between multi-view forward and denoising processes is insufficient to balance multi-view consistency and text-image alignment.Panfusion instead uses a dual-branch structure and adds a panoramic perspective to enhance text-image consistency.

IV. METHODOLOGY

CGGS formulates ego-centric multi-view scene generation around conditional image distributions, interpolated motion sequences, and 3D Gaussian rendering. Its methodology combines an Egocentric Generator, Layout Decorator, and Geometric Refiner to construct coherent 3D layouts and representations.

  • Problem Formulation: Ego-centric multi-view images X are modeled conditionally on the remaining views and camera trajectory settings C.Each image follows p{xi | x1, ... xi−1, xi+1, ... , xN, C}.
  • Problem Formulation: The images are interpolated to N′ views, while their ego-centric 2D priors form a motion sequence M for deriving dense point-cloud layouts.The interpolation satisfies N′ > N.
  • Problem Formulation: Each image xi is rendered from a 3D Gaussian representation G under camera pose ci using rendering operator R.The formulation expresses this rendering relationship as xi = R.
  • Overall Pipeline: CGGS uses an MV-LDM Egocentric Generator, a Flow-Depth Estimator Layout Decorator, and a Geometric Refiner in three synergistic stages.The generator conditions on camera trajectories and textual prompts with consistency augmentation, while the decorator produces a dense coarse point cloud because conventional SfM is inefficient and low-quality for ego-centric views.

A. Ego-centric Generator

The Ego-centric Generator synthesizes jointly coherent multi-view images while addressing semantic misalignment and structural artifacts caused by cross-view inconsistencies. It augments latent-diffusion training with a multi-scale consistency objective that harmonizes view-specific gradients.

  • Ego-centric generation requires holistic coherence and semantic alignment, but cross-view inconsistencies produce mismatched content representations and physically implausible fragmented structures.
  • MVDiffusion simultaneously synthesizes N multi-view images covering a 360-degree scene, yet perspective-dependent gradients can create conflicting optimization signals.
  • A consistency-augmented loss is added to the latent-diffusion objective to align gradients across perspectives during CAA-block training.
  • The regularizer uses all L convolutional layers of a VGG-16 network, integrating structural signals across multiple spatial scales through hierarchical composition and parameter sharing.The network supplies a structured inductive bias that aligns per-view gradient updates into a common subspace.
  • He Initialization with frozen, non-pretrained weights turns the VGG-16 feature extractor into a structured random projection without ImageNet-derived semantic biases.
  • The Jacobian transpose projects view-specific updates onto a shared singular subspace, increasing their cosine similarity and reducing inter-view gradient conflicts.The resulting objective combines precise score matching with a unified, multi-scale semantic prior.

B. Layout Decorator

Layout Decorator converts ego-centric 2D priors into a consistent coarse 3D layout by combining optical flow, point tracks, and depth estimation. The resulting back-projected point clouds address scale ambiguity, drift, and cross-view geometric misalignment despite narrow baselines and occlusions.

  • Layout Decorator: Monocular depth priors suffer from scale ambiguity and inconsistent deformations, producing stratified and misaligned point clouds under direct back-projection.Flow and point correspondences are introduced as relative geometric constraints to enforce cross-view alignment.
  • Layout Decorator: Optical flow establishes dense pixel-level correspondences across adjacent ego-centric views modeled as interpolated image sequences.The image sequence is treated as a short video stream with optical-flow motion sequences.
  • Layout Decorator: Long-term Point Tracks rectify cumulative flow drift, while joint optimization harmonizes the scale and geometry of initial depth priors into consistent per-frame depth maps.The depth estimation network acts as a geometric regularizer during the supervised correspondence process.
  • Layout Decorator: Back-projecting depth with camera calibration and merging pixels yields a unified coarse point cloud that faithfully reflects scene geometry despite narrow baselines and frequent occlusions.The correspondence loss provides supervised signals that iteratively enhance depth estimation and improve 3D layouts.

C. Geometric Refiner

Geometric Refiner builds on Layout Decorator point clouds to inject depth-aware structure into 3DGS, using mutual-information depth supervision and hierarchical supplementary-camera optimization to improve geometric consistency and reconstruction robustness.

  • Depth-aware refinement: Geometric Refiner uses Layout Decorator point clouds and structural information to guide Gaussian cloning and splitting, enabling finer detail while preserving geometric consistency.It incorporates depth-aware geometric structure into the original 3DGS framework.
  • Depth-aware refinement: MID loss maximizes mutual information between rendered and ground-truth depth distributions, aligning structural entropy for complex, non-linear geometric correspondence.The rendered depth comes from differential rasterization, while the Layout Decorator depth is treated as ground truth.
  • Depth-aware refinement: The total training loss combines MID with RGB objectives through the scalar weight λMID.λMID balances the mutual-information depth term against RGB objectives.
  • Depth-aware refinement: Unlike Pearson-based scale-invariant depth objectives, MID is designed to avoid suppressing high-frequency geometric details and sensitivity to outliers that can produce over-smoothed structures.Pearson-based objectives handle global-scale ambiguity but assume a linear relationship between predicted and reference depths.
  • Hierarchical optimization: Supplementary cameras progressively expand the base camera set with shared intrinsics and diverse hierarchical viewpoints, mitigating limited-viewpoint bias during Gaussian optimization.The strategy is inspired by supplementary-camera arrangements in HoloDreamer and multi-stage virtual cameras in DreamScene360.
  • Hierarchical optimization: At each expansion stage, rendering across all cameras provides depth and RGB supervision, while gradual camera growth improves reconstruction robustness and 3D Gaussian integrity.Each stage applies rotational and translational offsets defined by ΔRk and Δtk.

V. EXPERIMENTS · A. Implementation Details

CGGS is evaluated with real-world multiview datasets and an implementation spanning prompt expansion, ego-centric image generation, geometric initialization, and staged 3D Gaussian optimization. The reported setup specifies dataset roles, generator training resources, reconstruction-loss weights, and per-scene optimization time.

  • A. Implementation Details: Scene-description prompts imitate MVDiffusion’s sentence structure and are expanded with GPT-4 across varied scenarios.The implementation uses Matterport3D, RealEstate10k, and CO3Dv2 as real-world multiview datasets.
  • A. Implementation Details: The Ego-centric Generator uses N = 8 views, a 90° horizontal FOV, a 45° rotation angle, and λaug = 0.5.It generates eight ego-centric multiview images at 512 × 512 resolution.
  • A. Implementation Details: Layout Decorator uses interpolation number N′ = 20 to support sequence-based geometric initialization.The implementation-details passage specifies this interpolation setting for Layout Decorator.
  • A. Implementation Details: Matterport3D supports text-driven ego-centric generation, while RealEstate10k and CO3Dv2 support flow-depth estimation at scene and object levels.Matterport3D contains 194,400 images and 10,800 panoramic views across 90 building-scale scenes.
  • A. Implementation Details: Fine-tuning the CAA blocks takes about 35 ∼40 hours on four NVIDIA RTX A6000 GPUs, using batch size 4 per GPU.The generator produces eight ego-centric multiview images with 512 × 512 resolution.
  • A. Implementation Details: Geometric Decorator implements hierarchical optimization by adding camera sets at a predefined initialization iteration and deploying four further cameras at each subsequent stage.Added cameras share intrinsic parameters K and extrinsic parameters Eij and are arranged around each base camera with extrinsics Ei.
  • A. Implementation Details: The reconstruction loss sets λMID = 0.05 and λSSIM = 0.2, disables opacity reset, and otherwise follows 3DGS configurations.L1 is defined as the per-pixel absolute difference between reference and rendered images.
  • A. Implementation Details: 3D Gaussian optimization takes about 3 minutes per scene while disabling opacity reset to accelerate convergence and maintain rendering quality.The remaining configurations are consistent with 3DGS.

B. Generation Results · C. Qualitative Comparison · D. Quantitative Comparison

CGGS produces text-aligned, cross-view-consistent 2D priors and geometrically coherent 3D scenes, with qualitative comparisons showing detailed geometry and high-fidelity novel views. Quantitative evaluation reports superior semantic alignment and overall perceptual performance, while ablations identify consistency-augmented loss as important for cross-view quality.

  • B. Generation Results: CGGS generates egocentric 2D priors with strong cross-view consistency in style and content while remaining aligned with text prompts.These results reflect the effectiveness of the Ego-centric Generator.
  • B. Generation Results: CGGS produces vivid 3D scenes from complex prompts with geometrically coherent structures.The result highlights the contributions of the Layout Decorator and Geometric Refiner.
  • C. Qualitative Comparison: Compared with LucidDreamer, Director3D, and DreamScene360, CGGS generates abundant geometric details and high-fidelity novel views.LucidDreamer shows abrupt style and content shifts under modest viewpoint variation, while Director3D tends toward simple layouts for intricate text.
  • D. Quantitative Comparison: The quantitative comparison evaluates generation and reconstruction quality across diverse GPT-4-generated descriptions spanning scene types, complexity, style, and semantic content.Metrics include CLIP-Score for semantic alignment and Q-Align and CLIP-IQA measures such as Sharp and Colorful.
  • D. Quantitative Comparison: Text2Room is evaluated only on indoor scenes because its outdoor rendered views contain large black artifacts and missing geometry.For LucidDreamer, CGGS supplies the first generated multi-view image as the required initial input; Director3D is not evaluated for image reconstruction quality because it uses no intermediate reference images.
  • D. Quantitative Comparison: 26.253: CGGS achieves the highest CLIP Score, indicating superior semantic alignment between generated 3D scenes and textual descriptions.Director3D and DreamScene360 show strengths in local attributes such as sharpness and resolution, but CGGS has the best overall perceptual performance.
  • D. Quantitative Comparison: Without consistency-augmented loss Laug, cross-view texture discrepancies, abrupt background artifacts, and physically implausible scene anomalies become pronounced.Examples include exposed bedroom ceilings and floating, distorted beach trees.
  • D. Quantitative Comparison: The Laug ablation reports that consistency-augmented loss improves diffusion-process training, semantic alignment, and perceptual quality.The study compares Ego-centric Generator training with and without Laug, including training time.

E. Ablation Study

The ablations show that consistency augmentation improves semantic alignment and cross-view coherence, while the Layout Decorator and Geometric Refiner strengthen 3D structure and visual fidelity. Together, these components address sparse-view failures and upstream inconsistencies in ego-centric scene generation.

  • Ego-centric Generator: Removing Laug causes chaotic layouts, degraded cross-view texture consistency, and implausible image artifacts.The Ego-centric Generator is described as crucial for semantic alignment and cross-view coherence.
  • Ego-centric Generator: Consistency augmentation reverses the quality relationship, making global semantic and visual coherence surpass inter-view coherence.Without Laug, inter-view output quality exceeds global quality; with consistency augmentation, the global level surpasses the inter-view level.
  • Layout Decorator: Under sparse-view settings with little overlap, the proposed Layout Decorator provides more reliable 3D structure than conventional SfM methods, which may diverge or yield suboptimal spatial layouts.The comparison substitutes COLMAP while keeping other CGGS modules unchanged and evaluates it with and without identical camera trajectories.
  • Geometric Refiner: Combining MID and HO produces higher-fidelity content closest to ground truth, while depth supervision alone slightly degrades rendering quality through stricter structural constraints.HO denotes hierarchical optimization, PD Pearson Depth loss, and MID the MID loss; configuration (e) is the full setting.

F. Generalization Analysis

CGGS demonstrates domain-free generalization across diverse indoor and outdoor scenes, including out-of-domain environments absent from training, while preserving semantic faithfulness and coherent novel-view synthesis. However, per-scene optimization increases computation time and limits generalization.

  • Domain-Free Generation & Generalization: The framework produces richly detailed, high-fidelity scenes with considerable diversity, cohesive semantic content, and harmonized visual styles for intricate textual descriptions.These properties are highlighted in the additional generation results.
  • Domain-Free Generation & Generalization: Additional results across indoor and outdoor scenarios show semantically faithful 2D priors, coarse 3D layouts, and effective novel-view synthesis from complex textual descriptions.The results are presented in Fig. 9 and Fig. 10 and support the effectiveness of the full pipeline.
  • Domain-Free Generation & Generalization: CGGS generates diverse out-of-domain scenes, including urban environments and scenes dominated by humans, animals, and other biological entities.The evaluation includes camels in deserts, people in markets, and creatures underwater.
  • Limitations and Future Work: Per-scene optimization remains a limitation because it increases computation time and hampers generalization.Future work targets dynamic ego-centric scene synthesis and visual-language navigation with LLMs.

VI. CONCLUSION

CGGS is a text-to-3D framework for ego-centric 3D scene generation that combines high-fidelity prompt-aligned 2D synthesis, reliable layout initialization, and Geometric Refiner optimization with 3D Gaussians.

  • CGGS addresses ego-centric 3D scene generation through a text-to-3D framework.
  • Ego-centric Generator synthesizes high-fidelity 2D content aligned with textual prompts.
  • Layout Decorator produces reliable scene initialization from ego-centric 2D priors.
  • Geometric Refiner further optimizes the scene using 3D Gaussians as the scene representation.
Loading 2607.03819v3…