PaperlayerView exact revision on arXiv

Source-linked AI summary

GS-Voxel: Fitting-Free Structured Latents for Large-Scale 3DGS Generation

Ming Qian, Zijian Wang, Minchao Sun, Jincheng Xiong, Hang Zhang, Mu Xu, Chi Wang, Baoquan Chen

arXiv:2608.17988v1cs.CV

TL;DR

Pre-optimized 3DGS reconstructions are unordered and spatially irregular, limiting their direct use with structured latent generators for large outdoor scenes. GS-Voxel deterministically converts them into sparse structured latents and supports aerial 3DGS generation, including tiled synthesis across areas larger than the training crop.

  • Problem

    Pre-optimized 3DGS is unordered, spatially irregular, and variable in primitive count, whereas structured latent models require discrete spatial features with fixed channel dimensions.

  • Method

    GS-Voxel deterministically organizes compatible pre-optimized 3DGS primitives into sparse voxels, while a factorized VAE separately encodes voxel geometry and local Gaussian attributes.

  • Results

    Overlap-aware tiled inference generated aerial scenes spanning up to 1,400 m×800 m, substantially beyond the 200 m×200 m training crop.

  • Takeaways & Limitations

    GS-Voxel provides structured latents for compatible pre-optimized aerial 3DGS reconstructions and supports large-area scene generation beyond a single training crop.

  • Takeaways & Limitations

    The current implementation targets SH0 aerial scenes, and robustness to sensor variation and domain shifts among rendered, real, and generated conditions remains unevaluated.

Abstract

from arXiv · show

Many scalable latent 3D generators operate on structured tensors, whereas pre-optimized 3D Gaussian Splatting (3DGS) reconstructions are unordered, spatially irregular, and vary widely in primitive count. We present GS-Voxel, a fitting-free structured latent framework, and evaluate it for large-scale aerial 3D Gaussian scene generation. GS-Voxel deterministically converts a compatible pre-optimized 3DGS reconstruction into sparse active voxels without additional per-scene optimization, retaining the sub-voxel positions and rendering attributes of the selected primitives. A GS-specific factorized VAE then separately encodes voxel geometry and local Gaussian attributes into sparse 3D latents whose size grows with the number of occupied voxels rather than being limited by a fixed scene-wide primitive count. We train image-conditioned flow models in the GS-Voxel latent space to generate aerial 3DGS scenes. A key application enabled by GS-Voxel is large-area scene generation: overlap-aware tiled inference extends synthesis beyond a single training crop conditioned on satellite-view images. Our results show that GS-Voxel provides structured latents for pre-optimized aerial 3DGS reconstructions, with latent capacity that grows with the number of occupied voxels.

1. Introduction

GS-Voxel addresses the representation bottleneck of large-scale aerial 3DGS by converting unordered, irregular, variable-cardinality reconstructions into fitting-free structured sparse latents. A factorized VAE and image-conditioned flow models support aerial scene generation, including overlap-aware tiled synthesis for large areas.

  • Motivation: Large aerial 3DGS reconstructions are unordered, spatially irregular, variable in primitive count, and can exceed 3.0 million primitives per 200 m × 200 m tile.These properties make direct use of common global primitive budgets difficult.
  • GS-Voxel: GS-Voxel deterministically converts supported pre-optimized 3DGS reconstructions into sparse active voxels without additional per-scene optimization.The conversion avoids fitting to a fixed-cardinality scene-wide Gaussian template and global Optimal Transport rearrangement.
  • Latent representation: A factorized VAE separately encodes voxel geometry and local Gaussian attributes into structured sparse 3D latents.The framework retains the sub-voxel positions and rendering attributes of selected primitives.
  • Generation: Image-conditioned flow models trained in GS-Voxel’s latent space generate aerial 3DGS scenes.The evaluation targets large-scale aerial 3D Gaussian scene generation.
  • Large-area synthesis: Overlap-aware tiled inference enables large-area aerial 3DGS synthesis beyond a single training crop.This demonstrates a large-area scene generation application of GS-Voxel.

2. Related Work

Prior work spans object-centric latent 3D generation, outdoor scene creation, and diffusion-based 3DGS synthesis. Closest structured-3DGS approaches impose fixed-cardinality or fitted voxel representations, while TripoSplat supports adaptive output budgets through a different input pipeline.

  • Latent 3D generation: Latent 3D methods decouple compression from generative modeling, but existing feed-forward and latent frameworks remain largely object-centric.Earlier methods lift 2D priors into 3D through score distillation or view-conditioned image synthesis.
  • Outdoor scene creation: Outdoor scene systems synthesize training data, build controllable urban assets, or reconstruct scenes from aerial observations, with Sat2City using sparse-voxel latent diffusion.Sat2City v2 adapts a pretrained structured latent model to real satellite–mesh pairs and generates textured mesh assets.
  • 3DGS generation: Recent 3DGS generators primarily use view-conditioned, view-aligned, teacher-guided, image-diffusion, or functional parameterization strategies.Examples include DiffusionGS, teacher-guided approaches, DiffSplat, and DiffGS.
  • Structured 3DGS latents: Can3Tok structures irregular Gaussian sets with canonical queries but caps reconstructions at 100K Gaussians and selects 40K Gaussians for each VAE input.Its VAE maps the globally fixed-cardinality set to a 64×64×4 latent and decodes a fixed number of Gaussians.
  • Structured 3DGS latents: GaussianCube fits a fixed number of Gaussians before rearranging them into a predefined voxel grid, whereas L3DG learns sparse VQ-VAE latents for object- and room-scale synthesis.GaussianCube uses densification-constrained fitting and Optimal Transport; L3DG performs latent diffusion.
  • Adaptive output capacity: TripoSplat supports adaptive output budgets by encoding feature-augmented surface samples rather than optimized 3DGS parameters, unlike GS-Voxel’s voxel-dependent slot capacity.GS-Voxel’s total slot capacity grows with active voxels but does not adapt decoded density at inference time.

3. Method

GS-Voxel deterministically converts compatible pre-optimized 3DGS reconstructions into sparse voxel-aligned features without per-scene fitting, then encodes voxel geometry and Gaussian attributes with separate sparse VAEs. The representation retains local Gaussian information while providing fixed-width voxel features compatible with sparse 3D latent modeling.

  • Motivation: GS-Voxel addresses the mismatch between unordered, spatially irregular 3DGS primitives and sparse-convolutional models requiring discrete coordinates with fixed channel dimensions.Native 3DGS cannot be passed directly to sparse-convolutional latent models because regions contain variable primitive counts.
  • GS-Voxel Construction: The conversion voxelizes primitives, discards those below an opacity threshold, and retains up to Kin entries per active voxel after sorting by opacity logits.Remaining slots are zero-padded, producing deterministic local ordering and consistent channel dimensions across active voxels.
  • GS-Voxel Construction: GS-Voxel preserves sub-voxel positions and normalized rendering attributes, quantizing retained local features into a sparse voxel-aligned representation.The inverse mapping dequantizes active-voxel features and reconstructs Gaussian parameters over the active support.
  • Factorized VAE: The factorized VAE separates support modeling from attribute reconstruction through a Geometry VAE and a Local Attribute VAE operating on occupied voxels.The Geometry VAE encodes support on a coarsened lattice at (R/8)^3, while the Local Attribute VAE encodes Gaussian attributes on the fine support.
  • Factorized VAE: The Local Attribute VAE decodes fixed per-voxel Gaussian slots on supplied fine support and trains through rendered-image consistency rather than explicit primitive-count matching.Kout need not equal Kin when the rendered 3DGS matches target observations.

4. Training Data Construction

The training data pipeline converts large aerial 3DGS scenes into normalized, filtered windows, renders diverse virtual-camera views, and retains reliable supervision. It combines geometric preprocessing with view-quality filtering to construct training and conditioning-view data.

  • Training Tile Generation and Voxel Filtering: Each aerial 3DGS scene is cropped into fixed-size windows, normalized to [−0.5, 0.5]3, filtered for floating outliers with DBSCAN, and converted into GS-Voxel.Windows use W = 200 m and stride S = 150 m, yielding 50 m overlap; DBSCAN uses ε=0.02 and mmin=50.
  • Virtual Camera Setting: A virtual aerial camera rig samples a 10×10 xy grid across [−0.7, 0.7]2 over five height layers, using layer-specific pitch, compass yaw, sampling density, and field of view.Orientations are set directly from pitch and yaw, while random perturbations vary xy position by ±15% of grid spacing, height by ±0.1, pitch by ±5◦, and yaw by ±30◦.
  • Virtual Camera Setting: The camera rig produces approximately 3.9K candidate renders per window for aerial-view supervision.The grid extends beyond the scene footprint to capture peripheral content at oblique angles, with denser sampling and wider FOV near the ground.
  • VLM-Based View Filtering: Geometric coverage filtering discards renders whose mean accumulated opacity is below τα = 0.9 before VLM-based quality scoring removes unreliable views.Rejected views include empty boundary views, oblique views through Gaussian holes, and renders with blur or floaters.
  • VLM-Based View Filtering: Only retained views supervise the Local Attribute VAE and contribute to the conditioning-view pool used by the flow models.The filtering targets occlusions, blind spots, reconstruction artifacts, and severely degraded supervision.

5. Experiments

Experiments establish the training setup, validate GS-Voxel design choices and factorized VAE components, and demonstrate image-conditioned generation under both standard and large-area evaluation settings.

  • Experimental Setup: The dataset contains approximately 18K samples covering 200 × 200 m2 areas, with 800 validation samples and source-scene-level separation between splits.No source scene or spatial region is shared between training and validation.
  • GS-Voxel Construction: Using R = 256 and Kin = 16 provides the adopted trade-off between direct-conversion fidelity and sparse-support size.Increasing R from 256 to 512 at Kin = 4 raises PSNR from 26.82 to 39.20 while increasing active support from approximately 377K to 980K voxels; the higher-resolution setting uses about 2.6× as many active voxels.
  • Local Attribute VAE: Kout = 4 gives the best PSNR, SSIM, and LPIPS among the tested Local Attribute VAE slot budgets.Kout = 1 lowers reconstruction quality, whereas Kout = 8 adds slots without improving reconstruction under the current objective.
  • Factorized VAE Analysis: The factorized VAEs achieve 0.99 Shape IoU for support reconstruction and 23.09 PSNR, 0.62 SSIM, and 0.331 LPIPS for attribute reconstruction.Geometry and local attributes are evaluated separately because the two VAEs model different quantities.
  • Large-Area Synthesis: Overlap-aware tiled inference generates large-area 3DGS scenes from geographically disjoint satellite-view conditions, with examples spanning up to 1,400 m×800 m.The method extends local content at a fixed ground sampling resolution and decodes scene latents into standard 3DGS primitives.

6. Conclusion

GS-Voxel is presented as a fitting-free structured latent representation for large-scale aerial 3D Gaussian scene generation. Its deterministic sparse-voxel construction and factorized VAE organize selected 3DGS primitives while supporting inputs with millions of primitives.

  • Core contribution: GS-Voxel deterministically organizes selected primitives from compatible pre-optimized 3DGS reconstructions into sparse voxels.The representation is fitting-free and instantiated for large-scale aerial 3D Gaussian scene generation.
  • Core contribution: Its factorized VAE separately encodes occupied support and local Gaussian attributes.
  • Scalability: The design handles millions of input primitives without fitting each scene.
  • Future work: GS-Voxel may provide a basis for future scalable aerial 3D generation and downstream simulation.

7. Limitations and Future Work

GS-Voxel is currently limited to compatible SH0 aerial 3DGS settings, with unresolved robustness, representation-capacity, and coding-efficiency questions. Future work targets broader scene domains, adaptive representations, quantized latents, and downstream evaluation.

  • Current scope: The present implementation and experiments target compatible pre-optimized 3DGS reconstructions in SH0 aerial scenes.GS-Voxel is a representation and factorized VAE framework rather than a general reconstruction method.
  • Data and domain robustness: Training depends on the scale and availability of suitable real-world 3DGS scenes, while robustness to sensor variation and rendered, real, or generated conditioning domains remains unevaluated.Flow models are trained with rendered bird’s-eye-view images but can accept real or generated satellite-view images at inference.
  • Representation limits: Top-Kin local capacity can discard primitives in extremely dense voxels, and finite spatial resolution can hinder preservation of very thin structures.Proposed remedies include adaptive resolution and learned local-capacity allocation.
  • Coding efficiency: The study does not yet report entropy-coded bitrates or rate–distortion curves, leaving memory and storage reduction unassessed.Future evaluation will consider quantized or entropy-coded GS-Voxel latents.
  • Broader domains: Future work will test the fitting-free voxel-local representation on indoor, street-level, and object-level scenes requiring adaptations to attributes, normalization, and training data.These settings differ in camera distributions, spatial scales, and appearance characteristics.
  • Downstream evaluation: Future evaluation should determine whether generated aerial scenes are sufficiently faithful and controllable for planning, simulation, or emergency-response analysis.These downstream uses are identified as targets for assessing practical suitability.

A. GS-Voxel Conversion Procedures

GS-Voxel deterministically converts normalized SH0 Gaussian crops into sparse voxel features through thresholding, voxel assignment, local packing, and quantization. Reconstruction unpacks valid slots and decodes them back into standard 3DGS primitives, while coordinate recovery requires retaining the crop frame separately.

  • Input normalization and coordinate recovery: The conversion assumes a pre-normalized cubic crop and maps it to [−0.5, 0.5]3 while preserving relative Gaussian sizes.Opacity logits, spherical-harmonic coefficients, and rotations remain unchanged; the original frame is recovered using retained (c, L).
  • Default configuration: The default configuration uses R = 256, h = 1/256, Kin = 16, t8 = 5, 8-bit attributes, and log-scales in [−24, −4].Centers within δb = 2/128 outside the cube are clamped, while larger deviations invalidate normalization.
  • Conversion algorithm: Conversion verifies bounds, discards primitives with opacity pi ≤ t8/255, assigns survivors to voxels, computes local attributes, and groups them by voxel.The procedure derives normalized offsets, color, opacity, scale, and rotation before packing voxel-local primitive slots.
  • Reconstruction algorithm: Reconstruction unpacks each active voxel’s Kin quantized slots, removes slots below t8, decodes valid attributes, and returns the concatenated Gaussian set.Decoded positions, opacities, colors, scales, and rotations are appended as standard 3DGS primitives.

B. Details of Overlap-Aware Tiled Inference

Overlap-aware tiled inference extends scene generation beyond one training crop by assembling overlapping tiles conditioned on aligned satellite-image patches. Overlap latents, staged generation, shared noise, and feathered accumulation help preserve continuity and reduce visible seams.

  • Overview: Large scenes are built tile-by-tile with overlapping boundaries, conditioning each tile on the aligned patch from a supplied large-footprint satellite-view image.The conditioning patch matches the tile’s ground sampling resolution.
  • Tile Traversal and Overlap: Tiles follow a BFS-spiral traversal from the center; each spans 323 coarse-grid cells, corresponds to a 2563 fine grid, and overlaps neighbors by eight cells with stride 24.Tiles are arranged on the horizontal plane.
  • Overlap Handling: For subsequent tiles, stored neighbor-overlap latents are noised to the current sampling time and constrain repaint-style inpainting, while non-overlap regions sample freely.This preserves boundary context from previously generated neighbors.
  • Seam Reduction: Generation propagates coarse structure, predicts fine active geometry support, then decodes attributes into Gaussian parameters; shared overlap noise and feathered accumulation reduce seams.Small coarse-grid holes are filled by nearest-donor copying before required subdivisions are enforced.
  • Settings: 50 Euler steps are used per flow-matching stage with classifier-free guidance strength 3.0, and feathering uses a validity offset of two coarse-grid cells.These are the stated tiled-inference settings.

C. Prompt for filtering

The prompt defines strict visual-quality criteria for 3DGS renders, distinguishing fatal internal reconstruction artifacts from acceptable sharp scene boundaries. It assigns scores from 0.00 to 0.90+ based on obstruction, structural integrity, sharpness, and scene coverage.

  • Key Definitions: Fatal artifacts include blurry or smoky blobs, ghosting floaters, and torn-paper holes inside or over scene objects.Clean black regions at scene edges or corners are acceptable when the transition is sharp and clean.
  • Critical Failure: Score 0.00 applies when a giant blurry blob or smoky smoke obscures more than 30% of important scene content.Score 0.25 applies to internal fatal flaws such as torn-paper holes or blurry floaters overlapping main structures.
  • Pass Criteria: Score 0.55 applies to complete or clean-edged images that are generally soft, hazy, or geometrically distorted.Blurry details such as windows and lines remain compatible with this category.
  • Boundary Handling: Scores increase to 0.70 or 0.80 for structurally sound sharp images with significant or tiny clean black boundaries, respectively.The boundary transition must be sharp; a sharp clean boundary receives only a minor penalty rather than a fatal score.
  • Top Score and Output: Score 0.90+ requires crystal-clear quality across the entire view, 100% scene coverage, and no flaws.The output must contain exactly one numeric score in the interval [0, 1] and no additional text.
Loading 2608.17988v1…