TriFold: Folding irregular 3D Gaussians into structured latents for 3D Scene Generation

1 Introduction

• Native 3DGS scenes vary widely in primitive count and spatial density, while neighboring Gaussians can differ sharply in geometry and appearance.This heterogeneity motivates a latent that is structured for generative modeling while preserving scene-specific cardinality, spatial distributions, and local attributes.Source: §1 · p. 1
• TriFold encodes variable-cardinality Gaussian sets into fixed-layout triplane latents using multiple learned queries per cell and cross-plane attention.The variational latent jointly represents geometry and appearance.Source: §1 · pp. 2–3
• The decoder localizes active regions and uses optimal-transport-guided set prediction to reconstruct Gaussian candidates with content-adaptive local counts.Query capacity is an upper bound, not a fixed allocation; Gaussian centers use optimal-transport matching, while other attributes receive full-scene rendering supervision.Source: §1 · pp. 2–3
• TriFold trains latent flow matching for point-cloud-conditioned synthesis and single-image-conditioned amodal generation.For partial images, generation preserves observed content while inferring plausible geometry and appearance in unobserved regions.Source: §1 · p. 3
• Experiments report high-fidelity reconstruction and generation in both conditional settings.The authors present this as evidence for structured triplane latents in native 3D Gaussian scene generation.Source: §1 · p. 3

2 Related Work

• Native 3DGS adapts primitive count and spatial density to scene content, whereas latent diffusion and flow-matching models typically use fixed-shape tensors.Triplanes offer a regular layout balancing spatial expressiveness and computational efficiency, but mapping adaptive Gaussian sets into them while retaining scene-dependent density remains a challenge.Source: §2 · p. 3
• Prior Gaussian-latent methods impose structure through grids, fixed-cardinality canonical tokens, sparse spatial support, or continuous fields.These approaches use different constraints, including fixed global or local capacity, occupancy-dependent support, and continuous functional parameterization.Source: §2 · p. 4
• TriFold instead regularizes the latent as a fixed-layout triplane while directly encoding Gaussian sets with scene-dependent cardinality and density.It jointly represents geometry and appearance and decodes native primitives with adaptive spatial allocation and local counts.Source: §2 · p. 4
• For amodal reconstruction, TriFold targets a direct Gaussian-to-latent-to-Gaussian interface, unlike cited approaches using scene tokens, volumetric priors, or point-cloud-structured object latents.The stated target is scene-level native 3DGS with jointly represented geometry and appearance and content-adaptive primitive distributions and counts.Source: §2 · p. 4

3.1 Unified Geometry–Appearance Gaussian Latent Space

• The VAE maps unordered, variable-cardinality Gaussian sets—with position, scale, rotation, opacity, and appearance—into a fixed latent representation.Source: §3.1 · p. 5

3.2 Triplane Encoder

• On each plane, projected Gaussian features are aggregated into multiple learned-query slots, with cross-plane attention and a variational bottleneck producing the triplane latent.Source: §3.2 · p. 5

3.3 Triplane Decoder

• The decoder predicts occupancy first, then uses a query-based set decoder and optimal-transport matching to recover local Gaussian candidates with adaptive spatial allocation and primitive counts.Source: §3.3 · p. 5

3.4 Joint Supervision for Gaussian VAE Training

• Occupancy training balances spatial coverage and sparsity with pointwise regression and voxel-center margins; phantom-voxel negatives teach rejection of false-positive proposals.Source: §3.4 · pp. 6–7
• Optimal transport softly matches predicted candidates to local targets; centers receive direct supervision, while rendering constrains scale, rotation, appearance, and opacity.Source: §3.4 · p. 6
• TriFold better captures scene geometry and appearance than GaussianGPT in the qualitative VAE reconstruction comparison.Source: Figure 3 · p. 7

3.5 Scene Generation over Gaussian Latents

• With the VAE frozen, a conditional latent flow-matching model uses image features lifted into aligned planes or supports point-cloud-conditioned completion.Source: §3.5 · p. 7

4.1 Datasets and Evaluation Protocol

• Evaluation uses InteriorGS crops and uncropped Aria scenes, with scene-level train/test splits made before cropping to prevent scene overlap.The datasets provide over 19,600 InteriorGS crops and approximately 8,600 Aria samples; candidate views for single-image pairs are filtered for informativeness.Source: §4.1 · pp. 7–8

4.2 Scene Reconstruction

• 5.15 dB: TriFold improves PSNR over GaussianGPT and achieves the best results across PSNR, SSIM, LPIPS, and CD.Table 1 compares Gaussian representation methods; its CD values are multiplied by 103.Source: §4.2 · p. 8, Table 1 · p. 8

4.3 Scene Generation

• The single-image model achieves unobserved-region FID of 67.81 and COV-CD of 85.24%, while full-point-cloud conditioning yields better observed-region appearance.Single-image appearance is evaluated separately in observed and unobserved regions; full-point-cloud evaluation covers the complete scene.Source: §4.3 · p. 9, Table 2 · p. 9

4.4 Ablation Studies

• Training on more data brings consistent gains, including further improvements when Aria is added, and InteriorGS-only training transfers to Aria without Aria training data.The cross-dataset evaluation is on the Aria test set; an in-domain trained model is provided as a reference.Source: §4.4 · p. 9, Table 4 · p. 10, Table 3 · p. 10
• The multi-query encoder and query-based set decoder each improve the baseline, work complementarily, and benefit further from rendering supervision, position loss, and phantom-voxel training.The full model includes phantom-voxel training, which adds gains across all reported metrics.Source: §4.4 · p. 10, Table 5 · p. 11
• Point-based backbones outperform the sparse 3D CNN, QFormer improves over mean pooling, and the set decoder beats a fixed-output MLP.The comparisons support query-based aggregation for triplane-cell collisions and adaptive set prediction for variable-cardinality Gaussians.Source: §4.4 · p. 10, Table 6 · p. 10

• TriFold is a native scene-level 3D Gaussian VAE that maps Gaussian sets with adaptive cardinality and density into fixed-layout triplane latents.Its latent jointly represents geometry and appearance and supports conditional generation, including amodal completion from partial observations.Source: §5 · p. 11

• Scene-level Gaussian cardinality and local density vary substantially, complicating fixed-dimensional latent modeling; a useful representation regularizes layout while preserving adaptive allocation.The appendix describes total primitive counts varying by more than a factor of four, alongside clutter and complex indoor occlusions.Source: A Appendix · p. 16
• Native 3DGS modeling means encoding pre-fitted Gaussian sets and reconstructing adaptive Gaussian outputs without an intermediate 3D representation.This differs from image-conditioned 3DGS VAEs, whose latents are shaped by image-conditioned 3D reconstruction.Source: A Appendix · pp. 16–17
• Across 100 scene crops, primitive counts range from 351,306 to 1,457,726, with a median of 682,845; occupied-voxel density has a long tail.97.8% of occupied voxels contain at most 24 Gaussians, while the maximum reaches 389 per voxel.Source: Figure 7 · p. 17
• The local decoder uses K = 24 as a candidate-capacity ceiling, retaining content-dependent subsets and at most K matching targets in over-populated voxels.Rendering supervision still uses views rendered from the complete fitted scene before local selection.Source: A Appendix · p. 17

• Each fitted Gaussian has 59 values: a 3D center, 48 degree-3 spherical-harmonic coefficients, an opacity logit, three log-scales, and a rotation quaternion.Source: §A.1 · p. 17
• InteriorGS scenes use fixed-extent B = 8 crops, while Aria uses a per-scene robust center-based scale; both map to a padded cubic domain of side length 1.1 at 2563 resolution.Center and occupancy coordinates are transformed consistently, and Gaussian log-scales are adjusted for the spatial scale change.Source: §A.1 · pp. 17–18
• Over-populated voxels retain at most 24 target primitives for set-level matching, selected with an opacity- and scale-aware heuristic; rendering supervision uses the complete scene.Source: §A.1 · p. 18
• Occupancy supervision samples centers, two shells, and far-field negatives, then randomly uses 250,000 cached locations per iteration.The four sample groups account for 15%, 20%, 35%, and 30%, respectively; the shell radii are 0.47 and 1.19 voxels.Source: §A.1 · p. 18

A.2 Gaussian VAE Architecture

• Four learned queries per occupied cell summarize unordered projected primitives into multiple slots, reducing the information bottleneck of single-vector pooling.The queries capture diverse summaries of primitives colliding in a cell.Source: §A.2 · p. 18
• Plane embeddings, cell offsets, Fourier encodings, and 3D-context features encode each primitive’s plane, cell, and location before aggregation.The cell center is also Fourier-encoded for the learned queries.Source: §A.2 · p. 18
• Sparse cross-plane attention lets tokens with matching query indices attend across each primitive’s aligned xz, xy, and yz cells.Updates shared by multiple primitives are averaged, avoiding dense attention across all plane cells.Source: §A.2 · p. 18
• Occupancy prediction identifies candidate regions, where 24 learned queries cross-attend to plane context and predict local Gaussian attributes and objectness.Sinkhorn matching supplies order-independent correspondence, and inference prunes candidates using predicted objectness and opacity.Source: §A.2 · p. 19

A.3 Objectness-Aware Set Prediction and OT Matching

• Objectness measures whether a query should instantiate a Gaussian, while opacity measures its rendering contribution; inference ranks candidates using a unified score.The reported VAE selection threshold is 0.1, with an activation margin of 0.15 for matched queries.Source: §A.3 · p. 19
• Candidate enumeration uses discrete thresholding without differentiation through set membership; learned surrogate losses optimize the scores that govern selection.The threshold is a deployment decision based on learned scores rather than a manually tuned post-hoc pruning heuristic.Source: §A.3 · p. 19
• Each voxel predicts K candidates for an unordered target set of at most K Gaussians, with a dummy no-Gaussian entry absorbing unmatched slots.The matching cost is based solely on candidate-to-target Gaussian center distance; Sinkhorn iterations solve the entropy-regularized transport problem, with gradients stopped through the plan.Source: §A.3 · pp. 19–20
• The center-based transport plan both assigns target Gaussians for center supervision and provides soft objectness targets, eliminating a separate objectness-matching heuristic.The non-dummy transport mass yields the soft target; objectness logits use balanced binary cross-entropy.Source: §A.3 · p. 20
• Phantom-voxel training gives false-positive proposals negative objectness supervision, suppressing ghost Gaussians at locations without target primitives.After occupancy warm-up, training alternates between ground-truth and predicted supports.Source: §A.3 · p. 20

A.4 Baseline Implementation and Training

• Baselines use official implementations and retain their architectures, objectives, and optimization settings, with data-loading adapted for a shared dataset and split.Can3Tok scenes are downsampled to its fixed 40,000-Gaussian budget; all baselines are trained from scratch for 80K steps.Source: §A.4 · pp. 20–21
• Under scene-level adaptation, GaussianGPT converges to meaningful reconstructions, while adapted Can3Tok and DiffGS produce substantially lower-fidelity solutions.The reported degraded cases remain in the comparison and should not be interpreted as general limitations of the original methods in their designed domains.Source: §A.4 · p. 21
• Geometric and rendering metrics use common evaluation protocols, and degraded predictions are reported without alignment, exclusion, or special-case metric values.Center-wise Chamfer Distance uses a common normalized coordinate frame; PSNR, SSIM, and LPIPS use the same rendering views.Source: §A.4 · p. 21

A.5 Conditional Latent Flow Matching

• Condition tokens are concatenated with noisy scene tokens for joint self-attention, with separate timestep modulation and patch embeddings for the two token types.Corresponding spatial locations share the same two-dimensional RoPE coordinates.Source: §A.5 · p. 21
• Image-conditioned diffusion uses InteriorGS only; Aria evaluates the Gaussian VAE but is excluded from diffusion training.The fixed spatial extent B = 8 applies specifically to InteriorGS.Source: §A.5 · p. 21
• Image conditioning lifts dense DINOv3 features into 3D using estimated monocular depth, camera parameters, and a shared global depth-scale factor.The lowest-confidence 10% of valid depth pixels are discarded before back-projection.Source: §A.5 · pp. 21–22
• Lifted image features and target Gaussian latents use the same fixed-extent crop transform and canonical coordinate frame.For InteriorGS, the crop extent B is fixed and only the predefined crop translation varies; the image condition requires no target 3DGS coordinates or target-derived spatial statistics.Source: §A.5 · p. 22
• Image features are mean-splatted into aligned xz, xy, and yz condition planes; point-cloud conditions instead encode positions and colors through the frozen triplane and VAE encoders.The resulting spatial condition tokens use dedicated patch embeddings, and either condition is zeroed with 10% probability during training for classifier-free guidance.Source: §A.5 · p. 22
• TriFold’s contribution is structured generation of native scene-level Gaussians, not a new geometric estimator.Its image conditions use pretrained monocular depth and frozen visual features.Source: §A.5 · p. 23

A.6 Training and Evaluation Details

• The VAE is jointly trained end to end with occupancy, center reconstruction, objectness, rendering, phantom-voxel suppression, and KL objectives; rendering supervision separately computes foreground and background RGB errors and downweights background pixels.Training randomly samples 32 camera viewpoints per scene and renders at 640 × 640 resolution. The latent transformer generator is trained for 100k steps.Source: §A.6 · pp. 23–24

A.8 Expanded Ablation Results and Analysis

• Sinkhorn improves PSNR from 25.61 to 25.83, LPIPS from 0.103 to 0.096, and CD from 2.29 to 2.27, while hard matching yields higher SSIM (0.908 versus 0.904).Increasing triplane resolution from 64 to 256 improves PSNR by 4.76 dB and SSIM by 0.053, while reducing LPIPS by 0.059 and CD by 0.38.Source: Table 7 · p. 26, Table 8 · p. 26, §A.8 · p. 26
• PTv3 features show sharper transitions around floors, walls, furniture, and object boundaries than PointNet or sparse 3D CNN features, supporting more accurate assignment to triplane cells.The visual comparison is Figure 10; Table 6 provides its corresponding quantitative comparison.Source: Figure 10 · p. 26, §A.8 · p. 26
• Increasing encoder slots from one to four improves all four metrics, including a 1.43 dB PSNR increase and 0.11 lower CD; gains from four to eight are smaller.Increasing latent channels from 16 to 32 changes PSNR by only 0.06 dB, SSIM by 0.001, LPIPS by 0.002, and CD by 0.02; the reported compact operating point is Q = 4 and Cz = 16.Source: §A.8 · pp. 26–27, Table 9 · p. 27, Table 10 · p. 27
• Increasing decoder capacity from K = 8 to K = 24 generally improves reconstruction, while K = 32 adds only marginal gains and leaves CD unchanged at reported precision.On the reported 256^3 grid, 97.8% of occupied voxels in the fitted scenes contain at most 24 Gaussians; rendering supervision uses the complete target scene.Source: Table 11 · p. 27, §A.8 · p. 27
• Removing any one of the four encoder slots preserves the recognizable global layout but degrades distinct local structures, with artifacts varying by removed slot.Together with slot-count gains, this supports complementary, non-redundant local information, but not strictly disentangled semantic meanings.Source: §A.8 · p. 27, Figure 11 · p. 28

A.9 Latent Representation Analysis

• Linear interpolation between scene latents produces smooth decoded triplane-feature transitions, and reconstructions remain largely preserved under Gaussian noise up to σ = 0.5.Degradation becomes pronounced at σ = 1.0.Source: §A.9 · p. 28, Figure 12 · p. 29

A.10 Additional Conditional Generation Results

• Point-cloud-conditioned samples span varied indoor layouts and form holistic scene-scale environments with robustly generated major surfaces and semantic arrangements.Single-image-conditioned samples preserve prominent input cues, including decorations, windows, cabinets, and furniture, with geometric consistency across novel views.Source: §A.10 · p. 28