GAE: Learning a Geometry-Native Latent Space for 3D-Consistent World Generation

• Geometry foundation models recover depth, cameras, and point maps in a common 3D frame, while generated video can drift geometrically and stray from requested camera paths.The paper traces this gap to the latent space generators evolve.Source: §1 · p. 2
• DA3’s multi-level hierarchy is generation-mismatched: no single level jointly preserves semantics, spatial and cross-view structure, and smooth transport; 3,072 channels span about 11 effective dimensions, with condition numbers from 108 to 1016.Earlier features retain stronger scene structure but are harder to model continuously, while later features are smoother but lose geometric and semantic information.Source: §1 · p. 2
• GAE compresses DA3’s four feature levels into one bottleneck decoded by frozen geometry heads and a learned RGB head, while token and pairwise alignment organize it for flow modeling.Token-wise alignment to co-located C-RADIO features improves transport smoothness and semantic organization; matching posterior pairwise similarities to DINOv2 features restores relational geometry.Source: §1 · p. 2
• A standard DiT-style conditional flow with condition dropout supports text-to-image, camera-controlled video, and reference-conditioned novel-view synthesis, jointly generating target latents that decode to RGB and geometry.Conditioning combines text, metric Plücker rays, and reference views.Source: §1 · pp. 2–3
• GAE reduces FVD by 12.7% on RealEstate10K and 23.1% on DL3DV versus the strongest competing latent; camera-trajectory error is roughly halved on RealEstate10K.The comparisons use matched flow models and training protocols.Source: §1 · p. 3

Generative representation spaces

• GAE learns a compact state natively readable by a geometry foundation model, unlike appearance- or semantic-centered generation representations.Prior methods include variational and vector-quantized tokenizers, latent diffusion, and representation autoencoders; REPA aligns denoiser features without changing the output state, and latent-diffusibility studies show reconstruction alone does not determine generation quality (Kingma & Welling, 2014; van den Oord et al., 2017; Esser et al., 2021; Rombach et al., 2022; Tong et al., 2026; Singh et al., 2026; Zhong et al., 2026a).Source: §2 · p. 3

Geometry-aware visual generation

• Across geometry-aware generation, geometry typically augments an appearance-led state as a control, modality, auxiliary representation, or objective; GAE derives its sole evolving latent from geometry.Prior designs include scene-based novel-view synthesis, pose-conditioned diffusion, camera-controlled video, joint RGB-and-geometry outputs, geometry-feature alignment or conditioning, coupled geometry and appearance latents, and geometric-reward post-training (Mildenhall et al., 2020; Kerbl et al., 2023; Liu et al., 2023; Shi et al., 2024; Liu et al., 2024; Gao et al., 2024; Yu et al., 2024; 2025b; Wang et al., 2024b; He et al., 2025; Yin et al., 2026; Stan et al., 2023; Krishnan et al., 2025; Kwon et al., 2025; Guizilini et al., 2025; Wu et al., 2026; Wan et al., 2026; Huang et al., 2026; Mi et al., 2026; Xiang et al., 2026; An et al., 2026).Source: §2 · p. 3

Geometry foundation models as generative states

• Unlike GLD and latent Riemannian flow matching, which directly generate high-dimensional geometry features, GAE learns one compact Euclidean reparameterization of the full hierarchy.GLD cascades selected DA3 or VGGT levels and propagates the remaining hierarchy; latent Riemannian flow matching jointly evolves VGGT’s four normalized levels on their product manifold (Jang et al., 2026; Weijler et al., 2026). GAE reconstructs the hierarchy for frozen geometry readout and learned RGB decoding.Source: §2 · p. 3

3.1 GEOMETRY-NATIVE CODEC

• GAE fuses and compresses the full DA3 hierarchy into one spatial latent, enabling RGB and frozen-head geometry readouts without a hierarchy-level generation cascade.It preserves the DA3 patch grid while compressing channels to 64 or 128, compared with 3,072 channels in a raw DA3 level.Source: §3.1 · p. 5

3.2 ORGANIZING THE LATENT FOR GENERATION

• C-RADIO (Heinrich et al., 2025) token alignment improves transport and semantic neighborhoods but can damage relations; DINOv2 (Oquab et al., 2024) similarity matching restores them in the flow latent.Both objectives shape the codec latent before generative training, and their teachers and projector are discarded after codec training.Source: §3.2 · p. 6

3.3 CONDITIONAL FLOW MATCHING AND GENERATION

• A single conditional flow transformer evolves target-view latents while clean reference tokens, Plücker camera rays, and text condition generation across text-only, camera-controlled, and reference-conditioned modes.Reference tokens are encoded using observed references alone; all output slots remain noisy, avoiding ODE clamping and permitting a single standard Euler sampler.Source: §3.3 · p. 7

4.1 LATENT-SPACE STUDY

• Fusing the DA3 hierarchy into 64- or 128-channel GAE yields the best overall balance of transport, conditioning, and structure among geometry-native representations.Pixel VAEs are well-conditioned but weakly structured, semantic RAEv2 has strong semantic neighborhoods but no native geometry readout, and raw DA3 features are geometry-native but high-dimensional and poorly conditioned; both raw DA3 layers have 3,072 channels but effective rank around 10.Source: §4.1 · p. 8, Table 2 · p. 8, Figure 3 · p. 8

4.2.1 RGB RECONSTRUCTION

• GAE-128 matches or slightly exceeds raw DA3 L0 on PSNR, LPIPS, rFID, and rFVD while using 24 times fewer channels.Raw L0 reconstructs appearance well with a separately trained RGB head, but its 3,072-channel representation is ill-conditioned and nearly rank-degenerate; raw L3 loses substantially more reconstructable appearance information.Source: §4.2.1 · p. 8, Table 3 · p. 8

4.2.2 GEOMETRY RECONSTRUCTION

• Both GAE variants improve on raw DA3 L0 across depth, point-map, and pose metrics on both datasets, while retaining the codec’s RGB fidelity.Against external systems, GAE has lower camera errors than Gen3R and GLD and lower point-cloud errors than GLD, although Gen3R is slightly better on some depth metrics using a separate geometry latent.Source: §4.2.2 · p. 9, Table 4 · p. 9

4.3.1 RGB GENERATION

• FVD falls 12.7% on RealEstate10K and 23.1% on DL3DV for GAE-64 versus the best non-GAE controlled latent; it also leads paired metrics.GAE-128 achieves the best RealEstate10K FID. Under identical reference images and camera trajectories, GAE-64 better preserves object identity and fine structure as viewpoints move away from the reference.Source: §4.3.1 · p. 9, Table 5 · p. 9, Figure 4 · p. 10

4.3.2 3D CONSISTENCY

• ATE falls 52.8% on RealEstate10K and 23.3% on DL3DV with GAE-64; the best controlled MEt3R scores are 0.1208 and 0.1347, respectively.The two GAE variants achieve the two best controlled VGGT results across both datasets, and Figure 5 shows more coherent scenes and camera motion under the same conditioning.Source: §4.3.2 · p. 10, Table 6 · p. 11, Figure 5 · p. 11

4.3.3 GEOMETRY GENERATION

• GAE-64 leads all six controlled geometry metrics on DL3DV; on RealEstate10K it leads depth and pose, while GAE-128 leads Chamfer and point-map errors.The geometry references are Pi3 estimates on real target frames and dataset camera poses, not DA3-GIANT evaluations. Figure 6 shows GAE tracking prescribed trajectories and preserving scene structure better than Gen3R and GLD; qualitative showcases also include an 81-view rollout and text-conditioned RGB with decoded depth.Source: §4.3.3 · pp. 10–12, Table 7 · p. 12, Figure 6 · p. 12

4.4 ABLATION STUDIES

• Representation shaping improves transport, semantic organization, and relational structure, while backbone capacity and three conditioning choices add complementary generation gains.Token alignment improves transport but weakens relational structure, which Lstruct restores. Clean reference tokens reduce reference–target point-cloud gaps; metric ray conditioning improves scale-sensitive camera recovery, appearance, and view consistency; text-to-image co-training improves FVD and LPIPS without sacrificing MEt3R.Source: §4.4 · p. 13, Table 8 · p. 13

5 CONCLUSION

• GAE reparameterizes multi-level geometry features into a compact latent jointly decodable to RGB and geometry and organized for flow modeling.The paper presents this geometry-native latent as a shared interface between perception and generation.Source: §5 · p. 13
• Under a controlled generator and training protocol, GAE improves reconstruction, visual generation, 3D consistency, and camera control over pixel, semantic, and raw geometry representations.Source: §5 · p. 13
• Larger-scale experiments confirm that GAE’s gains persist with more data, higher resolution, and longer generation sequences.Together, the results show that geometry can define the state in which generation occurs.Source: §5 · p. 13

A IMPLEMENTATION AND EVALUATION DETAILS

• The appendix expands the main paper’s codec, flow, conditioning, training, and evaluation details, with this subsection focused on the geometry-native codec.This point summarizes the scope of the appendix and identifies the focus of A.1.Source: A IMPLEMENTATION AND EVALUATION DETAILS · p. 17, §A.1 · p. 17

A.1 GEOMETRY-NATIVE CODEC DETAILS

• DA3-GIANT (Lin et al., 2026) remains frozen during codec training; its four output levels are normalized, reshaped to spatial grids, and concatenated by channel.The codec preserves the patch grid and compresses only the channel dimension to Cz ∈{64, 128}.Source: §A.1 · p. 17
• The codec uses convolutional encoder and decoder pyramids with spatial self-attention, compressing channels while retaining the feature hierarchy’s spatial layout.The feature levels are concatenated along the channel dimension before compression.Source: §A.1 · p. 17
• The codec reconstructs normalized features, RGB, and frozen DA3 dense-prediction outputs, using geometry pseudo-targets from DA3 features rather than dataset depth labels.The frozen DA3 head is not adapted to reconstructed features, so geometry quality must remain readable through its original interface.Source: §A.1 · p. 17
• C-RADIO (Heinrich et al., 2025), DINOv2 (Oquab et al., 2024), and the Lrepr projection head are used only for codec training and discarded afterward.Codec training samples from the posterior; latent statistics, flow training, and inference use its mean.Source: §A.1 · p. 17

A.2 FLOW-MATCHING PARAMETERIZATION

• Flow training uses standardized posterior means as targets, interpolates them with Gaussian noise, predicts the clean latent, and converts that prediction to velocity.Training occurs after the codec is frozen.Source: §A.2 · p. 17
• The timestep is shared across clip views; controlled models use 28 encoder blocks of width 768 and six decoder blocks of width 2048, changing only latent projections for 128- versus 64-channel codecs.The DDT-style transformer uses 1 × 1 latent tokens and cross-view self-attention; temporal models add frame-axis rotary embeddings.Source: §A.2 · p. 18

A.3 INFERENCE-ALIGNED MULTIMODAL CONDITIONING

• Only requested output latents form the noisy ODE state; clean reference evidence, camera parameters, and text remain fixed as inference-available conditioning signals.Reference and camera signals use complementary attention pathways, while text provides global semantic context.Source: §A.3 · p. 18
• Reference views are encoded jointly using only observed references as context, avoiding target-set leakage and a train–test mismatch from unavailable target views.Geometry encoders are set-based.Source: §A.3 · p. 18
• Reference tokens are contextual and solver-invariant, while every output slot—including one at a reference camera—remains an ordinary noisy flow variable.Tokens use t = 0, are prepended to noisy targets, and are excluded from the prediction head; this avoids clamping and preserves the same ODE parameterization for conditional and unconditional sampling.Source: §A.3 · p. 18
• World-space Plücker camera rays condition query/key self-attention at the latent token grid, spatially relating observed evidence to requested views.Ray channels represent viewing direction, ray-moment orientation, and—when metric camera sidecars are available—metric translation magnitude; rays are assigned from observed or requested cameras respectively.Source: §A.3 · p. 18
• Frozen-language-model text enters cross-attention as global context, while independent condition dropout supports text-only, camera-controlled, reference-conditioned, or jointly conditioned generation and classifier-free guidance.The same model supports image and video generation regimes.Source: §A.3 · p. 18

A.4 TWO-STAGE TRAINING AND SAMPLING

• Stage 2 freezes all Stage 1 modules and trains only the conditional flow model; sampling integrates its velocity field from t = 1 to t = 0.Requested output slots begin as Gaussian noise, reference images are encoded once as clean conditions, and resulting latents are denormalized and decoded.Source: §A.4 · pp. 18–19
• Controlled comparisons hold the DDT-style flow, data mixture, optimization schedule, and sampling protocol fixed, varying only the latent backend and shape-dependent projections.Training uses nine 252×252 views from an equal RealEstate10K and DL3DV mixture, with one to four reference views; inference uses nine views, one clean reference, 50 Euler steps, and CFG scale 2.Source: §A.4 · p. 19
• The separate qualitative final model uses a frozen 672×378 DA3-GIANT space–time codec and trains on 81-view clips with one to four prefix references.It uses a 0.93B-parameter trainable flow trained on 80 GPUs; showcased outputs sample all 81 views jointly from one reference using 50 Euler steps and CFG scale 2.Source: §A.4 · p. 19

A.5 CONTROLLED EVALUATION PROTOCOL

• Evaluation uses shared held-out pools of 64 scenes from RealEstate10K and DL3DV, with nine views per example, joint target sampling, 50 Euler steps, and CFG = 2.FID and FVD measure frame- and clip-level distributional quality; LPIPS, PSNR, and SSIM compare paired target frames along the prescribed camera trajectory. Blue cells rank seven latent variants trained with the shared flow recipe; GLD and Gen3R are external system-level references.Source: §A.5 · p. 19
• Camera trajectories are reconstructed from complete real and generated nine-view clips, aligned by Umeyama Sim(3), then scored with ATE, RPEt, and RPEr.VGGT is independent of evaluated latent backbones; DA3-GIANT provides a backbone-overlap cross-check.Source: §A.5 · p. 19
• Reproj@50 measures normalized round-trip pixel error for the top-confidence half of points, while MEt3R (Asim et al., 2025) measures symmetric cross-view feature disagreement; both are lower-is-better.Reprojection uses recovered world points aligned to the dataset camera frame (Jang et al., 2026); MEt3R renders upsampled features across each generated-view pair with visibility and z-buffer occlusion masks.Source: §A.5 · p. 20
• Native depth and point maps decoded from each latent are compared with Pi3, using per-view median-scale depth alignment and camera-trajectory Sim(3) alignment for point clouds.Camera ATE and RPEr are measured against the input dataset trajectory; this native readout is distinct from post-hoc RGB reconstruction used for the consistency table.Source: §A.5 · p. 20

B LATENT-SPACE DIAGNOSTIC DEFINITIONS

• The diagnostic suite covers seven properties, with xLNC∗ extending latent-neighbor consistency across views using dense correspondences.Transport and semantic-neighborhood diagnostics follow Zhong et al. (2026a), while LDS and SRSS follow Singh et al. (2025).Source: B LATENT-SPACE DIAGNOSTIC DEFINITIONS · p. 20
• Evaluation uses ImageNet-S at resolution 256 for single-view measurements and groups views from each short scene clip for multi-view measurements.Spatial neighborhoods are mapped to image coordinates before pair sampling, tokens are ℓ2-normalized, and diagnostics use the raw flow-model representation without diagnostic-only PCA removal or feature centering.Source: B LATENT-SPACE DIAGNOSTIC DEFINITIONS · p. 20
• Lower within-group velocity fraction ρ indicates less ambiguous transport for samples sharing a class or scene.It compares within-class covariance for single-view evaluation, or within-scene-clip covariance for multi-view evaluation, with total covariance; the trace ratio is dimensionless and cannot be improved by appending inactive channels.Source: B LATENT-SPACE DIAGNOSTIC DEFINITIONS · p. 20
• Covariance condition number κ flags anisotropy or near-rank deficiency, while effective rank estimates how many latent directions carry substantial variance.The two spectral statistics distinguish nominal channel count from effective participation in the covariance spectrum.Source: B LATENT-SPACE DIAGNOSTIC DEFINITIONS · p. 20
• LNC@5 measures local semantic organization by the fraction of an ImageNet-S descriptor’s five nearest neighbors that share its label.Descriptors are foreground-pooled and compared by cosine similarity; this metric does not measure reconstruction fidelity or patch-level geometry.Source: B LATENT-SPACE DIAGNOSTIC DEFINITIONS · p. 21
• LDS and SRSS assess distinct within-image structure: spatial similarity decay and foreground-region coherence against distant background content.LDS compares local and far token-pair cosine similarity using common image coordinates (approximately 24 and 96 pixels). SRSS compares nearby foreground positives with spatially distant background negatives.Source: B LATENT-SPACE DIAGNOSTIC DEFINITIONS · p. 21
• xLNC∗ chance-normalizes cross-view retrieval accuracy, where 0 is chance and 1 is perfect correspondence.UFM-Refine matches view tokens; retrieval is correct within the matched target neighborhood, and the main table uses pairs with sufficient visual overlap. Unlike LDS and SRSS, it tests same-scene retrieval across viewpoint changes.Source: B LATENT-SPACE DIAGNOSTIC DEFINITIONS · p. 21

C.1 INDEPENDENT 3D CHECK

• Figure 9 compares appearance-native, semantic-native, and geometry-native generated states across two scenes.The comparison includes generated views and independently reconstructed point clouds and camera trajectories.Source: §C.1 · p. 21
• An independent reconstruction pipeline recovers a point cloud and camera trajectory from each generated output clip.This provides a check of whether the generated frames jointly support a coherent 3D scene and camera path.Source: §C.1 · p. 21
• The reconstruction check separates visual plausibility from 3D-scene and camera-path coherence.Source: §C.1 · p. 21

C.2 RGB COMPARISONS

• Displayed PSNR is computed over the eight generated views of that scene, not averaged across the dataset.Source: §C.2 · p. 21
• Figures 10 and 11 compare methods under the same reference image and target camera trajectory, jointly generating the eight remaining views.The examples show one scene from each dataset's shared 64-scene evaluation pool.Source: §C.2 · p. 21, Figure 10 · p. 22, Figure 11 · p. 23