PaperlayerView exact revision on arXiv

Source-linked AI summary

GenIA: Generative Reconstruction with Test-Time Input Alignment

Stefano Esposito, Naama Pearl, Polina Karpikova, Samuel Rota Bulò, Lorenzo Porzi, Peter Kontschieder, Andreas Geiger, Jonathon Luiten

arXiv:2610.12388v1cs.CV

TL;DR

Complete 3D reconstruction from sparse observations remains challenging because generative models may not faithfully match observed geometry, appearance, or pose. GenIA grounds a frozen generative prior in observations at inference time, improving reconstruction and pose across static and dynamic settings and outperforming several recent method classes.

  • Problem

    Complete 3D assets must be recovered from monocular or sparse multi-view observations, but generative models may fail to faithfully reproduce observed geometry, appearance, or pose.

    Source: §1 · pp. 1–2

  • Method

    GenIA aligns SAM3D’s frozen generative prior to geometric and photometric observations at inference time, using geometry-grounded placement and visibility- and rendering-guided appearance alignment.

    Source: §1 · p. 2

  • Results

    GenIA improves pose and reconstruction across monocular, multi-view, and dynamic inputs, outperforming optimization-based, per-frame image-to-3D, and video-to-4D methods on real and synthetic benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    With externally supplied dynamic geometry, GenIA produces temporally coherent reconstructions and accurate tracks while improving static monocular and multi-view pose and novel-view quality.

    Source: §6 · p. 8

    Dynamic geometry currently relies on an external predictor, and coarse shape is not directly grounded in observed silhouette and depth.

    Source: §6 · p. 8

Abstract

from arXiv · show

Reconstructing complete 3D object assets from monocular or sparse multi-view observations remains challenging. Generative 3D foundation models can complete object geometry beyond the observed views, but their predictions may not faithfully reproduce the observed geometry, appearance, or pose. We introduce GenIA, a framework for test-time input-aligned generation that grounds SAM3D's generative prior in geometric and photometric observations without retraining the foundation model. We improve object pose by deriving translation and scale from geometry while retaining the learned rotation prior, and align appearance through visibility-biased attention, cross-observation fusion, and differentiable rendering guidance during denoising. An optional post-denoising refinement further adapts the appearance latent, lightweight decoder adapters, and object placement to the observations. Our framework also supports externally supplied geometry; when given temporal shapes of dynamic objects, it recovers a shared, input-aligned canonical appearance and stable world-space placement. Across synthetic and real benchmarks, GenIA improves pose prediction and object reconstruction from monocular, multi-view, and dynamic inputs, outperforming recent optimization-based, per-frame image-to-3D, and video-to-4D methods. Our project page is available at https://facebookresearch.github.io/GenIA.

1. Introduction

Generative 3D models can complete unseen regions, but may not faithfully match observed geometry, appearance, or pose. GenIA aligns the generative prior to observations at inference time, without retraining, across monocular, multi-view, and dynamic inputs.

1. Introduction

  • The central challenge is combining generative priors that complete unseen regions with reconstruction constraints that reproduce observed evidence faithfully.Source: §1 · p. 2
  • Prior approaches trade off input fidelity and completion, while retrained alignment methods specialize to particular input settings and do not address dynamic objects.SAM3D may miss observed details; pixel-aligned predictors offer weaker unseen-region completion and typically lack explicit pose.Source: §1 · p. 2
  • GenIA aligns a generative prior to geometric and photometric observations at inference time, without retraining, across monocular, multi-view, and dynamic inputs.Observed regions are constrained while generation completes what remains unseen.Source: §1 · p. 2
  • GenIA retains the learned rotation prior while deriving translation and scale from depth, and aligns appearance through visibility-aware conditioning, cross-observation fusion, and differentiable rendering.Optional post-denoising refinement jointly optimizes appearance latent, lightweight decoder adapters, and object placement.Source: §1 · p. 2
  • GenIA improves pose and object reconstruction for monocular and multi-view inputs, produces temporally consistent dynamic reconstructions, and outperforms optimization-based, per-frame image-to-3D, and recent video-to-4D methods on real and synthetic benchmarks.Source: §1 · p. 2

2. Related work

Prior reconstruction and generation methods variously emphasize observed imagery, complete objects, or temporal modeling. GenIA instead grounds pose and appearance at inference time on a frozen generator and uses existing coherent geometry for dynamic inputs.

2.1. 3D reconstruction and generation

  • Observed-image-faithful reconstruction methods recover only visible geometry, while object-centric generation aims to infer complete geometry and appearance from text or sparse observations.Pixel-aligned Pixal3D [28] contrasts with methods that generate canonical-space assets; ReconViaGen [3] and CUPID [15] learn stronger input alignment during training.Source: §2.1 · p. 2
  • ReconViaGen [3] and CUPID [15] incorporate alignment during training, whereas GenIA grounds pose and appearance at inference time on a frozen generator across monocular, multi-view, and dynamic inputs.Source: §2.1 · p. 3

2.2. 4D reconstruction and generation

  • Dynamic reconstruction methods include per-sequence optimization and direct 4D prediction, but recover geometry only for surfaces observed in the input, leaving unseen regions incomplete.Optimization-based methods are costly per sequence.Source: §2.2 · p. 3
  • Video-to-4D approaches use learned priors or spatio-temporal latent spaces, while GenIA leverages temporally coherent geometry from existing backbones and recovers observation-grounded appearance and stable world-space trajectories.This avoids generating 4D geometry itself and does not require dataset-level retraining.Source: §2.2 · p. 3

3. Preliminaries

The preliminaries define the observation inputs and SAM3D representations that GenIA builds on. SAM3D predicts coarse shape and pose before generating a structured latent for fine geometry and appearance.

3. Preliminaries

  • Each observation consists of an image, object mask, depth map, camera intrinsics, and camera extrinsics.Source: §3 · p. 3

3.1. SAM3D

  • SAM3D predicts coarse occupancy, appearance, and rigid pose from a single observation, using monocularly estimated depth and intrinsics to form a conditioning pointmap.Its camera extrinsics are assumed to be identity, so camera and world space coincide.Source: §3.1 · p. 3
  • SAM3D uses two denoising stages: the first predicts coarse shape and pose, and the second predicts a structured latent encoding fine geometry and appearance.Both stages use 25-step rectified conditional flow matching; classifier-free guidance extrapolates between conditioned and condition-free velocity predictions.Source: §3.1 · p. 3
  • The coarse stage jointly denoises shape and pose, while the fine stage denoises SLAT over active voxels of the coarse occupancy and decodes it to 3D Gaussians.Source: §3.1 · p. 3
  • The pipeline obtains or injects coarse shape, denoises per-frame pose, predicts observation-aligned SLAT appearance, and optionally refines the reconstruction against input views.The figure illustrates single-view input; multi-view and dynamic extensions are described in the text.Source: Figure 2 · p. 4

4. Method

GenIA aligns SAM3D’s frozen generative prior to geometric and photometric observations at inference time, supporting monocular, multi-view, and dynamic inputs. It combines geometry-grounded placement, visibility- and rendering-guided appearance generation, and optional test-time refinement.

4. Method

  • GenIA operates at inference time without updating pretrained SAM3D weights, and its grounding framework supports geometrically consistent pointmaps and camera parameters from different geometry predictors.Optional refinement updates only object-specific appearance and pose variables.Source: §4 · p. 4

4.1. Optional Shape Injection

  • With external geometry, GenIA voxelizes the supplied shape, inverts it into SAM3D’s occupancy latent, and keeps the shape fixed while denoising pose.For dynamic inputs, the supplied temporal shapes also specify a deformation field relating each frame to the canonical shape.Source: §4.1 · p. 4

4.2. Pose prediction

  • Depth-derived object center, median depth, and angular size guide five fixed-point updates to translation and scale, while the denoised rotation is retained.Multi-view translation and scale are mapped to world coordinates and aggregated into a shared pose; dynamic objects retain frame-specific placement.Source: §4.2 · pp. 4–5

4.3. Appearance prediction

  • At each denoising step, GenIA biases attention toward patches observing each voxel, applies rendering guidance, and fuses per-observation velocities using visibility-based weights.For dynamic inputs, frame predictions are mapped to a canonical grid and fused using the same visibility-weighted rule.Source: §4.3 · pp. 5–6

4.4. Optional Test-time refinement

  • Optional test-time refinement jointly optimizes the canonical latent, a low-rank decoder adapter, and world-space poses while keeping pretrained parameters fixed.The latent anchor preserves completion of unobserved regions, while decoder adaptation provides instance-specific capacity beyond the frozen decoder.Source: §4.4 · p. 6

5. Results

GenIA aligns a frozen generative model to observations across static and dynamic reconstruction settings. Its results balance input fidelity with completion of unseen regions, while runtime, motion, and memory impose practical trade-offs.

5.1. Analysis

  • On dynamic sequences, GenIA costs 13× SAM3D’s single-image runtime including ActionMesh, versus 160× for Lift4D and 250× for HiMoR.Figure 6 plots median runtime relative to SAM3D against mean CLIP-I; TTR adds a largely view-independent cost while improving CLIP-I.Source: Figure 6 · p. 8
  • On single-view inputs, GenIA better aligns appearance than generative baselines and preserves backside geometry, while pixel-aligned methods can degrade on unseen surfaces.Figure 5 shows GenIA with TTR completing unseen surfaces with colors consistent with the input; TripoSplat and CUPID show washed-out colors or appearance drift.Source: §5.1 · pp. 7–8, Figure 5 · p. 7
  • Without TTR, GenIA outperforms all multi-view baselines, including ReconViaGen; TTR further improves fidelity and perceptual quality.Depth-3DGS leads training-view PSNR and co-PSNR but remains incomplete in unobserved regions.Source: §5.1 · p. 8
  • On dynamic scenes, GenIA produces coherent shape and appearance; its DAVIS results rank second to HiMoR on training-view PSNR and δavg, but lead in reprojection error and CLIP-I.On ActionBench, it achieves the best held-out LPIPS, while HiMoR leads training-view PSNR and co-PSNR.Source: §5.1 · p. 8
  • Large inter-frame motion can substantially degrade reconstruction when coarse-geometry or pose errors propagate; with TTR, peak CO3D one-view memory reaches 31.8 GB.Without TTR, peak memory is 19.7 GB, versus SAM3D’s 18.0 GB.Source: §5.1 · p. 8

5.2. Ablations

  • The appearance ablation adds visibility attention bias, rendering guidance, and TTR cumulatively, progressively improving training- and test-view quality on a static monocular GSO scene.Ground-truth coarse geometry isolates the appearance components; Figure 7 shows train- and held-out-view PSNR and co-PSNR.Source: §5.2 · p. 8, Figure 7 · p. 8

6. Conclusion

GenIA aligns a frozen generative 3D prior to observations and improves static pose and novel-view quality. Dynamic reconstructions are temporally coherent with external geometry, but shape grounding and pose correction remain limited.

6. Conclusion

  • GenIA aligns a frozen generative 3D prior to observations through depth-grounded translation and scale, visibility-aware appearance conditioning, and rendering guidance.It also uses cross-observation fusion and supports externally supplied shapes.Source: §6 · p. 8
  • GenIA improves pose and novel-view quality for static monocular and multi-view inputs and yields temporally coherent reconstructions and accurate tracks with external dynamic geometry.Source: §6 · p. 8
  • Dynamic geometry still relies on an external predictor, coarse shape is not directly grounded in observed silhouette and depth, and rotation is only weakly corrected by noise-sensitive registration.Pose errors from masks, depth, and cameras can also degrade differentiable-rendering supervision.Source: §6 · p. 8

A.1. Extended ablations

Extended ablations examine appearance components and cumulative pose components across static and dynamic settings. Depth-grounded translation and scale provide the largest reported dynamic pose gains, while ICP’s effects vary by dataset.

A.1. Extended ablations

  • The appearance ablation cumulatively adds visibility attention bias, rendering guidance, and test-time refinement to SAM3D’s appearance prediction.Quantitative results are reported in Tables 3a and 3b, with qualitative comparisons in Figures 8 to 11.Source: §A.1 · p. 13
  • Depth-grounded translation and scale reduce dynamic reprojection error by 1.4–2.8× and relative jitter by 2–4×, while also improving static-view PSNR.Rotation averaging leaves DAVIS essentially unchanged and modestly reduces ActionBench reprojection error and jitter.Source: §A.1 · p. 13
  • ICP halves Chamfer distance for static reconstruction, but on deforming geometry its effects differ: it worsens DAVIS appearance and reprojection error while improving ActionBench PSNR.The authors attribute the DAVIS outcome as likely reflecting noise in MapAnything depth and pose predictions.Source: §A.1 · p. 13

A.2. Additional results

The appendix adds qualitative comparisons across benchmarks and shows two representative dynamic-scene failure cases.

A.2. Additional results

  • Figures 12–15 provide qualitative benchmark comparisons, while Figure 16 presents two representative dynamic-scene failure cases.Source: §A.2 · p. 13

A.3. Dynamic shape injection

Dynamic shape injection uses fixed-topology temporal meshes to establish canonical correspondences, derive deformation, and fuse frame-wise appearance. The geometry is held fixed while SAM3D denoises pose when external shapes are supplied.

A.3. Dynamic shape injection

  • Fixed-topology meshes index the same point across frames; their shared topology supplies deformation and coarse-geometry correspondences for appearance fusion.Each deformed shape is inverted independently.Source: §A.3 · p. 13
  • Kabsch aligns each canonical vertex’s eight-neighbor set to its deformed counterpart, and the resulting rotations plus vertex positions define a deformation field that is identity at the canonical frame.The field warps canonical Gaussian centers with vertex motion and covariances with local rotations.Source: §A.3 · p. 13
  • At every ODE step, per-frame appearance predictions map to the canonical grid, where visibility masks collisions before predictions are averaged.A canonical voxel receives no contribution from a frame if no voxel maps to it; the canonical frame always reaches it through the identity correspondence.Source: §A.3 · p. 13
  • With injected shape latents, SAM3D fixes the target geometry and denoises pose using an interpolated shape token to preserve the training-time trajectory.The intermediate token combines the fixed clean shape latent with a fixed Gaussian-noise latent.Source: §A.3 · p. 13

A.4. Voxel-to-patch mapping

Voxel footprints are mapped into conditioning-stream patch grids using each stream’s image transformation, then expanded to account for projection quantization.

A.4. Voxel-to-patch mapping

  • For each conditioning stream, a voxel’s pixel footprint is transformed using that stream’s crop, padding, and resizing operations before mapping to the 37×37 DINOv2 patch grid.Cropped streams use a mask-bounding-box crop; full-image streams use pad-to-square.Source: §A.4 · p. 13
  • Each voxel’s resulting patch set is dilated by one patch to account for projection quantization.Source: §A.4 · p. 13
  • In dynamic appearance fusion, a canonical voxel receives no contribution from a frame if that frame maps no voxel to it; the canonical frame always reaches it.The canonical-frame correspondence is the identity.Source: §A.4 · p. 13

A.5. Passive-stream compensation

The appendix details passive-stream compensation, rendering-loss safeguards, and test-time refinement alongside evaluation protocols and metric caveats. It also specifies baseline alignment and tracking procedures used to interpret comparisons.

A.5. Passive-stream compensation

  • Pose ablations find ICP most beneficial for static multi-view alignment, halving Chamfer distance, while on DAVIS it improves depth at the cost of appearance quality.The appearance ablation table compares the Attn, RG, and TTR component ladder; GSO-30 and ActionBench share ground-truth shape and pose, while CO3D and DAVIS use predicted geometry.Source: Table 2 · p. 14, Table 3 · p. 14
  • OFF, APPROX, and EXACT yield nearly identical reconstruction metrics on GSO-30 and CO3D; APPROX adds essentially no cost, while EXACT incurs 1.3–1.8× appearance-block overhead.The closed-form offset approximately restores mask and global-token attention share while preserving the visible-patch boost.Source: §A.5 · pp. 13–14, Figure 18 · p. 22

A.6. Rendering loss details

  • Rendering losses composite observations with a resampled background, and per-Gaussian stop-gradients restrict rendering gradients to directly observed geometry.Without masking, semitransparent unseen primitives can receive unconstrained gradients that push surfaces away from the generative prior, producing holes and floaters; an opacity anchor penalizes drops below initial opacity to preserve solid unseen surfaces.Source: §A.6 · pp. 14–16

A.7. Test-time refinement details

  • Test-time refinement optimizes the canonical latent, Gaussian-decoder adapter, and pose for 100 iterations per object, while keeping the SAM3D model and mesh decoder frozen.It samples one observation per iteration; scale is not refined, and the rank-4 DoRA adapter starts at the unadapted decoder prediction.Source: §A.7 · pp. 16–17

A.8. Depth-3DGS baseline

  • Depth-3DGS is a prior-free per-scene baseline initialized from foreground depth pixels and optimized for 1,000 iterations with photometric and inverse-depth losses.It uses the same cameras and depth as GenIA, with degree-0 spherical harmonics for view-independent color.Source: §A.8 · p. 18

A.9. ICP registration

  • ICP aligns the coarse shape to the metric input pointmap with one similarity transform per object, composed with every frame’s pose to absorb systematic alignment error while preserving motion.It uses one-directional nearest-neighbor matching from observed to rendered points, which handles occlusion and avoids comparing unrelated surfaces under small misregistration.Source: §A.9 · p. 18

A.10. Pose refinement for baselines

  • To isolate reconstruction quality from pose error, baselines are refined against inputs with shape and appearance fixed; input-camera-frame methods are exempt because they are already aligned.Methods without pose prediction receive GenIA-style translation and scale initialization, rendering-based rotation search, and the same refinement.Source: §A.10 · pp. 18–19

A.11. Pseudo-ground-truth 2D tracking

  • Cycle-consistent CoTracker3 queries remove 40–60% of tracks and more than halve reference error; tracking comparisons measure agreement with that pseudo-reference, not absolute accuracy.The initial query-to-surface distance bounds attainable error below one pixel for every method on both datasets, and the same tracks are used across methods.Source: §A.11 · pp. 19–21

Related papers

Loading 2610.12388v1…