Source-linked AI summary
Guiding Image-to-3D Generation with Test-Time Partial Observations
Jerred Chen, Simon Weber, Ronald Clark
TL;DR
Single-image image-to-3D models can generate plausible assets but leave geometry ambiguous, motivating explicit use of partial observations at test time. The paper introduces training-free ray-consistent guidance over occupancy representations, substantially improving geometric fidelity while preserving visual quality across observability levels.
Problem
Single-image image-to-3D generation leaves underlying geometry ambiguous, limiting applications that require geometric fidelity.
Method
The method guides pretrained image-to-3D models with a ray-consistent occupancy likelihood combining surface occupancy and free-space evidence, without retraining or finetuning.
Results
The approach consistently improves geometric fidelity across observability levels while preserving visual quality on SAM 3D and its multi-view extension.
Takeaways & Limitations
Pretrained image-to-3D models can integrate incomplete geometric evidence through explicit test-time guidance while retaining their learned priors for unobserved regions.
Takeaways & Limitations
The analysis assumes conditional independence across observed rays, and guidance behavior depends on the chosen conditioner and pretrained flow.
Abstract
from arXiv · showhide
Image-to-3D models can generate visually compelling 3D assets from a single RGB image, but their geometry is often only loosely constrained by the available observations, limiting their use in applications that require geometric fidelity. In many real-world settings, however, partial geometric observations of the object may be available at test time. We introduce a training-free framework for incorporating such evidence into pretrained image-to-3D generative models without retraining or finetuning. To do this, we guide generation using a ray-consistent observation likelihood defined over the model's occupancy representation, combining surface occupancy and free-space evidence. Applied to SAM 3D and its multi-view extension, our approach substantially improves geometric fidelity across different levels of observability, as well as visual quality. Our results demonstrate that pretrained image-to-3D models can effectively integrate partial geometric observations through explicit test-time guidance, complementing their learned generative priors without modifying the underlying model.
1. Introduction
Single-image image-to-3D generation can produce plausible assets while leaving unseen geometry ambiguous and inaccurate. The proposed training-free guidance uses partial geometric evidence at inference time to improve fidelity without modifying the pretrained model.
- Single RGB images leave depth, proportions, and unseen surfaces ambiguous, so generated geometry is often driven more by learned priors than true object shape.
- Partial observations can constrain visible regions while the pretrained shape prior plausibly completes unobserved geometry.
- Existing multi-view extensions improve consistency across RGB views but do not naturally accept general geometric evidence such as partial point clouds or depth.
- The method guides a pretrained model with an occupancy-grid energy combining observed-surface consistency and free-space constraints, without retraining or finetuning.
- The approach substantially improves geometric fidelity while preserving visual quality when evaluated on SAM 3D with partial observations from multiple views.
2. Related Works
Prior work developed test-time guidance for generative models and stronger image-to-3D systems, but geometric conditioning remains imperfect. This work applies training-free, geometry-aware guidance to improve consistency without retraining.
- The work combines training-free guidance with image-to-3D generation to improve geometric consistency without modifying the underlying model.
- Training-free guidance steers sampling toward desired constraints using additional objectives, extending from diffusion models to flow matching.
- Image-to-3D progress has produced high-quality assets through diffusion and flow-based models, including large-scale foundation models with structured geometry representations.
- SAM 3D can condition on geometric inputs such as point maps, but those mechanisms do not always faithfully follow the supplied geometry.
- Related partial-3D completion methods use retraining or finetuning, whereas the proposed approach guides generation without retraining.
3. Problem Formulation and Background
The paper separates a pretrained image-to-3D prior from explicit test-time geometric evidence and explains how flow matching generates samples. Its formulation treats observations through an energy that guides the learned flow while retaining the prior.
- 3.1. Problem Formulation: The problem formulation distinguishes learned conditioning, which defines the pretrained prior, from test-time evidence, which constrains generated geometry.
- 3.1. Problem Formulation: Additional observations O are incorporated through an observation model p(O | x), rather than requiring the learned conditioner to encode them faithfully.
- 3.1. Problem Formulation: The posterior formulation penalizes the negative log-likelihood of geometric observations, with β controlling evidence strength and separating the prior from the physical observation model.
- 3.1. Problem Formulation: Changing the conditioner changes the pretrained flow and therefore the effective landscape on which the same test-time observation energy acts.
- 3.1. Problem Formulation: The method applies this framework to partial geometric observations through guidance of the learned SAM 3D velocity without retraining.
- 3.2. Flow Matching and SAM 3D Generation: Flow matching transforms noise into target samples by following a learned velocity field along a probability path.
- 3.2. Flow Matching and SAM 3D Generation: SAM 3D uses two stages: a geometric model predicts coarse shape and pose, then a texture model predicts refined shape and texture decoded into meshes or Gaussian splats.
- 3.2. Flow Matching and SAM 3D Generation: At inference, SAM 3D repeatedly applies its learned velocity update from noise to the final prediction, which can be modified to satisfy desired properties.
4. Method
The method incorporates partial geometric evidence into SAM 3D at test time by deforming its pretrained flow and applying practical guidance from a ray-consistent occupancy likelihood.
- 4. Method: The framework extends SAM 3D sampling with partial geometric evidence at test time, without retraining the model.It first analyzes posterior guidance as a deformation of the pretrained conditional flow, then derives a practical point-estimate rule.
- 4.1. Guidance as a Deformation of the Flow Landscape: Test-time guidance deforms the pretrained flow landscape rather than defining a separate generative process.The deformation depends on both the conditional flow landscape and the observation landscape induced by the conditioner.
- 4.2. Guided SAM 3D: The learned SAM 3D velocity is guided by predicting a clean endpoint, evaluating observation energy, and adding its gradient to the velocity.A time-dependent guidance strength scales the observation-gradient correction before the discrete sampling update.
- 4.2. Guided SAM 3D: Guided SAM 3D corrects velocity using the observation gradient, steering generation toward a final sample whose geometry matches the partial observations.The correction combines the native velocity and guidance gradient through separate weighting factors.
- 4.3. Ray-Consistent Observation Likelihood: The observation energy uses a ray-consistent likelihood in which traversed voxels are empty and the measured surface voxel is occupied.This occupancy-grid construction represents the observed camera-ray geometry directly.
- 4.3. Ray-Consistent Observation Likelihood: The resulting guidance combines surface-hit occupancy evidence with ray-survival free-space evidence.A depth observation is probable only when the ray remains empty until the measured surface and becomes occupied there.
5. Experiments
The experiments evaluate training-free geometric guidance against image-to-3D baselines across observability levels, using geometry, rendering, and qualitative metrics. The method improves geometric fidelity while preserving visual quality, including under limited views.
- Setup: The evaluation compares SAM 3D, MV-SAM3D, and SpaceControl using Chamfer Distance, Ground-Truth Sided Distance, and LPIPS.SAM 3D and MV-SAM3D use their default settings, while SpaceControl receives modality-consistent image-to-3D inputs.
- Qualitative results: Qualitative comparisons show more accurate geometry than baselines while preserving generated appearance under high and medium observability.Figures report Gaussian splats and sided-distance heatmaps, with RGB images and partial point clouds shown as available inputs.
- Metrics: Table 1 reports geometry and rendering comparisons across input-view counts, with distance metrics scaled by 10^3.The table evaluates reconstruction accuracy using CD and SD alongside LPIPS rendering quality.
- Benchmark: The benchmark uses GSO-30 with 5, 2, or 1 selected views for high, medium, and low observability, respectively, plus projected partial RGB-D measurements.Lower-observability inputs are strict subsets of higher-observability inputs.
- Results: Across all observability levels, the method substantially surpasses SAM 3D and MV-SAM3D, outperforming all baselines in low-observability geometry and novel-view metrics.It remains geometrically competitive with SpaceControl at medium and high observability while consistently outperforming it in novel-view synthesis.
6. Ablations
The ablations identify guidance weighting, free-space weighting, sampling steps, and cooldown as important design choices affecting geometry and visual quality.
- Guidance design: Major geometric improvements require significantly upweighting the guidance gradient; additional free-space weighting and more sampling steps improve performance further.Without extra guidance weighting, performance remains on par with vanilla SAM 3D.
- Sampling and quality: More sampling steps improve geometric accuracy but can produce unsmooth surfaces, holes, or floaters and reduce visual quality.A guidance-weight cooldown schedule restores appearance and novel-view quality at some cost to geometric performance.
7. Conclusion
The paper concludes that explicit test-time guidance can integrate partial geometry into pretrained image-to-3D models while retaining learned priors for completing unseen regions.
- Conclusion: A ray-consistent occupancy likelihood combines surface occupancy and free-space evidence to steer generation toward observed geometry while preserving plausible completion of unobserved regions.The framework is training-free and applied to SAM 3D and its multi-view extension.
- Conclusion: Across observability levels, explicit geometric guidance improves geometric fidelity while preserving visual quality and complements rather than replaces pretrained generative priors.The conclusion contrasts explicit guidance with relying solely on learned conditioning pathways.
Supplementary Material
The supplementary material contains a proof of Proposition 1 and an extended set of visualizations.
- Supplementary Material: The supplement provides a proof for Proposition 1 and additional visualizations.Its organization consists of Section A for the proof and Section B for extended visual results.
A. Proof of Proposition 1
The proposition’s proof derives the guided potential from the corresponding intermediate marginal and shows that an x-independent term does not affect the velocity field.
- The corresponding intermediate marginal determines the ideal velocity used in the proof.
- The proof substitutes the intermediate marginal into Eq. (11) to obtain the exact guided potential.
- The term C_t is independent of x and therefore has no effect on the velocity field.
B. Extended Visualizations
The extended visualizations compare reconstructed objects and error maps across baselines and conditioning choices, showing stronger results from explicit guidance, especially with single- and multi-view inputs.
- Extended Visualizations: Ours-SV outperforms all other baselines on the lunch bag visualization, while SpaceControl fails to generate unobserved regions.The comparison includes Gaussian-splat fronts and backs plus bidirectional sided-error heatmaps.
- Extended Visualizations: Ours-MV further improves performance by using additional multi-view information to constrain generation.
- Extended Visualizations: Matching partial-point-cloud conditioning produces only minor performance improvements for SAM 3D and MV-SAM 3D.
- Extended Visualizations: The figures compare Gaussian-splat fronts and backs with GT-to-prediction and prediction-to-GT sided-error heatmaps for each baseline.Methods are color-coded according to their accepted image or geometric inputs.