PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Spherical Occupancy Profiles for Multi-View 3D Reconstruction and Generation

YiHsuan Tsai

arXiv:2608.23206v1cs.CV

TL;DR

The paper asks whether spherical occupancy profiles can unify discriminative and generative 3D reconstruction while representing image-conditioned uncertainty. It distills these profiles from multi-view Gaussian fields, decodes them discriminatively, and models them generatively, achieving strong depth reconstruction, tunable multi-solution sampling, morphology correction, and real-photo front-end transfer.

  • Problem

    Existing discriminative and generative 3D reconstruction methods use varied intermediate representations, motivating a unified interface between image evidence and final shape.

  • Method

    The paper uses spherical occupancy profiles P(r) = T(r) o(r), distilled from multi-view Gaussian fields, with a per-ray FiLM decoder and a VAE plus latent diffusion generative branch.

  • Results

    The unified representation supports median soft-depth error 0.035–0.036 on an independent test split, best-of-N generative samples matching discriminative accuracy, tunable solution spread, morphology correction, and normalized Chamfer 0.010–0.011 on DTU photographs.

  • Takeaways & Limitations

    Ray-wise occupancy profiles provide a compact, interpretable, uncertainty-aware interface serving both reconstruction and generation.

  • Takeaways & Limitations

    Image-conditioned generation was trained and evaluated only on synthetic turntable captures, while the profile representation is center-anchored and not suited to open scenes without extension.

Abstract

from arXiv · show

We study spherical occupancy profiles-the ray-wise occupancy probability profiles P(r) = T(r) o(r) distilled from multi-view 3D Gaussian reconstructions-as a unified intermediate representation for both discriminative and generative 3D reconstruction from images. On a 999-object subset of Google Scanned Objects with 48 turntable views each, we train (i) a discriminative per-ray decoder that injects global view-averaged and ray-specific image evidence into a FiLM-conditioned profile head, reaching median soft depth error 0.035 (normalized) on an independent 90-object test split, and (ii) a generative pipeline built on a profile VAE and a latent diffusion model, which supports unconditional sampling that matches the reconstruction manifold and image-conditioned multi-solution reconstruction whose per-object solution spread is quantifiable and tunable via classifier-free guidance. We further analyze the morphology of predicted profiles: post-hoc power sharpening and a learned sharpening target both recover ground-truth profile width without degrading depth, exposing a monotonic width-peak frontier in the L1-per-ray loss family and motivating a principled redefinition of morphology gates. Real-photo validation on two DTU scenes confirms the pipeline transfers to non-synthetic input. Our results suggest that ray-wise occupancy profiles offer a compact, learned, and uncertainty-aware interface between multi-view reconstruction and generative priors.

1 Introduction

The paper introduces spherical occupancy profiles as a shared, volume-distilled interface for discriminative and generative multi-view 3D reconstruction. Its two branches achieve accurate single predictions, controllable multi-solution reconstruction, morphology correction, and transfer to real DTU photographs.

  • Representation: Spherical occupancy profiles provide a compact, interpretable interface between multi-view image evidence and 3D reconstruction or generation.Their peak position encodes depth, width captures volumetric softness, and peak height provides per-ray confidence.
  • Discriminative reconstruction: The discriminative per-ray decoder reaches median soft depth error 0.035 on an independent 90-object test split, about 60% better than the mean-profile baseline.It combines view-averaged global features with ray-specific image evidence through FiLM conditioning.
  • Generative reconstruction: The generative branch uses a profile VAE and latent diffusion to support unconditional sampling and image-conditioned multi-solution reconstruction.Per-object solution spread is quantifiable at about 18% of inter-object scale and tunable through classifier-free guidance.
  • Morphology: Post-hoc power sharpening and sharpened-target training recover ground-truth profile width without degrading depth, revealing a monotonic width–peak frontier in the L1-per-ray family.These findings motivate separating normalized shape and confidence axes when redefining morphology gates.
  • Real-photo validation: The full image-to-point-cloud pipeline transfers to non-synthetic input in validation on two real DTU scenes.The representation and fixed front-end were evaluated on real photographs.

2 Related Work

Related work spans feed-forward reconstruction, volumetric and occupancy representations, diffusion-based 3D generation, and multi-solution uncertainty modeling. The paper positions profile-domain latent diffusion as an analogous approach that additionally offers continuous control over solution spread.

  • Feed-forward 3D reconstruction: Feed-forward reconstruction predicts a 3D representation from one or a few images in a single forward pass but commonly commits to one reconstruction.Recent methods partially address ambiguity using hallucination-aware diffusion priors.
  • Neural fields, Gaussians, and occupancy representations: Gaussian Opacity Fields enable ray-wise occupancy profiles through analytic ray marching of view-independent volumetric opacity.The resulting profile is the transmittance-weighted occupancy P(r) = T(r) o(r).
  • Diffusion-based 3D generation: Diffusion-based 3D generation includes score distillation, image-conditioned view diffusion, consistent image-stack generation, and depth-aware per-view methods.The paper instead diffuses the latent space of a profile VAE so samples remain on the profile reconstruction manifold.
  • Multi-solution reconstruction and uncertainty: Multi-solution reconstruction methods sample from a posterior over 3D and can use sample variance as an uncertainty map.The paper provides an analogous profile-domain capability and uses classifier-free guidance as a continuous control over solution spread.

3 Background

The paper distills ray-wise transmittance-weighted opacity from Gaussian Opacity Fields into spherical profiles centered on each object. These profiles preserve radial depth and opacity information while supporting a mesh-free dataset pipeline from multi-view captures.

  • Volumetric scene representation: A 3D Gaussian scene represents geometry and appearance with anisotropic Gaussians fitted to multi-view images through differentiable alpha-blended rendering.Each Gaussian has a mean, covariance, opacity, and view-dependent color.
  • Gaussian Opacity Fields: Gaussian Opacity Fields replace directional opacity with view-independent opacity integrated over a volumetric kernel, enabling analytic ray marching.Transmittance and per-point opacity can be evaluated without Monte Carlo sampling.
  • Spherical occupancy profiles: A spherical occupancy profile is the ray-wise product P(r) = T(r) o(r), describing where the surface lies along each direction and with what confidence.Its radial integrals reproduce volume-rendered ray integrals, preserving full depth and opacity information along each ray.
  • Profile distillation: Profiles are distilled along 64 × 128 Fibonacci-lattice directions with 96 radial bins and normalized maximum range approximately 2.25.The resulting radial bin width is approximately 0.024 in normalized units.
  • Dataset and supervision: The corpus contains 999 Google Scanned Objects rendered with 48 turntable views each and split into 819 training, 90 validation, and 90 held-out test objects.Random saturated backgrounds are used during training to improve opacity hygiene.

4 Method

The pipeline uses spherical occupancy profiles as a shared interface for direct and generative reconstruction, combining image-conditioned decoding with latent generative modeling.

  • Unified representation: Both branches map multi-view images through the same volume-distilled spherical occupancy profile representation.The discriminative branch predicts profile fields directly, while the generative branch operates in the latent space of a profile VAE.
  • Discriminative reconstruction: The global image condition is a 512-dimensional mean-pooled ResNet18 feature across K=8 input views.The shared encoder produces multi-scale feature maps, and the global vector is obtained by averaging view features.
  • Discriminative reconstruction: The per-ray decoder predicts each 96-bin profile on a 64 × 128 ray grid using positional ray embeddings and FiLM modulation driven by the global image vector.Three FiLM blocks apply per-layer affine transformations before a linear head and sigmoid output.
  • Discriminative reconstruction: Ray-specific image evidence is added by projecting reference points for each ray into all views and bilinearly sampling multi-scale image features.The resulting pathway augments global conditioning with evidence tied to individual ray directions.
  • Profile sharpening: Sharpened-target training applies a per-ray power transform to the max-normalized ground truth as a drop-in training change without modifying the decoder.This addresses profile-width behavior through the supervision target rather than architecture.
  • Generative reconstruction: The generative branch combines a profile VAE with latent diffusion, image conditioning, and classifier-free guidance for tunable multi-solution sampling.The VAE encodes profile fields, while diffusion samples latent profiles conditioned on a global image vector; decoded samples quantify solution spread.
  • Evaluation: Surface depth is read from each decoded profile using either its centroid or its argmax bin, while Chamfer distance evaluates the resulting point cloud.Profile width is separately measured by the predicted-to-ground-truth FWHM ratio.

5 Experiments

Experiments evaluate discriminative accuracy, profile morphology, generative sampling, and transfer to real DTU photographs, showing improved depth, controllable profile shape, and multi-solution behavior.

  • Experimental setup: 819/90/90 GSO objects form the training, validation, and held-out test split, with K=8 views used for discriminative models.Evaluation uses fixed turntable frames, while training views are randomly sampled per object.
  • Discriminative reconstruction: 0.0338 val / 0.0384 test soft depth error is achieved by d8, compared with 0.084 for the mean-profile baseline.Increasing training data from 400 to 819 objects reduces validation error from 0.0426 to 0.0338.
  • Discriminative reconstruction: −7.3% test improvement is obtained by adding ray-specific evidence, while sharpened-target v2 γ=2 reaches 0.0348 val / 0.0356 test and width ratio 1.0.The sharpening target makes profiles narrower without reducing accuracy.
  • Discriminative reconstruction: Figure 2 shows d8 profiles as wider and lower than ground truth, whereas per-ray conditioning sharpens them and γ=2 sharpening recovers ground-truth width on covered rays.The comparison uses three covered rays per object selected at the 25th, 50th, and 75th peak-bin percentiles.
  • Profile morphology: Table 2 reports FWHM as predicted-to-ground-truth width ratio and peak as maximum occupancy, with soft-depth error as a reference.Ground truth has FWHM 1.0 and peak approximately 1.
  • Profile morphology: The morphology frontier shows that increasing sharpening monotonically decreases width and peak while improving soft-depth error from 0.0364 to 0.0348.Both learned and post-hoc sharpening follow this trajectory, making the original joint gate unattainable.
  • Generative reconstruction: 7.0 decoded-sample width matches reconstruction width, while sample pairwise Chamfer is 0.097 versus 0.150 for the inter-object ground-truth scale.Naive Gaussian prior sampling instead yields width 13 and pairwise Chamfer 0.41.
  • Generative reconstruction: 0.0337 soft-depth error is reached by image-conditioned diffusion with best-of-8 sampling and CFG weight w=8, matching the discriminative level of d8.A single conditional sample reaches 0.0489, while best-of-8 reaches 0.0386 conditional and 0.0383 unconditional.

6 Discussion

The morphology analysis identifies a monotonic width–peak frontier caused by the L1-per-ray objective and supports separating shape and confidence evaluation. The discussion also positions discriminative and generative branches as complementary while documenting dataset, input, and representation limits.

  • Morphology analysis: Post-hoc or learned sharpening removes over-width profiles without architectural changes, showing the width gap is objective-induced rather than an information limit.The analysis links sharpening to a revised morphology gate that separates normalized width from confidence.
  • Discriminative vs. generative reconstruction: The discriminative branch is the more accurate single predictor, whereas the generative branch adds quantifiable ambiguity and can match discriminative accuracy through best-of-N sampling with strong guidance.Conditional solution spread is reported as 18% of inter-object scale.
  • Limitations: The study is limited by its 999-object turntable corpus, synthetic-only training and evaluation for image-conditioned generation, center-anchored profiles, and front-end-only DTU validation.The authors identify larger multi-pose data, coarse-to-fine conditioning, real-photo generation training, and per-view ray grids as next steps.
  • Representation: The profile representation preserves both surface and confidence information and can be re-integrated into volume rendering, distinguishing it from view-domain and tri-plane alternatives.Peak position yields a surface point, peak height provides per-ray confidence, and the field remains volumetrically usable.

7 Conclusion

The paper presents spherical occupancy profiles as a unified representation for reconstruction and generation, with demonstrated discriminative, generative, morphology, and real-photo results. It concludes that this representation is compact, interpretable, and uncertainty-aware, while identifying scaling and real-photo training as next steps.

  • A per-ray FiLM decoder reaches median soft-depth error 0.035–0.036 on an independent test split while remaining robust to ray-conditioning bottleneck width.
  • Profile VAE and latent diffusion support unconditional manifold-matching samples and image-conditioned multi-solution reconstruction with quantifiable, guidance-tunable spread.
  • Morphology analysis identifies predicted profile width as an L1-objective artifact addressable through learned sharpening or post-hoc power transforms, motivating revised morphology gates.
  • The fixed image-to-point-cloud front-end transfers to real DTU photographs with normalized Chamfer 0.010–0.011.
  • The authors characterize ray-wise occupancy profiles as compact, interpretable, and uncertainty-aware, and propose corpus scaling, depth-guided conditioning, and real-photo training as next steps.
Loading 2608.23206v1…