Source-linked AI summary
AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generation
Paritosh Mittal, Yen-Chi Cheng, Maneesh Singh, Shubham Tulsiani
TL;DR
Existing 3D inference tasks often require separate systems and substantial paired data, while incomplete observations leave multiple plausible shapes. AutoSDF learns a non-sequential autoregressive prior over a discrete 3D latent grid and combines it with lightweight task-specific conditionals. The resulting framework performs competitively across completion, reconstruction, and language-guided generation, including diverse outputs.
Problem
Task-specific 3D inference systems require substantial compute and data, while completion and conditional generation must represent multiple plausible shapes from insufficient information.
Method
AutoSDF learns a non-sequential autoregressive prior over a low-dimensional discrete 3D latent representation and combines it with task-specific naive conditionals.
Results
The unified approach yields compelling results across shape completion, single-view reconstruction, and language-guided synthesis, performing competitively with specialized baselines.
Takeaways & Limitations
The prior supports diverse, high-quality completions and multiple plausible reconstructions while preserving observed structure and responding to language descriptions.
Takeaways & Limitations
The conditional formulation only approximates the joint distribution and is suboptimal with large-scale task-specific data; it also applies only to spatially structured 3D representations.
Abstract
from arXiv · showhide
Powerful priors allow us to perform inference with insufficient information. In this paper, we propose an autoregressive prior for 3D shapes to solve multimodal 3D tasks such as shape completion, reconstruction, and generation. We model the distribution over 3D shapes as a non-sequential autoregressive distribution over a discretized, low-dimensional, symbolic grid-like latent representation of 3D shapes. This enables us to represent distributions over 3D shapes conditioned on information from an arbitrary set of spatially anchored query locations and thus perform shape completion in such arbitrary settings (e.g., generating a complete chair given only a view of the back leg). We also show that the learned autoregressive prior can be leveraged for conditional tasks such as single-view reconstruction and language-based generation. This is achieved by learning task-specific naive conditionals which can be approximated by light-weight models trained on minimal paired data. We validate the effectiveness of the proposed method using both quantitative and qualitative evaluation and show that the proposed method outperforms the specialized state-of-the-art methods trained for individual tasks. The project page with code and video visualizations can be found at https://yccyenchicheng.github.io/AutoSDF/.
1. Introduction
AutoSDF addresses the resource demands of task-specific 3D inference by learning a generic shape prior that supports completion, reconstruction, and generation. Its non-sequential prior conditions on arbitrary partial observations, while lightweight task-specific conditionals extend it to paired modalities.
- Task-specific 3D systems require substantial compute and data resources for tasks such as completion and single-view prediction.
- AutoSDF treats completion and image- or language-conditioned prediction as distributions over plausible 3D structures conditioned on inputs.
- The method learns an expressive autoregressive shape prior from abundantly available raw 3D data and combines it with task-specific conditionals requiring paired data.
- Discrete, low-dimensional representations make autoregressive training tractable while preserving essential information for high-quality decoding.
- Randomized conditioning enables the prior to use arbitrary observed subsets, such as four chair legs, for shape completion.
- The unified framework combines the prior with naive task-specific conditionals and improves over task-specific state-of-the-art methods.
2. Related Work
Prior work uses autoregressive models and diverse 3D representations for generation and completion, but often relies on sequential conditioning, deterministic outputs, or specialized systems. AutoSDF applies compact discrete latent modeling and non-sequential conditioning to diverse 3D generation tasks.
- Autoregressive Modeling: Autoregressive models factorize structured-output distributions into conditional products and provide density estimation with stable training.
- Autoregressive Modeling: Prior non-sequential conditioning work enables completion but suffers from quadratic Transformer complexity when modeling low-level pixels.
- Autoregressive Modeling: AutoSDF extends quantized compact latent representations and autoregressive generation from images to 3D shapes with independently encoded symbols.
- Shape Completion: Shape-completion methods can infer plausible global shapes but may struggle with fine details, sparse inputs, or diverse samples.
- Single-view Reconstruction: Single-view reconstruction is ill-posed, and many existing approaches deterministically produce only one 3D output.
- Language-based Generation: Language-based generation is multimodal, whereas GAN-based methods can struggle to capture multiple output modes.
3. Approach
AutoSDF compresses volumetric 3D shapes into discrete latent grids, learns a Transformer prior that conditions on arbitrary latent subsets, and combines it with task-specific conditionals. This supports completion and multimodal conditional generation without directly modeling each complex conditional distribution.
- 3.1. Discretized Latent Space for 3D Shapes: The method represents 3D shapes with volumetric T-SDFs, compresses them into low-dimensional discrete latents, and models their distribution with a Transformer.
- 3.1. Discretized Latent Space for 3D Shapes: A 3D-VQ-VAE encoder and decoder map between shapes and a compact latent representation, while vector quantization selects codebook entries.
- 3.1. Discretized Latent Space for 3D Shapes: Patch-wise encoding makes latent codes depend on local regions, preserving correspondence between partial observations and partial latent variables.
- 3.2. Non-sequential Autoregressive Modeling: The latent space is a 3D token grid, reducing continuous shape-distribution learning to modeling a lower-dimensional discrete distribution.
- 3.2. Non-sequential Autoregressive Modeling: Randomly permuted latent-variable orders let the autoregressive Transformer condition on arbitrary observed locations rather than a fixed raster prefix.
- 3.3. Conditional Generation: For shape completion, partial shapes become observed latent variables, from which the model samples complete latent codes and decodes full 3D shapes.
- 3.3. Conditional Generation: For image or text conditions, the framework approximates the conditional distribution using the shape prior multiplied by independent naive conditionals.
4. Experiments
Experiments evaluate a shared autoregressive 3D shape prior on shape completion, single-view reconstruction, and language-guided generation. Across quantitative and qualitative comparisons, the approach produces diverse, plausible outputs while preserving observed structure and improving over baselines.
- Experimental scope: The method is evaluated on shape completion, 3D reconstruction, and language-guided generation using quantitative and qualitative experiments.
- Shape Completion: For shape completion, the method handles bottom-half and octant observations and is compared with MPC and PoinTr on ShapeNet.Ten complete shapes are generated for each partial input, with completion fidelity and diversity measured using UHD and TMD.
- Shape Completion: The autoregressive prior performs favorably against baselines in both fidelity and diversity across the two shape-completion protocols.
- Shape Completion: Qualitatively, generated completions are more diverse, preserve observed structures such as slanted chair legs, and remain realistic under structured partial observations.The model was trained on random observation sequences yet conditions on structured anchored observations not explicitly used for training.
- Single-view Reconstruction: For single-view reconstruction, a learned image conditioner is evaluated on ShapeNet and Pix3D against reconstruction baselines, including direct decoders and sequential variants.
- Single-view Reconstruction: The approach performs favorably across almost all reconstruction categories and generates multiple plausible shapes with meaningful variation in unobserved regions.Examples include differing chair armrests and cushions from the same back-view image.
- Language-guided Generation: In language-guided generation, the prior combines with a text-conditioned model trained from chair-description data to generate diverse shapes from descriptions.An evaluator reaches approximately 83% binary-classification accuracy; the approach is preferred over T2S 66:18 and JE 61:23.
- Language-guided Generation: Qualitative language results show plausible shapes correlated with descriptions, reasonable variation across text changes, and coherent global shapes from descriptions of specific parts.
5. Discussion
The paper presents a generic non-sequential autoregressive prior for multimodal 3D generation and reports compelling results across several tasks. It also identifies limitations concerning conditional approximation, representation type, alignment, and category coverage.
- Discussion: The unified prior is competitive with specifically designed baselines across shape completion, single-view reconstruction, and language-guided synthesis.
- Limitations: The conditional formulation only approximates the joint distribution, which is suboptimal with large-scale task-specific data despite helping in low-paired-data settings.
- Limitations: The framework applies only to spatially structured representations such as TSDFs or voxels, and adaptation to meshes or neural implicit functions remains unclear.
- Limitations: The method may be sensitive to shape alignment and is biased toward artificial categories with abundant CAD models.The authors state that it cannot be leveraged for 3D generation beyond those categories.
A.1. VQ-VAE Training
The method compresses 3D shapes into discrete latent grids, models them with a non-sequential autoregressive prior, and adds lightweight image or language conditionals for generation.
- VQ-VAE Training: P-VQ-VAE splits each 3D shape into local patches, independently encodes and vector-quantizes them, then jointly decodes the latent grid.The training objective combines reconstruction, vector quantization, and commitment losses.
- VQ-VAE Training: The P-VQ-VAE represents shapes with 512 codebook entries and latent vectors of dimensionality 256.
- Non-sequential Autoregressive Modeling: The transformer learns p(Z) by factorizing discrete latent variables in randomly permuted spatial orders rather than a fixed sequence.Its outputs are codebook distributions conditioned on previously ordered latent variables and query locations.
- Non-sequential Autoregressive Modeling: At inference, arbitrary observed latent variables are repeatedly completed by sampling, after which the decoder reconstructs the complete 3D shape.The same procedure supports unconditional generation from an empty sequence and completion from an incomplete sequence.
- Conditional Inference: Image and language modules predict per-location codebook distributions from pretrained ResNet-18 or BERT features, using cross-entropy training.A weight α balances the shape prior and conditional marginals, with α = 0.75 for images and α = 0.5 for language.
A.4. Experimental Details
Experiments compare the proposed representation and generation framework with task-specific baselines and use a neural evaluator for text-shape alignment.
- Single-view Baselines: The single-view baselines ResNet2TSDF and ResNet2Voxel predict 643 TSDF grids or 323 voxel grids from image features.ResNet2TSDF uses L1 loss, while the voxel baseline follows the corresponding image-lifting architecture.
- Neural Evaluator: The neural evaluator uses rendered images and point clouds, with a VGG classifier reaching approximately 95% held-out accuracy across eight ShapeNet classes.
- Neural Evaluator: Figure 8 evaluates whether the neural evaluator selects the target shape from two candidates given a text description using prediction confidence.The rightmost example has low confidence when the candidate shapes are similar.
- Text-to-shape Baselines: The joint text-shape baseline directly models the joint distribution with a BERT-conditioned transformer decoder, fixed rasterized generation order, and ShapeNet chairs only.The proposed method uses a prior-plus-conditional decomposition instead.
B. Autoregressive Results
The learned latent representation reconstructs inputs and supports diverse unconditional shape generation across categories, indicating a usable generic autoregressive shape prior.
- Reconstruction: P-VQ-VAE faithfully reconstructs input shapes and also reconstructs Pix3D shapes despite being trained on ShapeNet.
- Random Generation: Unconditional transformer generation starts from an empty sequence and produces diverse results across all categories.The authors report that this diversity shows the prior learns representations of generic shapes.
C. Shape Completion Results
Shape-completion evaluations compare the proposed method with baselines across additional qualitative examples and report more completion results.
- Comparisons: Additional comparisons with completion baselines are presented in Figure 11.
- Additional Results: Additional shape-completion results are presented in Figure 12.
D. Single-view Results
This section presents additional single-view reconstruction results and comparisons with competing methods. The results are shown in Figures 12 and 13.
- Additional single-view reconstruction results are presented.
- Figure 13 provides further comparisons with competing methods.
- Figure 12 provides additional results from the proposed method.
E. Language-guided Generation Results
The language-generation evaluation compares the proposed approach with JE and T2S using multiple random generations per text description. The results show stronger text alignment and realistic 3D shapes, while complex or ambiguous descriptions can reduce generation quality.
- For every text description, the comparison reports three random generations from each method.The compared methods are JE and Text2Shape (T2S).
- Generation quality is impacted by complex and ambiguous descriptions from held-out data.Some generated shapes can be random, such as a flying chair.
- The proposed method aligns better with text and generates realistic 3D shapes than JE and T2S.T2S generations show less diversity than the proposed approach.