Source-linked AI summary
ShapeFormer: Transformer-based Shape Completion via Sparse Representation
Xingguang Yan, Liqiang Lin, Niloy J. Mitra, Dani Lischinski, Daniel Cohen-Or, Hui Huang
TL;DR
Incomplete and noisy point clouds make 3D shape completion ambiguous, with multiple plausible completions possible. ShapeFormer models a conditional distribution over completions using a Transformer and compact sparse VQDIF sequences, then samples complete shapes. It reports improved completion quality and diversity over prior methods, while generation speed remains a practical limitation.
Problem
Incomplete and noisy point clouds leave missing geometry ambiguous, so multiple plausible shape completions may exist and deterministic local representations can fail on unseen regions.
Method
ShapeFormer uses a Transformer-based autoregressive model over compact sparse sequences from Vector Quantized Deep Implicit Functions to model p(SC|SP) and sample complete shapes.
Results
ShapeFormer achieves state-of-the-art multi-modal completion quality and diversity, with PartNet FPD improved by at most 1.7 over cGAN.
Takeaways & Limitations
The method produces multiple sharp, plausible completions faithful to partial observations and generalizes to unseen shape types such as cups and teapots.
Takeaways & Limitations
Generation currently takes 20 seconds per complete shape, limiting application in fields such as robotics.
Abstract
from arXiv · showhide
We present ShapeFormer, a transformer-based network that produces a distribution of object completions, conditioned on incomplete, and possibly noisy, point clouds. The resultant distribution can then be sampled to generate likely completions, each exhibiting plausible shape details while being faithful to the input. To facilitate the use of transformers for 3D, we introduce a compact 3D representation, vector quantized deep implicit function, that utilizes spatial sparsity to represent a close approximation of a 3D shape by a short sequence of discrete variables. Experiments demonstrate that ShapeFormer outperforms prior art for shape completion from ambiguous partial inputs in terms of both completion quality and diversity. We also show that our approach effectively handles a variety of shape types, incomplete patterns, and real-world scans.
1. Introduction
ShapeFormer addresses ambiguous, incomplete 3D observations by learning a distribution of plausible completions with a compact sparse representation and autoregressive Transformer. The approach targets diverse, high-quality completions while reducing the sequence complexity that makes direct 3D Transformer modeling difficult.
- Motivation: Surface completion must infer missing geometry from partial, unevenly sampled observations using priors and non-local evidence.Camera-based acquisition is limited by line of sight, producing missing parts and varying sampling densities.
- Motivation: Deterministic local deep implicit functions often fail on ambiguous inputs where multiple plausible completions exist, while generative combinations can waste capacity on redundant details.The limitation concerns unseen regions and representations with enormous statistical redundancy.
- Approach: ShapeFormer learns a distribution over possible shape completions with a Transformer-based autoregressive model using discrete vector-quantized local features.The representation reduces size while retaining the underlying structure needed for completion.
- Approach: VQDIF encodes non-empty local features as sparse discrete 2-tuples of position and content, reducing sequence length from cubic to quadratic in feature resolution.The sequences decode to deep implicit functions from which surfaces are extracted.
- Approach: ShapeFormer generates complete sequences conditioned on partial observations by sequentially predicting the next location and content, rather than merely filling unseen regions.This allows the model to regenerate noisy input features instead of preserving their noise automatically.
- Results: The method targets diverse, high-quality completions across CAD models, human bodies, and real-world scans with missing parts.The paper reports state-of-the-art multi-modal completion quality and diversity.
2. Related Work
Prior 3D completion methods use several output representations, each balancing fidelity, efficiency, topology, and computational cost. The paper situates its sparse discrete sequence representation and Transformer-based generation within this broader reconstruction and autoregressive-modeling landscape.
- Shape reconstruction and completion: 3D reconstruction from partial input is an ill-posed problem because unseen regions must be completed using data priors.The related-work discussion covers voxel, mesh, point-cloud, and deep-implicit representations.
- Shape reconstruction and completion: Voxel methods benefit from regular 3D convolutions but face cubic resolution complexity, making outputs either coarse or computationally heavy.Meshes are more data-efficient but introduce topology difficulties, while template-based methods can be limited to one topology.
- Autoregressive models and Transformers: Autoregressive models factor a high-dimensional joint distribution into a sequence of conditional distributions, and Transformers model long-range dependencies through self-attention.These properties motivate their use for generative 3D completion.
3. Method
ShapeFormer represents 3D shapes as compact sparse sequences of discrete local codes and models their conditional distribution autoregressively to sample complete shapes from partial observations.
- Shape completion maps a partial point cloud P to a complete watertight mesh M by estimating the distribution p(M|P).
- Sampling multiple complete sequences from the conditional distribution yields different surface reconstructions from the same partial observation.
- VQDIF encodes each non-empty grid cell with a dictionary index and location, forming short discrete sequences that capture global shape dependencies.The decoder converts the sequence into a feature grid and a local deep implicit function whose iso-surface is the reconstructed mesh.
- The encoder sparsifies a feature grid, flattens non-empty features in row-major order, and records their locations as sequence coordinates.Row-major ordering is selected because alternative orderings are less effective for generation.
- Vector quantization replaces encoded features with nearest dictionary embeddings and stores their indices, while commitment loss keeps features close to selected entries.The VQDIF objective combines reconstruction loss with commitment loss weighted by β; stop-gradient prevents the embedding from being affected by that loss.
- ShapeFormer autoregressively models p(SC|SP) by factoring each tuple into coordinate and value distributions, with the value conditioned on the current coordinate.Two decoder-only transformers separately predict coordinates and value indices, and generation samples both until an end token is reached.
4. Results and Evaluation
ShapeFormer is evaluated across ambiguous scans, part-level incompleteness, real scans, unseen shape types, human shapes, and representation efficiency. Results show diverse, faithful completions and a compact VQDIF representation with competitive reconstruction accuracy.
- Evaluation setup: Experiments cover ShapeNet partial scans, PartNet part-level incompleteness, real Redwood scans, out-of-distribution objects, human shapes, and VQDIF efficiency.The evaluation compares against global and local implicit methods, PoinTr, cGAN, and ShapeInversion.
- ShapeNet results: ShapeFormer achieves much better FPD for high-ambiguity ShapeNet scans while remaining competitive in accuracy for low-ambiguity scans.The paper notes that CD is unreliable under high ambiguity because plausible completions can be treated as errors.
- PartNet results: On PartNet, ShapeFormer provides better faithfulness, greater diversity, and high-quality shapes than GAN-based methods across Table, Chair, and Lamp.The reported metrics are UHD for faithfulness, TMD for diversity, and MMD and FPD for completion quality.
- Generalization and real-world results: ShapeFormer produces multiple plausible completions for real scans, unseen object types, and partial human bodies while preserving observed structure and varying with ambiguity.Examples include Redwood scans, teapots and cups, and human bodies with different possible poses.
- Representation analysis: Ours32 reaches similar accuracy to IF-Net while being significantly smaller through sparse and discrete VQDIF features.The representation-size comparison evaluates Ours8, Ours16, and Ours32 against OccNet, ConvONet, and IF-Net.
5. Conclusions
ShapeFormer learns a conditional distribution of shape completions and samples multiple plausible outputs, while VQDIF compresses 3D shapes into short sparse discrete sequences. The main practical limitation is sampling speed, with further work needed for efficient attention and geometric or physical reasoning.
- Conclusion: ShapeFormer models a conditional distribution from which multiple plausible completed shapes can be sampled.The distribution-based formulation is intended to produce sharp outputs rather than mean-regressed blurry shapes.
- Conclusion: VQDIF compresses shapes into short sequences of sparse, discrete local features and supports better completion quality and diversity than previous methods.The representation is introduced to facilitate generative learning for 3D shapes.
- Limitations and future work: Sampling speed currently limits application to fields such as robotics, requiring 20 seconds per generated complete shape.The authors also propose more efficient attention and adding geometric or physical reasoning as future directions.
Supplementary Material
This passage lists the paper's authors.
- The listed authors are Xingguang Yan, Liqiang Lin, Niloy J. Mitra, Dani Lischinski, Daniel Cohen-Or, and Hui Huang.
Abstract
The supplied passage describes the supplementary document rather than the paper’s abstract.
- Supplementary material: The supplementary document contains detailed descriptions, additional visual comparisons, further analysis, and model code.
A. Implementation Details
The paper defines scan ambiguity through normalized incomplete-to-complete distances, then validates the measure across sampled views. Lower ambiguity corresponds to scans whose salient structure or global form is captured.
- Ambiguity measure: Ambiguity measures the variety of potential complete shapes, but the paper uses incompleteness as a computable proxy because direct ambiguity is difficult to measure.The proxy is strongly correlated with ambiguity, although it cannot fully reflect cases such as top versus bottom scans of a table.
- Ambiguity measure: The proposed Amb metric uses normalized complete-to-partial Chamfer-L2 distance, which increases when a partial cloud misses more global structure.The partial-to-complete distance is treated as negligible, so only complete-to-partial distance is calculated and normalized by each point’s farthest distance within the complete shape.
- Ambiguity measure: B denotes the number of points in the complete cloud used by Amb.The displayed definition evaluates scan C against complete point cloud B.
- View sampling: The evaluation samples 70 views per shape: 64 evenly distributed over the view sphere and six orthogonal views, then sorts them by ambiguity score.A teapot example illustrates the resulting score distribution and associates scans with their positions on the curve.
A.2. Architectures
The architectures compress sparse 3D geometry into short discrete VQDIF sequences and use autoregressive transformers to generate complete shape sequences. VQDIF reconstructs implicit geometry from quantized sparse features, while ShapeFormer predicts coordinates and values sequentially.
- Training and configuration: The implementation uses a maximum sequence length of 812, with architecture parameters and component input sizes documented in Table 4.Both VQDIF and ShapeFormer are optimized with Adam; VQDIF uses learning rate 1e-4 and ShapeFormer uses 1e-5.
- VQDIF: VQDIF encodes a point cloud into a discrete sequence and decodes that sequence into a deep implicit function.Its encoder-decoder architecture is trained with complete inputs, while the trained encoder can encode sparse or dense partial clouds at test time.
- VQDIF: The VQDIF encoder uses local-pooled PointNet on a 64-resolution grid, then downsamples to resolution R while preserving sparse non-empty features.A small receptive field helps keep the representation sparse by limiting each feature’s covered region.
- VQDIF: The VQDIF decoder projects quantized sparse features to a grid, applies a 3D U-Net and upsampling, and uses an implicit decoder to reconstruct occupancy and shape.The decoder uses a large receptive field to combine local and global information when classifying probes as inside or outside the shape.
- ShapeFormer: ShapeFormer uses two causally masked decoder-only transformers: one predicts the next coordinate and the other predicts its value conditioned on that coordinate.Output heads convert transformer features into categorical distributions, and the architecture handles variable-length sequences with end tokens and additive embeddings.
B. More comparisons
Additional comparisons show strong performance on high-ambiguity scans and competitive accuracy on low-ambiguity scans. The method produces diverse completions where ambiguity is high and more accurate completions than deterministic baselines in the low-ambiguity examples.
- High-ambiguity scans: On high-ambiguity ShapeNet scans, ShapeFormer avoids the averaging effect visible in deterministic methods and produces better completion quality and diversity.The comparisons include additional examples in Figures 14 and 15.
- Low-ambiguity scans: On low-ambiguity scans, ShapeFormer achieves competitive accuracy while prior deterministic methods can remain unsatisfactory in unseen regions.Figure 16 compares ConvONet, IF-Net, and PoinTr against the proposed sampling strategies.
- Sampling strategies: The best-sampling strategy, Ours*, selects coordinate and value indices with the highest probability and often produces slightly more accurate shapes.The comparisons use Ours for top-.4 sampling and Ours* for top-.0 sampling.
C. More analysis
The analysis identifies representation length and sampling speed as key limitations, especially for intricate structures and interactive use. Generating complete sequences from scratch can also alter input geometry to overcome sparsity and noise.
- Limitations: The method currently uses VQDIF at R = 16 because transformer sequence length cannot be too long, which may fail on intricate structures.Figure 17 shows a failure case where details present in the input are not preserved in the completions.
- Limitations: Sampling speed is a practical limitation that prevents interactive applications.The paper reports a current sampling speed of 20 seconds per generated complete shape.
- Possible directions: More efficient attention mechanisms and adaptive quantization are proposed to reduce sequence-complexity constraints and preserve finer local structure.The paper also mentions advanced autoregressive sampling techniques such as parallel sampling.
- Input fidelity: Because complete sequences are generated from scratch, the results may slightly alter input geometry to overcome sparsity and noise.The paper suggests higher-resolution quantized features or incorporating high-resolution input features during decoding as possible improvements.