ShapeFormer: Transformer-based Shape Completion via Sparse Representation

1. Introduction

• Surface completion must infer missing geometry from incomplete observations; deterministic local implicit functions can fail on ambiguous unseen regions, while redundant representations burden generative models.Observed point clouds have variable sampling densities and missing parts because surface acquisition is limited by line of sight.Source: §1 · pp. 1–2
• ShapeFormer uses vector-quantized local codes to learn a distribution over possible completions, reducing representation size while retaining underlying shape structure.The model generates multiple high-quality completions conditioned on the partial input.Source: §1 · p. 2
• VQDIF sparsely encodes each non-empty local feature as a discrete position-content 2-tuple, reducing sequence length from cubic to quadratic in feature resolution.The compact, structured sequences decode to deep implicit functions for extracting high-quality surfaces while representing complex shapes with acceptable accuracy.Source: §1 · p. 2
• ShapeFormer autoregressively predicts distributions over the next feature’s location and content, conditioning complete-sequence generation on a prepended sequence from the partial observation.This generates whole sequences rather than preserving potentially noisy input features unchanged.Source: §1 · p. 2
• FPD on PartNet improves by at most 1.7 over prior multi-modal cGAN, and ShapeFormer reports state-of-the-art multi-modal completion quality and diversity.The reported scope includes ambiguous partial observations across CAD models and human bodies, as well as real-world scans with missing parts.Source: §1 · p. 2

Shape reconstruction and completion

• Voxel methods [18] [20] [32] face cubic-resolution costs, mesh methods constrain topology or self-intersect, while point clouds [1] [25] need surface recovery.Source: §2 · p. 2
• Implicit functions support high-quality reconstruction [13] and local priors refine detail [15] [43], but deterministic methods struggle with ambiguity; global-code generators [2] misalign, while point-cloud generators handle only simple shapes.The paper proposes a compact, structured representation for multimodal high-quality reconstruction of complex shapes.Source: §2 · p. 2

Autoregressive models and Transformers

• Transformers model long-range dependencies and enable diverse image completions [67], while 3D autoregressive point-cloud [69] and mesh [47] models are limited to small outputs or 1024 vertices.Autoregressive models factor joint distributions into conditional distributions via the chain rule.Source: §2 · p. 3
• Compressed discrete representations reduce redundancy [56] and enable high-resolution image synthesis [55], with sparse variants [22] [48]; concurrent work includes dense-grid AutoSDF and Point-BERT generative pre-training.ShapeFormer explores the compact-representation direction for surface completion.Source: §2 · p. 3

3. Method

• ShapeFormer models p(SC|SP) over compact shape sequences and samples complete sequences that decode into meshes faithful to partial point-cloud input.This reframes completion as estimating a distribution over watertight meshes rather than a single output.Source: §3 · p. 3

3.1. Compact sequence encoding for 3D shapes

• VQDIF encodes local shapes in non-empty grid cells as coordinate-and-code-index tuples, then decodes the sequence through a 3D U-Net into a deep implicit function.The encoder uses local-pooled PointNet features and vector quantization against a learned shape dictionary.Source: §3.1 · pp. 3–4
• VQDIF training combines reconstruction loss with a commitment loss, while dictionary embeddings are updated using exponential moving averages.The commitment loss keeps encoded features close to their nearest dictionary entries.Source: §3.1 · p. 4

3.2. Sequence generation for shape completion

• ShapeFormer generates complete sequences autoregressively, factorizing each tuple into coordinate and value predictions with two decoder-only transformers conditioned on the partial sequence.At inference, it samples coordinates and value indices with top-p sampling until an end token, while masking invalid coordinates to enforce monotonicity.Source: §3.2 · pp. 4–5, Figure 3 · p. 4

4.1. Shape completion results

• Structured local features preserve input details better than global-feature methods, while deterministic methods and non-generative Transformers struggle with multimodal completions.The VQDIF-only model also fails on highly ambiguous chair scans; ShapeFormer instead models a distribution whose sampled shapes are sharp and plausible.Source: §4.1 · pp. 6–7
• For high-ambiguity ShapeNet scans, ShapeFormer achieves much better FPD, while remaining competitive in accuracy on low-ambiguity scans; on PartNet it improves faithfulness, diversity, and shape quality over GAN-based methods.For PartNet, the reported metrics are UHD for faithfulness, TMD for diversity, and MMD and FPD for shape quality.Source: Table 1 · p. 5, Table 2 · p. 6, §4.1 · p. 7

4.2. More results

• On Redwood scans, ShapeFormer produces scan-faithful, plausible completion distributions that reflect viewpoint uncertainty; it also completes unseen shape types and human scans with varied plausible geometry and poses.Despite training only on 13 ShapeNet categories, it completes unseen cups and teapots, including alternative solid or hollow interpretations; for human scans, it preserves observed-part poses while varying unobserved poses.Source: §4.2 · pp. 7–8, Figure 6 · p. 7, Figure 7 · p. 7

4.3. Surface reconstruction with VQDIF

• VQDIF at feature resolution 32 achieves accuracy similar to IF-Net while using significantly less representation space, enabled by sparse local features and quantized integer indices.Accuracy loss is most salient at lower feature resolution; ShapeFormer uses Ours16, whose average sequence length is 217 and whose accuracy is comparable with ConvONet.Source: §4.3 · p. 8, Figure 9 · p. 8, Figure 10 · p. 8

Conclusions

• ShapeFormer learns a conditional distribution of completions, enabling multiple plausible sampled shapes and sharper outputs than regression to the mean.The authors report better completion quality and diversity than previous methods.Source: §5 · p. 8
• VQDIF compresses shapes into short sequences of sparse, discrete local features to facilitate generative learning for 3D shapes.The representation enables better completion quality and diversity than previous methods.Source: §5 · p. 8
• Generating one complete shape currently takes 20 seconds, limiting applications such as robotics.Future work may use more efficient attention to learn smaller VQDIF representations and improve completion quality, and incorporate geometric or physical reasoning to address ambiguities.Source: §5 · p. 8

Appendix A

• Appendix A details the ambiguity measure, model architectures, and training and testing statistics.Source: Abstract · p. 12

Appendix B

• Appendix B presents additional visual comparisons with previous methods on scans with high and low ambiguity.Source: Abstract · p. 12

Appendix C

• Appendix C provides further analysis of the method, including a discussion of its limitations.Source: Abstract · p. 12

• The supplementary material also includes the model code.Source: Abstract · p. 12

A.1. Ambiguity measure for partial point cloud

• Incomplete surface area is only a proxy for ambiguity: it is relatively easy to compute and strongly correlated with ambiguity, but cannot fully reflect it.For example, equally incomplete top and bottom table scans can differ greatly in ambiguity.Source: §A.1 · p. 12
• Figure 11 shows the sorted ambiguity scores for 70 teapot scans, with example scans and their score-colored complete shapes.Viewing direction greatly influences scan ambiguity.Source: Figure 11 · p. 12, §A.1 · p. 12
• The proposed Amb score uses complete-to-partial Chamfer-L2 distance, normalized by each complete-shape point’s farthest distance, to compare scans across shapes.The partial-to-complete distance is negligible, so only the complete-to-partial direction is calculated.Source: §A.1 · p. 12
• F-score can fail to distinguish how a partial cloud covers the complete shape, motivating the Chamfer-L2-based score.The passage contrasts partial-cloud coverage patterns that F-score does not differentiate.Source: §A.1 · p. 12
• The score is evaluated on 70 views per shape: 64 Fibonacci-sampled directions and six orthogonal views, sorted by score.The complete cloud’s point count is denoted by B.Source: §A.1 · p. 12
• Low-score scans capture global structure or salient features, whereas high-score scans leave multiple plausible global structures and are harder to complete.The paper reports that ShapeFormer handles such high-ambiguity scans better than existing completion methods.Source: §A.1 · p. 12

A.2. Architectures

• VQDIF maps point clouds to quantized sparse sequences and decodes them into an implicit function that predicts probe-point occupancy.It is trained on complete dense clouds (N = 32768) but can encode partial clouds at test time.Source: Figure 12 · p. 13, §A.2 · p. 14
• A local pooled PointNet builds a 64-resolution, 32-feature grid, which is down-sampled with a small receptive field before non-empty features are masked, flattened, and quantized.The saved empty feature supports projecting the sequence back to a feature grid in the decoder.Source: §A.2 · p. 14
• VQDIF decodes the projected sequence with a 3D U-Net, up-sampler, and ResNet-based implicit decoder designed for global shape context and occupancy prediction.The three-step U-Net integrates local and global information; the implicit decoder predicts occupancy probabilities for queried probe points.Source: §A.2 · p. 14
• ShapeFormer appends [END] tokens to variable-length SP and SC sequences, additively combines their learnable index embeddings, and uses two causal transformers to predict coordinates then values.The coordinate transformer conditions on previous tuples; the value transformer also conditions on the predicted next coordinate. [22] [48]Source: §A.2 · p. 14
• Two-layer output heads with softmax produce categorical distributions for sequence elements, and randomly masking partial sequences during training improves generalization.Each transformer has its own output head for the next sequence element.Source: §A.2 · p. 14

A.3. Details on training and sampling

• Training uses Adam with separate learning rates and schedules for VQDIF and ShapeFormer, while sampling produces one sequence in about 20 seconds.VQDIF uses a 1e−4 learning rate with step decay (step size 10, β = .9); ShapeFormer uses 1e−5 with no learning-rate scheduling. On a server with 10 Nvidia Quadro P6000 GPUs, convergence takes 30 hours on virtual scans, 8 hours on PartNet, and 16 hours on D-Faust; 24 sequences can be sampled in parallel in 5 minutes.Source: §A.3 · p. 14

B. More comparisons

• On high-ambiguity ShapeNet scans, ShapeFormer produces better-quality, more diverse completions than prior methods, whose deterministic outputs show an averaging effect.PoinTr exhibits scattering in ambiguous completion regions.Source: B. More comparisons · p. 14, Figure 14 · p. 15, Figure 15 · p. 16
• On low-ambiguity scans, ShapeFormer produces accurate, high-quality completions, while ConvONet, IF-Net [15], and PoinTr can perform poorly on unseen regions.The comparison shows one completion per sampling strategy, with ground truth in the first row; top-.0 sampling (Ours*) often yields slightly more accurate shapes than top-.4 sampling (Ours).Source: B. More comparisons · pp. 14, 18, Figure 16 · p. 17

C. More analysis

• With VQDIF restricted to R = 16, ShapeFormer may fail to preserve intricate structure, and slow sampling prevents interactive applications.Figure 17 shows intricate details in the input that are missing from the completions.Source: C. More analysis · p. 18, Figure 17 · p. 18
• Proposed remedies include more efficient attention, adaptive point-cloud quantization, and parallel autoregressive sampling.The attention proposals aim to reduce quadratic sequence-length complexity, while adaptive quantization would focus dependencies locally and use higher-level features for distant regions.Source: C. More analysis · p. 18
• Generating complete shapes from scratch may slightly alter input geometry to overcome sparsity and noise; high-resolution input features in decoding are proposed to improve fidelity [67].Higher-resolution quantized features are another proposed way to obtain more accurate generation.Source: C. More analysis · p. 18