Source-linked AI summary
Joint Rigid Alignment and Morphable Reconstruction from Partial 3D Face Scans
Eli A. Underwood, Nicolas Rondan, Angel V. Peterchev, J. Matias Di Martino
TL;DR
Partial 3D face scans leave pose and unobserved shape to be recovered together, while existing pipelines typically separate registration and completion. This paper predicts pose and morphable-model completion jointly, and reports stronger performance across held-out and real-scan evaluations than the compared baselines.
Problem
Partial face scans arrive in unknown poses and omit facial regions, so usable geometry requires both rigid alignment and completion of unseen shape.
Source: §1 · p. 1Method
A single network conditions a cross-attention morphable-coefficient regressor on GeoTransformer features and correspondences, then uses the morphed template for fine matching and pose estimation.
Source: §1 · pp. 1–2Results
On held-out Val identities, the method leads NSFP and GraphSCNet by 3–4× on GT error, and is most accurate and fastest on NoW and NPHM scans.
Source: §4.4 · p. 6, §6 · p. 8Takeaways & Limitations
Pairing per-vertex ground-truth error with Chamfer distance exposes collapse that Chamfer alone conceals, and the method avoids that failure across the reported settings.
Source: §1 · p. 2, §6 · p. 8No directly usable ground truth on NoW or NPHM lets the authors verify scan-referenced surface fit but not occluded-region completion quality.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Aligning a partial, real-world 3D face scan to a canonical representation requires solving two coupled problems at once: estimating the rigid pose that brings the scan into a common frame, and completing the facial shape in regions the scan never observed. Existing pipelines typically decouple these steps, running a generic point-cloud registration front end followed by a separate non-rigid fitting or deformation stage. Building on the GeoTransformer superpoint-matching architecture, we propose a single network that predicts both a rigid pose and a shape-complete point cloud in one forward pass: a cross-attention regressor, conditioned on transformer point features and predicted coarse correspondences, maps a partial scan to per-patch coefficients of a 3D morphable face model learned offline from a corpus of 8,000 identities. The resulting morphed template is then used, in the same forward pass, as the target for fine-grained Sinkhorn matching and rigid pose estimation, supervised purely by geometric alignment to ground-truth morphed geometry, so no hand-labeled coefficients are required. Against six classical and learned non-rigid registration baselines on held-out identities, our method leads the strongest ones by 3-4 times on per-vertex error and baselines that collapse the head onto the visible region by more than an order of magnitude, and is simultaneously the most accurate and fastest method on ground-truth-free scans from two independent real-scan datasets, NoW and NPHM. Code will be made publicly available.
1. Introduction
Partial 3D face scans require both rigid pose recovery and completion of unobserved facial geometry. The paper couples these tasks through superpoint matching and morphable-model completion in one network.
1. Introduction
- A partial scan in an unknown pose requires both rigid rotation-and-translation recovery and completion of facial regions the scan never observed.The stated uses include identity comparison, anthropometric measurement, prosthetic design, and animation retargeting.Source: §1 · p. 1
- Most benchmarked non-rigid registration methods collapse the head onto the visible region, with several times higher per-vertex error than methods that complete the shape.The authors attribute this to methods satisfying smoothness regularizers by collapsing rather than completing occluded geometry.Source: §1 · p. 1
- A cross-attention regressor uses GeoTransformer superpoint features and correspondences to predict morphable coefficients, then uses the morphed template for fine matching and pose estimation.Dense geometric alignment to ground-truth morphed geometry supervises completion without hand-labeled coefficients, with gradients flowing back into matching.Source: §1 · pp. 1–2
- Across all settings, the method avoids collapse and outperforms all baselines without external rigid alignment.The evaluation also includes held-out identities and real scans from NoW and NPHM.Source: §1 · p. 2
- The contribution includes a short scan-referenced flow refinement and an evaluation pairing per-vertex error with Chamfer distance to expose collapse that Chamfer alone conceals.The evaluation extends to NoW and NPHM real-scan datasets.Source: §1 · p. 2
2. Related work
Related work spans rigid registration, non-rigid surface fitting, morphable face models, and generic shape-completion priors. The paper positions its approach as combining a face-specific completion prior with point-cloud registration.
Rigid point cloud registration
- Rigid registration ranges from ICP and feature-based RANSAC pipelines to learned correspondence methods such as GeoTransformer, DFAT, and Regor.GeoTransformer uses geometric self- and cross-attention over superpoints to find correspondences without RANSAC.Source: §2 · p. 2
Non-rigid registration and surface fitting
- Figure 1 contrasts OAReg, ClusterReg, and CPD collapsing onto the visible region with the proposed method’s plausible full-head recovery.Both the baselines and the proposed method use the same partial input in the comparison.Source: Figure 1 · p. 2
- Non-rigid methods use deformation fields, smoothness or rigidity regularization, learned correspondences, or test-time optimization; the cited approaches lack a face-specific shape prior.The paper notes that smoothness or rigidity regularization alone does not prevent global collapse on partial face scans.Source: §2 · p. 2
3D morphable face models
- 3D morphable models encode face shape with a low-dimensional basis, commonly fit iteratively or regressed from an RGB image; this method uses the model as a registration target.Its decoder predicts patch-wise coefficients from geometric point features, and the morphed points serve as the fine-matching target in the same forward pass.Source: §2 · p. 3
Learned shape completion priors
- Generic completion networks learn missing geometry from partial inputs, whereas this method constrains completion to a face-specific PCA basis with anatomically localized coefficients.The paper associates this domain-specific prior with completing occluded facial geometry rather than merely preserving visible regions.Source: §2 · p. 3
3. Method
The method jointly predicts a complete face shape and rigid pose from one partial scan in a single network pass. Local morphable patches, transformer-based matching, and geometric supervision connect completion with pose estimation.
3. Method
- The forward pass predicts a complete point cloud and rigid pose from the same geometric features, rather than using independent rigid and non-rigid stages.The method first reconstructs the morphed template, then uses it for fine matching and pose estimation.Source: §3 · p. 3, §3.1 · p. 3
3.1. Preliminaries: a collection of local morphable patch models
- The model uses 32 overlapping local PCA bases, each built from 700 template vertices, so patch-specific correspondences can localize completion.The bases are learned from 8,000 synthetic head identities; their assembled output uses a fixed, parameter-free gather.Source: §3.1 · p. 3
3.2. Backbone and coarse matching
- A shared KPConv backbone and GeoTransformer encode the mean template and scan, then produce coarse superpoint correspondences supervised against ground-truth overlap.GeoTransformer attention uses pose-invariant pairwise distances and triplet angles.Source: §3.2 · pp. 3–4, Figure 2 · p. 4
3.3. Morphable-coefficient regression
- A cross-attention regressor predicts per-patch coefficients from transformer features, raw scan features, and features at predicted correspondences.These inputs let it use observed geometry for visible regions and learned completion for occluded areas.Source: §3.3 · p. 4
3.4. Morphing the reference and fine matching
- The morphed template replaces mean-face reference positions for Sinkhorn fine matching, whose local-to-global registration estimates pose against the completed shape.Dense correspondences are extracted within coarse-matched patch pairs before robust aggregation of per-patch rigid hypotheses.Source: §3.4 · p. 4
3.5. Losses
- A dense per-vertex geometric loss compares the predicted template with full ground-truth morphed geometry, and its weight is ten times the coarse and fine losses.Coefficients are not directly supervised; the shape-level signal sends gradients through the regressor, transformer, and backbone.Source: §3.5 · p. 5
3.6. Training data
- Training pairs come from the 8,000-identity corpus, with 7,500 identities used for training and 250 held out for evaluation.Partial scans use randomized Face-Cut or Self-Occlusion crops with randomized rigid transforms; noise and density augmentation are applied during training.Source: §3.6 · p. 5
3.7. Scan-referenced refinement
- The method does not fit the predicted template directly to scan surfaces, so it misses fine scan-specific detail in visible regions; test-time flow refinement recovers constrained detail while preserving completion where scans are absent.The refinement starts from the predicted pose and template, avoiding the external rigid-ICP pre-alignment used by NSFP from scratch.Source: §3.7 · p. 5
4. Experiments
The experiments evaluate completion and alignment on held-out synthetic scans and two real-scan datasets, using ground-truth vertex error where available and visible-region Chamfer distance elsewhere. The method leads the baselines on validation and real scans, while ablations examine refinement’s role and accuracy–latency trade-offs.
4.1. Datasets and protocol
- Validation uses 500 paired partial views from 250 held-out identities; NoW and NPHM are independent real-scan datasets without usable completion ground truth.Validation reports Self-Occlusion and Face-Cut separately; NPHM evaluation uses frontal face crops downsampled to 5k points.Source: §4.1 · pp. 5–6
4.2. Baselines
- Six non-rigid registration baselines receive the same 24-start point-to-point ICP pre-alignment, whose time is included in their reported runtime.Ours predicts pose end to end without an external rigid-alignment stage; iterative 3DMM-fitting baselines are omitted because they lack a starting pose for arbitrarily posed scans.Source: §4.2 · p. 6
4.3. Metrics
- Validation reports median per-vertex error using shared template topology and visible-region 1-way CD, which can reward implausible shape collapse.CD re-searches nearest neighbors after deformation, so it does not measure whether unseen geometry is plausible.Source: §4.3 · p. 6
4.4. Results: accuracy and Chamfer’s blind spot
- 3–4× lower GT error than NSFP and GraphSCNet; Ours leads every baseline on validation GT error, CD, and time.Four of six baselines have an order of magnitude worse GT error; ClusterReg can nevertheless have low CD despite 38–47mm GT error.Source: §4.4 · p. 6, Table 2 · p. 6
4.5. External dataset results
- 0.59mm vs. 0.61mm CD and 2.26s vs. 11.80s on NoW: Ours is more accurate and faster than NSFP, and Ours leads every baseline on NPHM CD and time.Real-scan evaluation has no shape-completion ground truth; NPHM uses face-only crops because its full-head scans include hair incompatible with the template.Source: §4.5 · p. 7, Table 3 · p. 7, Table 4 · p. 7
4.6. Qualitative results
- Figure 4 shows partial inputs, morphed references, and overlays for validation, NoW, and NPHM examples to inspect shape departure and surface fit.Validation rows include Self-Occlusion and Face-Cut crops; NoW and NPHM each show two representative scans.Source: §4.6 · p. 7, Figure 4 · p. 8
4.7. Ablation Studies
- On NoW, refinement lowers CD from 2.29mm to 0.59mm at roughly two additional seconds per scan.The forward pass already leads baselines on validation GT error and takes 0.12s per scan; refinement’s contribution is to visible-surface fit.Source: §4.7 · pp. 7–8
- From its cheapest refinement setting, Ours Pareto-dominates every baseline across tested NoW budgets; 500 steps beat all but NSFP at its highest budget.Increasing refinement to 10,000 steps continues improving CD without a plateau.Source: Figure 3 · p. 7, §4.7 · p. 8
5. Discussion and limitations
Real-scan results lack directly usable ground truth for occluded-region completion, and the linear PCA prior is bounded by its training population. The paper also notes the work’s dual-use context and synthetic training identities.
5. Discussion and limitations
- NoW and NPHM lack directly usable ground truth for evaluating occluded-region completion, so their scan-referenced accuracy cannot establish completion quality.NoW reference scans serve a different task, while NPHM full-head scans include hair incompatible with the hairless template.Source: §5 · p. 8
- The linear PCA prior is limited to the training population’s span, so under-represented identities, expressions, or ages may be poorly reconstructed.The basis is fit to 8,000 identities; a learned nonlinear basis could relax this constraint at the cost of the reported plausibility guarantee.Source: §5 · p. 8
- The method uses point geometry and synthetic training identities, but improved facial-geometry recovery remains dual-use.The synthetic identities are sampled from the published UHM model; the authors see no immediate deployment risk.Source: §5 · p. 8
6. Conclusion
The method jointly predicts facial-shape completion and rigid pose for partial scans, followed by short scan-referenced refinement. It leads held-out identities and ground-truth-free NoW and NPHM scans on the reported accuracy and speed comparisons.
6. Conclusion
- 3–4× lower per-vertex error than the strongest baselines on held-out Val identities, while leading NoW and NPHM scans in accuracy and speed.The comparisons are reported in Tables 2, 3, and 4.Source: §6 · p. 8
- The evaluation reports per-vertex ground-truth error alongside Chamfer distance, exposing collapse failures that Chamfer distance alone conceals.The authors report that most baselines collapse the head onto the visible region rather than completing occluded geometry.Source: §6 · p. 8
- A single network predicts morphable-model-based facial-shape completion and rigid pose for partial 3D face scans, followed by short scan-referenced refinement.The method jointly predicts completion and pose rather than treating them as separate outputs.Source: §6 · p. 8