PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Non-Rigid 3D Registration

Bailin Deng, Yuxin Yao, Roberto M. Dyke, Juyong Zhang

arXiv:2203.07858v2cs.CVcs.GRcs.LG

TL;DR

Non-rigid registration must align deformable surfaces despite challenges from noise, partial overlap, and efficiency needs. This survey reviews deformation representations, registration methods, and evaluation resources for dynamic shape acquisition and reconstruction. It reports strong intrinsic-method performance in anthropometric applications.

  • Problem

    Non-rigid registration must find reliable correspondences despite deformation, noisy or outlier-containing measurements, low surface overlap, and practical efficiency requirements.

    Source: §1 · p. 1

  • Method

    The survey reviews deformation-field representations, optimization- and learning-based registration methods, and datasets and benchmarks for dynamic 3D shape acquisition and reconstruction.

    Source: §1 · p. 1, §5 · p. 21

  • Results

    The survey reports strong performance of intrinsic methods in anthropometric applications.

    Source: §6 · p. 22

  • Takeaways & Limitations

    Ground-truth correspondences support comparison of registration accuracy using RMSE and geodesic error.

    Source: §5 · p. 21

    Extrinsic methods remain unsatisfactory for small-overlap alignment, especially with large deformation or noisy data, and may not distinguish unmatched source points.

    Source: §6 · p. 22

Abstract

from arXiv · show

Non-rigid registration computes an alignment between a source surface with a target surface in a non-rigid manner. In the past decade, with the advances in 3D sensing technologies that can measure time-varying surfaces, non-rigid registration has been applied for the acquisition of deformable shapes and has a wide range of applications. This survey presents a comprehensive review of non-rigid registration methods for 3D shapes, focusing on techniques related to dynamic shape acquisition and reconstruction. In particular, we review different approaches for representing the deformation field, and the methods for computing the desired deformation. Both optimization-based and learning-based methods are covered. We also review benchmarks and datasets for evaluating non-rigid registration methods, and discuss potential future research directions.

1. Introduction

Non-rigid registration aligns deformable surfaces for tasks such as dynamic shape reconstruction, but deformation representation, noisy or partial data, and efficiency make alignment challenging. The survey defines its scope, situates prior reviews, and previews its organization and notation.

1. Introduction

  • Non-rigid registration allows different source regions to deform to align surfaces, supporting dynamic shape reconstruction and applications in VR, AR, and entertainment.Unlike rigid registration, it accounts for non-rigid behavior such as articulation; noise, outliers, low overlap, and efficiency needs complicate reliable correspondence.Source: §1 · p. 1

1.1. Scope and Related Works

  • The survey focuses on 3D registration for dynamic shape acquisition and reconstruction in graphics and vision, including partial-to-complete matching but excluding bijective-correspondence and multimodal methods.It distinguishes its scope from earlier surveys focused on correspondence, older registration work, or medical imaging.Source: §1.1 · pp. 1–2

1.2. Organization

  • The review covers deformation representations, extrinsic and intrinsic optimization- and learning-based methods, evaluation datasets and benchmarks, and future research directions.Source: §1.2 · p. 2

1.3. Notations

  • The notation uses X and Y for source and target surfaces, which may be explicit point clouds or meshes or implicit signed-distance level sets.For explicit surfaces, the passage defines point sets X and Y and deformed source positions bX.Source: §1.3 · p. 2

2. Representation of the Deformation Field

Deformation-field representations trade expressive power against computational cost. The survey reviews dense, reduced-dimensional, patch-based, implicit-grid, and shape-prior approaches, each with distinct strengths and constraints.

Pointwise position variables and pointwise affine transformation

  • Pointwise positions offer direct flexibility but can create redundant degrees of freedom, while pointwise affine transforms better model local rotations and are often constrained near rigid transforms.Source: §2 · p. 2

Deformation graph-based methods

  • Deformation graphs reduce variables by assigning transformations to sparse surface nodes and blending their effects over nearby points, with neighboring-node consistency regularizing smoothness.The graph nodes are shown on the mesh in Figure 2.Source: §2 · pp. 2–3, Figure 2 · p. 3

RKHS-based methods

  • RKHS methods represent a displacement field with a Gaussian-kernel construction and commonly optimize it probabilistically with regularization such as coherence of nearby-point motion.Source: §2 · p. 3

Patch-based methods

  • Patch-based methods assign transformations or displacement models to surface regions and combine their effects for globally consistent deformation; they can extend deformation into regions without target correspondence.Some approaches use learned deformation modes or fuse patchwise face positions and normals into a complete mesh.Source: §2 · p. 3

Spline function-based methods

  • Spline-based deformation fields use control points and basis functions, with thin-plate splines and free-form deformation among the reviewed approaches.Control points may be placed on a grid or selected from feature correspondences.Source: §2 · pp. 3–4

Grid-based methods

  • Grid-based methods represent surfaces implicitly with SDFs or TSDFs and deform them using a field defined over the enclosing volume, potentially on a coarser grid for efficiency.The deformation may use local rigid transforms or regularized translations at grid points.Source: §2 · p. 4

Prior-based representations

  • Shape-prior representations encode class-specific structure, including skeletons, hand models, SMPL bodies, and parameterized face models, and can represent covered shapes with fewer degrees of freedom.Their performance may degrade for shapes outside the parametric model's coverage.Source: §2 · p. 4

Summary and discussion

  • Representations trade deformation detail against the number of variables and computational cost; reduced-dimensional approaches can miss fine-scale variation or require careful control-structure selection.Deformation graphs suit same-subject pose changes driven primarily by joint articulation; grids can handle topological change but high resolution increases memory use, while priors may fail outside modeled shape classes.Source: §2 · pp. 2, 4

3. Extrinsic Methods

Extrinsic methods search deformation space using distances measured in ambient 3D space. Their design starts from choosing criteria that measure alignment quality.

3. Extrinsic Methods

  • Given a deformation representation, registration searches the deformation space for a solution that best aligns the two surfaces.Source: §3 · p. 4
  • Alignment quality must be measured with criteria designed to compare the source and target surfaces.Source: §3 · p. 4
  • Extrinsic methods measure alignment using distances between surfaces in ambient 3D space.Source: §3 · p. 4

3.1. Optimization-Based Methods

Optimization-based registration minimizes an alignment error together with regularization on the deformation. The survey reviews choices for these terms, robustness, probabilistic models, initialization, and numerical solution.

3.1. Optimization-Based Methods

  • Registration minimizes an alignment term plus weighted regularization, which constrains the deformation, for example through smoothness.The balance parameter α controls the relative contribution of the alignment and regularization terms.Source: §3.1 · p. 5

3.1.1. Alignment term

  • Alignment terms include point-to-point and point-to-plane distances, distances to implicit SDF or TSDF surfaces, and probabilistic measures.When both surfaces use SDFs, their alignment can be measured by differences between the fields over sample points.Source: §3.1.1 · p. 5

3.1.2. Regularization term

  • Regularization promotes smoothness and local rigidity, while other terms can preserve landmarks, local geometry, feasible model parameters, temporal coherence, or collision-free poses.The formulation varies with the deformation representation and application.Source: §3.1.2 · pp. 5–6

3.1.3. Improving robustness

  • Reliability weights or sparsity-inducing robust norms reduce the influence of outlier or unmatched points and can permit less-smooth deformation near articulated joints.The survey also reviews Tukey, Geman-McClure, Huber-L1, and Welsch losses for alignment or smoothness terms.Source: §3.1.3 · pp. 6–7

3.1.4. Probabilistic models

  • Probabilistic methods model alignment with Gaussian mixtures, including CPD’s source-centroid and target-data formulation, and may add noise or local-structure modeling.Other approaches align two Gaussian-mixture density functions using their L2 distance.Source: §3.1.4 · p. 7

3.1.5. Initialization

  • Initialization ranges from a rough rigid transform, optionally with uniform scaling, to sparse feature-based correspondences when non-rigid deformation is substantial.Features include spin-images, PFH/FPFH, SHOT, and RGB-image SIFT; SHOT matches may be pruned using inconsistent geodesic distances.Source: §3.1.5 · pp. 7–8

3.1.6. Numerical solvers

  • Differentiable objectives use gradient-based or nonlinear least-squares solvers; non-smooth objectives may use first-order methods such as block coordinate descent or ADMM.ICP-like alternating solves and EM updates provide closed-form steps in some formulations, while first-order subproblems can be parallelized.Source: §3.1.6 · p. 8

3.2. Learning-Based Methods

Learning-based methods train models to predict deformation or correspondence for unseen data. The survey covers supervised and unsupervised approaches, including their representations and training losses.

3.2.1. Supervised methods

  • Supervised methods learn correspondence or deformation from ground-truth labels, using CNNs, transformers, deformation graphs, displacement fields, or parametric shape models.Some systems combine learned correspondences with graph-based optimization; others predict voxel displacement fields or fit human models such as SMPL.Source: §3.2.1 · pp. 8–9
  • Supervised training losses compare predicted correspondences or deformed points with ground truth, while displacement-field models may use separate coarse-alignment and refinement losses.Heatmap predictions can be trained with binary cross-entropy and negative log-likelihood losses.Source: §3.2.1 · p. 9

3.2.2. Unsupervised methods

  • Because ground-truth correspondence or deformation is difficult to obtain, unsupervised methods train on unlabeled data using alignment losses such as GMM, Chamfer, reconstruction, or Earth mover’s distance.A clipped Chamfer loss is also proposed for point clouds with outliers and missing data; point-to-point reconstruction loss is described as easier to train than Chamfer distance.Source: §3.2.2 · pp. 9–10
  • Unsupervised systems may add regularizers to prevent trivial solutions and promote desired structure, including symmetry, Laplacian detail, local rigidity, and parameter regularization.The multi-view projection loss provides directional information by comparing depth and mask projections across differently oriented planes.Source: §3.2.2 · p. 10

3.3. Applications and Systems

The survey describes non-rigid registration in dynamic reconstruction and tracking systems. It reviews template-less methods, prior-based systems, learning-based systems, and combinations of learning with optimization.

3.3.1. Template-less methods

  • Template-less systems fuse or register successive RGB-D observations to reconstruct deforming scenes, with methods addressing tracking drift, rapid motion, detail, or appearance.DynamicFusion runs in real time but is reported as not robust to fast movement and occluded areas.Source: §3.3.1 · p. 10

3.3.2. Methods based on priors

  • Prior-based systems use human, head, hand, or other template models to guide reconstruction and tracking, with some approaches adding skeletons or inertial measurements.A sparse skeleton may be less robust to fast motion or incomplete surfaces; a double-layer model is described as addressing that issue.Source: §3.3.2 · pp. 10–11

3.3.3. Learning-based methods

  • Learning-based systems contribute learned correspondences, features, deformation graphs, implicit representations, or pose estimates to reconstruction and tracking pipelines.The surveyed examples include globally consistent registration, dynamic fusion followed by implicit surface reconstruction, and real-time hand and deformable-object reconstruction.Source: §3.3.3 · p. 11

3.3.4. Methods combining optimization and learning

  • Hybrid systems use learning to provide correspondences, prior-model parameters, or solver aids, then apply optimization to refine registration or deformation.Examples include learned preconditioners for PCG and learned features used in an alignment term for Gauss-Newton optimization.Source: §3.3.4 · p. 11

3.4. Summary

Tables 1 and 2 summarize the section’s optimization-based and learning-based extrinsic methods.

3.4. Summary

  • Tables 1 and 2 summarize optimization-based and learning-based extrinsic methods, listing papers with both components in each relevant table.The passage identifies the tables’ scope but does not provide their entries.Source: §3.4 · p. 11

4. Intrinsic Methods

This section surveys intrinsic methods for aligning shapes in an embedded domain, including parametric, minimum-distortion, spectral/functional, shape-recovery, and learned approaches to establishing correspondences and reconstructing geometry. It also covers challenges such as partial overlap, eigenbasis stability, training constraints, and mesh orientation or connectivity.

4. Intrinsic Methods

  • Intrinsic methods map surfaces into another domain, where they are categorized as parametric, minimum-distortion, or spectral/functional; many establish correspondences rather than explicitly deforming the source.Correspondences can serve as hard or soft constraints in a non-rigid registration framework.Source: §4 · p. 11

4.1. Optimization-Based methods

  • Optimization-based methods seek a bijective point mapping, but the n! permutation space is impractical to search and partial matching expands the solution beyond bijections.When overlap is unknown, finding it depends on a mapping that itself depends on the overlap.Source: §4.1 · p. 13

4.1.1. Minimum-distortion methods

  • Minimum-distortion methods optimize correspondences using intrinsic measures, but partial matching is generally limited to near-isometric deformation and is sensitive to geometric and topological noise.[CRB∗16] report that state-of-the-art minimum-distortion methods tend to perform relatively poorly on partial correspondence compared with functional and learning-based approaches.Source: §4.1.1 · pp. 13–15
  • Sparse and coarse-to-fine sampling reduces assignment costs, while geodesic-based pruning removes inconsistent correspondences; partial-shape sampling and initialization remain challenges.Sampling methods include entropy-based ordering, extremity and curvature prioritization, and farthest-point sampling; geodesic pruning can be restricted to local neighborhoods or adapted to anisotropic distances.Source: §4.1.1 · p. 14
  • Minimum-distortion work predominantly targets isometric deformation; variants relax global consistency or address local anisotropy, while template recovery combines geodesic regularization with a corresponding-point data term.Conformal-map methods can be sensitive to distortion, and optimal-transport approaches for full-to-full matching are not guaranteed to find a global optimum.Source: §4.1.1 · pp. 14–15

4.1.2. Spectral/Functional mapping methods

  • Functional methods represent shapes with Laplace–Beltrami eigenfunctions and estimate a matrix C mapping corresponding eigenvectors; because the basis is isometry-invariant, C becomes less diagonal under non-isometric deformation.Regularization can promote Laplacian commutativity and orthogonality of the soft map.Source: §4.1.2 · pp. 15–16
  • For partial matching, functional-map methods estimate corresponding parts with correspondence and part regularizers, but part-to-part formulations can require alternating optimization across both shapes and scale poorly.[RCB∗17] assumes an injective mapping, while [CRM∗16] estimates both parts iteratively; [LRB∗16b] extends the approach to multiple parts, and [LRBB17] aligns eigenbases entirely in the spectral domain.Source: §4.1.2 · pp. 16–17
  • Coarse-to-fine functional maps work well for full matching but can fail on partial shapes because independently computed low-frequency eigenvectors may be inconsistent [MRR∗19].The approach incrementally adds higher-frequency eigenvectors; the survey illustrates the partial-matching instability in Fig. 7.Source: §4.1.2 · p. 17
  • The Hamiltonian operator localizes eigenvectors to an overlapping region, enabling isometric partial-to-full matching and disconnected components, but its non-convex optimization is slow and fails with too few eigenvectors [RTO∗19].The method estimates the overlap through a potential function and aligns Hamiltonian and Laplacian eigenvalues before computing the functional map.Source: §4.1.2 · pp. 17–18

4.1.3. Spectral/Functional shape recovery methods

  • Spectral shape recovery uses shape-difference operators, template-less deformation, and direct spectral alignment, with learned priors or extrinsic information addressing reconstruction ambiguities [HRA∗19, CPR∗19, ELC20].Intrinsic metrics can leave multiple isometric solutions; offset-surface edge lengths add extrinsic guidance, while shape-collection methods are limited by available examples and search complexity.Source: §4.1.3 · pp. 18–19

4.2. Learning-Based Methods

  • Learned intrinsic descriptors span spectral, geodesic, and chart-based operators, while directional functions based on parallel transport address ill-defined mesh orientations.For partial matching, the optimized spectral descriptor outperforms HKS and WKS; other approaches use anisotropic kernels or Zernike radial polynomials.Source: §4.2 · p. 19
  • Learning-based functional maps enhance descriptors through supervised or unsupervised objectives, but simple mapping techniques limit partial correspondence; cross-attention can down-weight non-overlapping features [COC15, LRR∗17, APO21].Supervised training requires costly pointwise correspondences, while later methods penalize geodesic distortion or functional-map heuristics.Source: §4.2 · p. 19
  • Learning-based reconstruction uses spectral or geodesic constraints and intrinsic feature matching, while spiral convolution operators remain sensitive to mesh connectivity [ETLTC20, CNH∗20, ENK∗21, BBP∗19, GCBZ19].The [ENK∗21] approach also uses an as-rigid-as-possible error term, so it is not fully intrinsic.Source: §4.2 · pp. 19–20

4.3. Summary

5. Datasets and Benchmarks

The survey reviews datasets and benchmarks for evaluating correspondence and registration, including quantitative and qualitative measures, ground-truth construction, and gaps in support for learning-based methods.

5. Datasets and Benchmarks

  • Ground-truth correspondences support RMSE and geodesic-error evaluation, while registration-dependent applications can provide indirect measures of performance.The survey also lists fitting error, coverage error, area ratio, rendering, distortion, and information transport as evaluation measures.Source: §5 · pp. 20–21
  • Anthropomorphic datasets, including facial datasets, are common; sparse fiducial landmarks and texture can support correspondence evaluation.Texture can enable reliable automatic landmarks or ground-truth correspondences.Source: §5 · p. 21
  • SHREC tracks have introduced challenges such as topological changes and partial correspondence through synthetic model modifications.Source: §5 · p. 21
  • Template alignment can provide dense correspondences for real meshes, but may introduce local alignment errors that texture or reliable landmarks can help improve.Template alignment methods include non-rigid registration, linear blend skinning, and parametric modeling.Source: §5 · p. 21
  • Only a few benchmarks provide training facilities for learning-based methods, limiting fair and consistent comparisons.The survey notes that methods may instead be trained on other datasets or generative models.Source: §5 · p. 21
  • Table 5 overviews datasets used to evaluate correspondence and registration methods.Source: §5 · p. 21, Table 5 · p. 21

6. Directions for Future Works

The survey identifies future directions spanning dynamic 3D acquisition, deformation representations, and optimization- and learning-based registration. It highlights limits involving data, partial overlap, deformation assumptions, and real-time use.

6. Directions for Future Works

  • Dynamic 3D reconstruction from monocular RGB video remains a research direction because existing methods mainly target rigid shapes and are too costly for real-time mobile use.RGB video is easier for the general public to capture than depth data, according to the survey.Source: §6 · pp. 21–22
  • Neural implicit representations offer a continuous, resolution-independent alternative to point clouds and meshes, motivating adaptation of existing registration methods and new methods that exploit them.Source: §6 · p. 22
  • For large, noisy deformations, handcrafted descriptors can be ambiguous or noise-sensitive, while learning-based methods remain limited by scarce, narrow correspondence-labeled datasets and unsatisfactory generalization.The survey calls for broader labeled datasets and/or methods with better generalization.Source: §6 · p. 22
  • Adaptive weights and robust norms help with partial overlap, but small overlap remains difficult, especially with large deformation or noisy data, because extrinsic formulations may not identify unmatched source points.Source: §6 · p. 22
  • Existing shape regularizations assume near-isometry, non-isometry with characteristics such as conformality, or combinations, and are less effective for some complex physical deformations.The survey points to data-driven modeling of deformation behavior as a possible direction.Source: §6 · p. 22
  • Anthropometric applications show strong intrinsic-method performance, but hand-crafted intrinsic partial matching is inadequate for larger real-time models and partial geometry can violate bijectivity assumptions.Learning-based methods may avoid expensive optimization, while improved overlap estimation may help intrinsic methods handle partial and non-isometric shapes.Source: §6 · p. 22
  • For handheld scanning in small patches, inconsistent standard Laplacian eigenfunctions make current functional mapping methods unsuitable.Source: §6 · p. 22
  • Topological changes from fused scan geometry remain challenging for intrinsic optimization, and extensive surface discontinuities are understudied because they challenge local-smoothness assumptions.The survey notes that data-driven approaches may help with topological-change scenarios.Source: §6 · p. 22
Loading 2203.07858v2…