Source-linked AI summary
LoopReg: Self-supervised Learning of Implicit Surface Correspondences, Pose and Shape for 3D Human Mesh Registration
Bharat Lal Bhatnagar, Cristian Sminchisescu, Christian Theobalt, Gerard Pons-Moll
TL;DR
LoopReg addresses human-scan registration when classical fitting is sensitive to initialization and supervised correspondence predictors are not end-to-end differentiable. It jointly learns scan correspondences and parametric human-model parameters through a self-supervised loop using an implicit, diffused 3D surface representation. Experiments show improved registration as additional unlabeled scans are processed after a supervised warm-start, while the method remains bounded by the need for that warm-start and limited testing of rare poses.
Problem
Human-model registration is sensitive to initialization, while existing learned correspondence initializers require labeled scans and non-differentiable downstream optimization.
Method
LoopReg jointly learns scan-to-model correspondences and parametric human-model parameters through a self-supervised loop with an implicit distance-field surface and diffused 3D model functions.
Results
LoopReg outperforms existing optimization- and learning-based approaches, and registration accuracy improves as more unlabeled scans are added after supervised warm-start training.
Takeaways & Limitations
LoopReg can leverage information across a corpus of unlabeled scans without requiring precomputed joints or landmarks for each input.
Takeaways & Limitations
The self-supervised loop requires a small supervised warm-start, and limited testing makes generalization to rare poses difficult to establish.
Abstract
from arXiv · showhide
We address the problem of fitting 3D human models to 3D scans of dressed humans. Classical methods optimize both the data-to-model correspondences and the human model parameters (pose and shape), but are reliable only when initialized close to the solution. Some methods initialize the optimization based on fully supervised correspondence predictors, which is not differentiable end-to-end, and can only process a single scan at a time. Our main contribution is LoopReg, an end-to-end learning framework to register a corpus of scans to a common 3D human model. The key idea is to create a self-supervised loop. A backward map, parameterized by a Neural Network, predicts the correspondence from every scan point to the surface of the human model. A forward map, parameterized by a human model, transforms the corresponding points back to the scan based on the model parameters (pose and shape), thus closing the loop. Formulating this closed loop is not straightforward because it is not trivial to force the output of the NN to be on the surface of the human model - outside this surface the human model is not even defined. To this end, we propose two key innovations. First, we define the canonical surface implicitly as the zero level set of a distance field in R3, which in contrast to morecommon UV parameterizations, does not require cutting the surface, does not have discontinuities, and does not induce distortion. Second, we diffuse the human model to the 3D domain R3. This allows to map the NN predictions forward,even when they slightly deviate from the zero level set. Results demonstrate that we can train LoopRegmainly self-supervised - following a supervised warm-start, the model becomes increasingly more accurate as additional unlabelled raw scans are processed. Our code and pre-trained models can be downloaded for research.
1 Introduction
LoopReg addresses unreliable model-based registration under large pose, shape, and clothing variation by jointly learning correspondences and human-model parameters end to end. Its implicit, differentiable surface representation supports self-supervised training, with accuracy improving as unlabeled scans are added.
- Motivation: Classical registration methods are vulnerable to local minima under large pose, shape, and clothing variation, while existing learned initializers require labeled data or non-differentiable optimization.LoopReg targets a joint end-to-end differentiable objective over correspondences and human-model parameters.
- Contribution: LoopReg jointly models scan data and a parametric human model in an end-to-end learning process for correspondences.The framework is designed to learn correspondence and model parameters together rather than only initialize a later optimization.
- Representation: LoopReg replaces UV correspondence parameterization with a continuous, differentiable implicit surface defined as a distance-field zero level set.This representation avoids the cuts, discontinuities, and distortion associated with planar surface parameterization.
- Self-supervision: Registration accuracy improves as more unlabeled data is added after a supervised warm-start, enabling self-supervised learning from raw scans.The approach uses a small amount of supervised initialization before incorporating additional unlabeled scans.
2 Related Work
Prior correspondence and model-registration methods face discontinuous or distortion-prone surface representations and are often not end-to-end differentiable. LoopReg combines an implicit 3D surface with diffused model functions to jointly differentiate correspondence and model parameters.
- Prior work: Existing correspondence and registration approaches include functional maps, unsupervised learning, implicit correspondences, parametric fitting, and optimization, but do not provide LoopReg’s joint formulation.The related work spans shape correspondence prediction and articulated-human model registration.
- LoopReg: LoopReg predicts scan-to-template correspondences with CorrNet and jointly optimizes the parametric model and network under self-supervision.The method maps each scan point to a canonical model and uses those correspondences during joint optimization.
- Prior work: Learned correspondence initializers typically require labeled scans and are followed by non-differentiable optimization of model parameters.This limits end-to-end training of correspondences and model parameters from unlabeled scan corpora.
3 Method
LoopReg replaces alternating scan-to-model fitting with a differentiable loop that jointly learns correspondences and per-scan human-model parameters. It diffuses the model into 3D so predictions near, but not exactly on, the canonical surface remain usable.
- 3.1 Classical Model-Based Fitting: The classical fitting baseline alternates correspondence estimation with model-parameter optimization, making the process non-differentiable and unsuitable for end-to-end training.It minimizes distances between scan points and corresponding deformed model points using non-linear ICP.
- 3.2 Proposed Formulation: LoopReg learns scan-to-model correspondences and instance-specific human-model parameters jointly through an end-to-end differentiable formulation.The method uses a neural backward map and a human-model forward map to close the registration loop.
- 3.2 Proposed Formulation: The canonical surface is represented implicitly through a distance field, while model functions are diffused from mesh vertices to a 3D region using closest-surface projection and barycentric interpolation.Diffusion produces a continuous function over R3 rather than only discrete vertices or a cut UV parameterization.
- 3.2 Proposed Formulation: A distance-transform penalty keeps predicted correspondences near the template surface, where the forward model is well behaved.The gradient of the distance transform points toward the closest surface point.
- 3.2 Proposed Formulation: Training uses a small supervised warm start, then combines self-supervised data-to-model and loop losses with pose-and-shape regularization on unlabeled scans.The data-to-model term pulls the deformed model toward the scan, stabilizing correspondence learning; good initialization is required for self-supervised training.
- 3.2 Proposed Formulation: CorrNet implements the learned correspondence map with a PointNet++ backbone that predicts correspondences for input scan points.
4 Experiments
LoopReg is evaluated on undressed and dressed human scans, outperforming prior registration approaches and improving as additional unlabeled scans are added. The experiments also examine correspondence prediction and the role of supervised warm-start data.
- Registration comparisons: LoopReg generalizes across undressed and fully clothed scans, including complex poses, while using point clouds without additional joint or landmark information.The evaluation includes scans from multiple sources and compares against existing optimization and learning-based approaches.
- Semi-supervised training: Performance improves significantly as more unsupervised scans are added, especially for the demanding vertex-to-vertex metric.Table 2 uses 2631 scans in total, including 1000 scans for supervised warm-start.
- Registration comparisons: 2.2mm versus 2.9mm average surface-to-surface error shows LoopReg outperforming prior methods on dressed scans without pre-computed joints.The comparison is reported for SMPL+D fitting, with prior approaches performing particularly poorly on complex poses.
- Correspondence prediction: The method outperforms existing supervised and unsupervised correspondence prediction approaches on the FAUST correspondence task.The comparison uses results reported in the corresponding prior papers.
- Semi-supervised training: A supervised warm-start is important before LoopReg can train effectively with self-supervised data.The warm-start uses 1000 scans, fewer than the order of magnitude more supervised data required by 3D-CODED for optimal performance.
5 Conclusions
The paper concludes that LoopReg enables end-to-end semi-supervised scan registration by jointly learning correspondences and model parameters. Its distance-field representation supports differentiable predictions in unconstrained 3D space without requiring a 2D surface parameterization.
- Conclusions: LoopReg jointly learns model parameters and a correspondence function in an end-to-end differentiable formulation that leverages unlabeled scan corpora.Unlike instance-specific optimization, the method is trained across a corpus of scans.
- Conclusions: Representing the parametric model as a distance-field zero level set diffuses its function throughout R3 and removes the prerequisite of 2D surface parameterization.The representation permits predictions in unconstrained R3 before mapping them to the model surface.
- Conclusions: Experiments report that LoopReg outperforms existing optimization- and learning-based registration approaches without requiring precomputed joints or landmarks.The formulation can still incorporate such information as additional objectives.
Broader Impact
The work targets controllable registration of 3D human scans for graphics and vision applications. Its broader applicability is limited by scarce 3D data, difficult generalization testing, and privacy concerns around personal human scans.
- Broader impact: Registering scans to a controllable parametric model supports downstream applications including animation, AR, VR, gaming, and aggregate 3D data processing.The paper describes registration as a prerequisite for many computer graphics and computer vision applications.
- Limitations: Limited 3D data makes extensive testing and generalization difficult, leaving systems brittle on out-of-sample inputs such as rare human poses.The paper identifies this limited-testing bottleneck as a barrier to reliable applications.
- Broader impact: Collecting, processing, and storing personal 3D human data raises potential privacy challenges while public awareness remains limited.The paper frames this as a broader concern for deep-learning-based human 3D vision.
LoopReg: Supplementary Material
The supplementary material provides notation guidance and additional experimental results for correspondence prediction and scan registration.
- Supplementary material: The supplement first explains the paper's notation and then presents additional correspondence-prediction and qualitative scan-registration results.These materials are intended to assist reading the main paper and extend its reported evaluations.
1 Legend for Notations
The supplementary legend defines the paper’s notation for scans, the parametric human model, its diffused functions, and training losses.
- Notation: The notation covers scan points, canonical and posed human-model surfaces, model parameters, and transformations.Key symbols include S, s_i, M, M′, M_T, x, θ, β, D, v_i, and G_k(θ, β).
- Notation: The paper defines diffused human-model functions over a 3D domain H ⊂ R3, including g_M and parameterized variants.Diffusion extends functions from the model surface into the surrounding 3D domain.
- Notation: Training notation distinguishes self-supervised, unsupervised, supervised, and model-regularization losses.The unsupervised loss is defined as L_unsup = L_self + L_d→m.
- Notation: CorrNet notation includes correspondence regressors for individual body parts, while Table 2 compares correspondence-prediction approaches.The legend is provided to improve readability because the paper uses many symbols.
2 Results: Correspondence prediction
LoopReg performs strongly on FAUST correspondence prediction despite being trained primarily with self-supervision, outperforming the compared supervised and unsupervised methods.
- Correspondence prediction: LoopReg performs better than competing correspondence approaches on the FAUST test set, despite not being directly trained for cross-shape correspondence.FAUST contains 200 noisy scans of undressed people in challenging poses, and evaluation uses geodesic distance to ground-truth correspondences.
3 More qualitative results
Qualitative results show that LoopReg produces high-quality registrations and continuous scan-to-template correspondences for dressed and undressed humans in complex poses.
- Qualitative results: LoopReg produces high-quality registrations for both undressed and dressed scans in complex poses.The supplementary results show undressed registrations in Fig. 1 and dressed registrations in Fig. 2.
- Qualitative results: The method predicts continuous correspondences from scan points to the canonical human template for both clothing conditions.These correspondences are visualized in Fig. 3.
4 Limitations and Future Work
The self-supervised registration loop is differentiable but depends in practice on a small supervised warm-start, and performance remains below methods using additional manual or landmark cues.
- Limitations and Future Work: A small supervised warm-start is important because poorly initialized self-supervised training can become stuck in a local minimum.The formulation jointly differentiates through correspondences and instance-specific human-model parameters.
- Limitations and Future Work: Registration quality remains below methods that use precomputed joints, facial landmarks, and manual intervention, although those cues could be integrated later.The paper presents this integration as potential future work rather than a required component of LoopReg.