Source-linked AI summary
Dense Human Body Correspondences Using Convolutional Networks
Lingyu Wei, Qixing Huang, Duygu Ceylan, Etienne Vouga, Hao Li
TL;DR
Dense correspondence between human scans is difficult under clothing, arbitrary poses, partial observations, and large deformation. The paper learns per-pixel descriptors through multi-task body-region classification with smooth embeddings, and reports accurate dense matching for full or partial human shapes, while acknowledging dependence on the training prior and remaining outliers.
Problem
Accurate dense correspondence for partial human shapes with arbitrary large deformations, clothing, and pose variation was not available from existing methods.
Method
The method learns per-pixel depth-image descriptors through shared-tower classification tasks and repeated random segmentations that enforce smoothness near body-region boundaries.
Results
The framework establishes accurate and dense correspondences between partial scans of clothed subjects in arbitrary poses, with few outliers and correct front-to-back matching in a real capture example.
Takeaways & Limitations
A smooth embedding makes learned descriptors suitable for dense matching across full or partial human shapes with varying pose and clothing.
Takeaways & Limitations
The supervised framework cannot handle arbitrary shapes because its prior is entirely based on the training-data class, and clothed training data is limited by manual correspondence annotation.
Abstract
from arXiv · showhide
We propose a deep learning approach for finding dense correspondences between 3D scans of people. Our method requires only partial geometric information in the form of two depth maps or partial reconstructed surfaces, works for humans in arbitrary poses and wearing any clothing, does not require the two people to be scanned from similar viewpoints, and runs in real time. We use a deep convolutional neural network to train a feature descriptor on depth map pixels, but crucially, rather than training the network to solve the shape correspondence problem directly, we train it to solve a body region classification problem, modified to increase the smoothness of the learned descriptors near region boundaries. This approach ensures that nearby points on the human body are nearby in feature space, and vice versa, rendering the feature descriptor suitable for computing dense correspondences between the scans. We validate our method on real and synthetic data for both clothed and unclothed humans, and show that our correspondences are more robust than is possible with state-of-the-art unsupervised methods, and more accurate than those found using methods that require full watertight 3D geometry.
1. Introduction
Existing correspondence methods struggle with clothed humans, arbitrary poses, partial scans, and large deformations. The paper introduces a deep-learning framework that learns smooth descriptors for accurate dense matching across such inputs.
- Challenges: Existing methods generally require small deformations and sufficient overlap, while clothed-body approaches are limited by compatible or zero-genus surface topologies.Accurate dense correspondence for partial shapes, including single-view RGB-D scans with arbitrarily large deformations, had not yet been proposed.
- Approach: The proposed network computes dense correspondences for clothed subjects in arbitrary complex poses from full models, partial scans, or depth maps.Training combines depth maps generated from SCAPE, Yobi3D, and MIT data.
- Approach: The method learns distinctive feature vectors for human-body subregions using an AlexNet-based classification network.A unified feature descriptor combines training data from unclothed bodies and clothed subjects.
- Approach: Repeated mesh segmentations produce smoother feature embeddings for geodesically nearby points, reducing outliers near segmentation boundaries.The paper also reports that this design can drastically reduce the amount of training data required.
- Contribution: The approach is presented as the first accurate and dense correspondence method for clothed human shapes with partial input data and greater efficiency than traditional non-rigid registration.The contribution targets partial scans and clothing while retaining dense matching.
- Approach: The framework combines training data from human bodies in different poses with clothed subjects in a canonical pose.This unified learning framework supports the joint treatment of varying pose and clothing data.
2. Related Work
Human-shape correspondence is difficult because clothing, pose variation, topology changes, partial data, and large deformation undermine conventional geometric methods. Related approaches rely on restrictive deformation assumptions, fragile descriptors, or learned representations that motivate the paper’s deep-learning strategy.
- Motivation: Human clothing, pose variation, and pose-induced topology changes make existing shape-correspondence methods difficult to apply.These factors distinguish human correspondence from more constrained geometric-shape matching problems.
- Geometric methods: Naively matching n points on two surfaces requires O(n!) computation, so practical methods depend on prior knowledge about allowable deformations.Nearly-isometric or conformal assumptions make the search more tractable but restrict applicability.
- Geometric methods: Variational correspondence methods optimize geodesic or diffusion-distance objectives, but often become NP-hard combinatorial problems and require nearly-isometric surfaces.Spectral, Markov-random-field, and convex relaxations are used to address the optimization difficulty.
- Descriptors: Feature-descriptor methods seek matches among selected points, yet descriptors invariant to human and clothing deformation and robust to topology changes remain challenging.Local geometric descriptors are insufficient under deformation, while spectral descriptors are sensitive to partial data and topological noise.
- Learned descriptors: Deep neural networks provide a data-learned alternative to ad hoc geometric descriptors for correspondence computation.The paper extends this idea from image correspondence to full and partial scans of clothed humans.
3. Problem Statement and Overview
The method learns per-pixel descriptors through shared classification tasks, including sparse key-point and dense segmentation labels, then uses descriptor similarity to establish dense correspondences across full or partial human shapes.
- The framework computes dense correspondences across full or partial human shapes from depth maps of people in arbitrary poses and varying clothing.
- A descriptor f maps each depth-image pixel to a feature vector intended to depend on anatomical location rather than pose, clothing, body shape, or viewing angle.
- The shared descriptor tower is trained jointly on sparse key-point classification and dense pixel-wise segmentation tasks from heterogeneous datasets.
- Multiple randomized body segmentations make descriptor similarity correlate with body-surface distance, producing smoother embeddings near segmentation boundaries.
- Per-vertex descriptors are averaged across rendered depth maps, and nearest-neighbor search in feature space establishes correspondences for full or partial scans.
- The resulting correspondences can be supplied to an existing deformation method when surface alignment is required.
4. Implementation Details
Training combines human-shape datasets covering pose and clothing variation with simulated depth images, while the network efficiently produces per-pixel descriptors and labels for correspondence experiments.
- Training Data Generation: Training data combines SCAPE, MIT, and Yobi3D models to cover pose variation, dense correspondences, and diverse clothing and local geometry.SCAPE provides 71 registered meshes of one person; MIT provides animation sequences of three characters; Yobi3D provides 2000 clothed digital characters.
- Training Data Generation: Each model is rendered from 144 viewpoints as a 512 × 512 depth image, approximating commercial depth-camera input.
- Training Data Generation: Experts annotate 33 shared anatomical key points, which become 33 training classes after propagation to simulated scans.
- Network Design and Training: The network uses a descriptor extraction tower that outputs a 16-dimensional descriptor for every pixel while processing the entire depth image.Whole-image processing shares computation among adjacent patches and is intended to improve testing-time efficiency.
- Network Design and Training: A classification module predicts either key-point classes or dense segmentation classes from the per-pixel descriptors.
- Network Design and Training: Training randomly selects classification tasks, partial scans, and dense-label segmentations during 200,000 stochastic-gradient-descent iterations.
- Evaluation: Evaluation covers full-to-full, partial-to-full, and partial-to-partial matching on real and synthetic datasets, with synthetic correspondence error visualized against ground truth.
5. Results
The method is evaluated on real and synthetic naked and clothed data, including full and partial matching, challenging poses, garments, props, and large deformations. It improves correspondence accuracy over a recent baseline, supports template-based reconstruction, and runs correspondence computation in 0.5–1 s, while retaining boundary and out-of-distribution errors.
- Evaluation: The method is evaluated on real and synthetic datasets containing naked and clothed subjects, with both full and partial matching.Synthetic evaluation includes SCAPE, MIT, FAUST, and Mixamo; FAUST and Mixamo were not used for training.
- Evaluation: Real Kinect One examples cover side views, bent postures, dresses, vests, backpacks, and hats in single depth-map frames.Each partial frame contains 512 × 424 pixels.
- Error analysis: Correspondences are accurate in most surface regions but show larger errors at depth-map boundaries, hands, and feet.The paper attributes these errors to segmented clusters being slightly too large in those areas.
- Comparisons: Average correspondence accuracy improves for both intra-subject and inter-subject FAUST pairs compared with Chen et al..Table 2 reports average error and worst-pair average error in centimeters, while noting potentially worse WE.
- Comparisons: The method handles larger deformations between full templates and partial scans than other non-rigid registration algorithms.The comparison concerns full-template to partial-scan matching.
- Application: The method is used to reconstruct geometry and motion by deforming a template to each captured frame, producing faithful and smooth reconstructions without explicit temporal-coherency enforcement.The application is demonstrated on both real and synthetic data.
- Limitations: The current implementation produces wrong correspondences for poses and clothing substantially different from the training data and limits correspondence resolution to 512×512 depth-map pixels.The resolution limit is attributed to GPU memory constraints; grouped outliers may aid outlier detection.
- Performance: 0.5–1 s is the reported correspondence-computation time, depending on input resolution.Feature-descriptor extraction never exceeds 1 ms per depth map, while complete training takes about 250 hours.
6. Conclusion
The paper concludes that a smooth learned embedding enables accurate dense correspondence for partial scans of clothed subjects in arbitrary poses. It emphasizes efficiency and unified treatment of full models and partial depth-map inputs, while identifying training-data coverage and difficult-to-prune outliers as remaining boundaries.
- Conclusion: A smooth embedding reduces misclassification artifacts at segmentation boundaries when classification networks are used for correspondence matching.The paper enforces smoothness through a loss based on integrating multiple random segmentations.
- Conclusion: The segmentation-based training scheme significantly decreases the amount of training data needed by avoiding exhaustive pairwise distance computation between descriptors.The comparison is with methods trained on pairs or triplets of samples.
- Conclusion: The framework unifies treatment of human body shapes and clothed subjects and handles both full models and partial scans such as single-view depth maps.The paper describes this as its first such framework compared with existing classification networks.
- Conclusion: The method is particularly effective for accurate dense correspondence between partial scans of clothed subjects in arbitrary poses.This conclusion is stated as the central demonstrated outcome.
- Limitations: The supervised framework cannot handle arbitrary shapes because its prior is entirely based on the class of training data.Future work proposes additional body shapes, more occlusion examples, simulated garments, improved filtering, and larger training sets.
- Limitations: Wrong correspondences remain difficult to prune in some cases, negatively affecting surface registration.The paper suggests more sophisticated filtering and global treatment of multiple input shapes as possible improvements.
Appendix I. Comparison
On FAUST intra- and inter-subject pairs, the method establishes state-of-the-art correspondences using per-vertex descriptors obtained by aggregating multi-view depth-map features. Nearest-neighbor matching in feature space already outperforms existing global shape-matching methods and is comparable to state-of-the-art non-rigid registration.
- Comparison: The method achieves state-of-the-art performance on intra-subject and inter-subject FAUST correspondence pairs.Evaluation uses the same set of randomly sampled pairs as Chen et al. [10].
- Method: Per-vertex descriptors are computed by rendering each 3D scan from multiple viewpoints and averaging the resulting per-pixel feature descriptors.Correspondences are then established by nearest-neighbor search in feature space.
- Comparison: The direct descriptor-matching method significantly outperforms existing global shape-matching methods and is comparable to a state-of-the-art non-rigid registration method.The compared global methods do not require initial poses as input.