Source-linked AI summary
Human Body Model Fitting by Learned Gradient Descent
Jie Song, Xu Chen, Otmar Hilliges
TL;DR
Recovering 3D human pose and shape from a single image requires inferring model parameters from indirect observations. This paper learns per-parameter updates for iterative fitting from pose and shape data, and reports improved convergence, speed, and accuracy over SMPLify, with state-of-the-art results on challenging in-the-wild evaluation.
Problem
Recovering 3D human pose and shape from a single image is an ill-posed problem because model parameters must be inferred from indirect, low-dimensional image observations.
Source: §1 · p. 1Method
A deep network predicts per-parameter, state-dependent updates for iterative fitting, trained using human pose and shape data rather than images.
Source: §1 · p. 3Results
The method improves convergence, speed, and accuracy over SMPLify when using the same MoCap data, and reports state-of-the-art results especially on 3DPW.
Source: §1 · p. 3, §5 · p. 13Takeaways & Limitations
The authors suggest improved optimization-based fitting could help provide more training data by fitting 2D-annotated images.
Source: §5 · p. 14
Abstract
from arXiv · showhide
We propose a novel algorithm for the fitting of 3D human shape to images. Combining the accuracy and refinement capabilities of iterative gradient-based optimization techniques with the robustness of deep neural networks, we propose a gradient descent algorithm that leverages a neural network to predict the parameter update rule for each iteration. This per-parameter and state-aware update guides the optimizer towards a good solution in very few steps, converging in typically few steps. During training our approach only requires MoCap data of human poses, parametrized via SMPL. From this data the network learns a subspace of valid poses and shapes in which optimization is performed much more efficiently. The approach does not require any hard to acquire image-to-3D correspondences. At test time we only optimize the 2D joint re-projection error without the need for any further priors or regularization terms. We show empirically that this algorithm is fast (avg. 120ms convergence), robust to initialization and dataset, and achieves state-of-the-art results on public evaluation datasets including the challenging 3DPW in-the-wild benchmark (improvement over SMPLify 45%) and also approaches using image-to-3D correspondences
1 Introduction
Recovering 3D human pose and shape from a single image is an ill-posed inverse problem, while existing optimization methods can be slow and difficult to tune. The paper proposes a learned update rule for iterative fitting, trained with human pose and shape data rather than images.
1 Introduction
- Single-image 3D pose and shape recovery infers model parameters from indirect, low-dimensional observations and is generally ill-posed.The introduction frames this as a longstanding problem with downstream applications.Source: §1 · p. 1
- Iterative fitting can be slow and hard to solve because inversion is non-convex, prior terms create competing minima, and heuristics require domain knowledge.The authors also identify iterative refinement, no 3D-annotated training images, and reduced dataset overfitting as appealing properties of optimization.Source: §1 · pp. 2–3
- The paper replaces hand-crafted gradient descent with a deep network predicting per-parameter, state-dependent updates for fitting a human model to 2D observations.The stated aim is to make iterative optimization competitive with regression methods that use extensive annotated image data.Source: §1 · p. 3
- Training uses human pose and shape data without images; the network learns updates that keep fitting on the natural-pose-and-shape manifold and can recover from local minima.The method typically converges in ≤4 iterations; at inference it directly optimizes 2D reprojection error without further priors or regularization.Source: §1 · p. 3
- The approach is presented as fast, accurate, and robust, with state-of-the-art results especially on the challenging in-the-wild 3DPW dataset.The contribution summary also describes its training as requiring only 3D MoCap data and no image-to-pose correspondences.Source: §1 · p. 3
2 Related Work
Prior human-model fitting uses optimization with keypoints or silhouettes, while regression methods estimate model parameters from images but can align coarsely. The paper situates its approach among learned-gradient methods and combines parameter updates learned by a network with iterative fitting.
2 Related Work
- Earlier optimization-based human-pose methods used silhouettes or 2D keypoints, and SMPLify automatically fits SMPL parameters to detected keypoints with strong priors.Related fitting pipelines were generally not real-time and could be sensitive to initialization and difficult to tune.Source: §2 · p. 4
- Direct regression methods estimate human-model parameters from a single RGB image, but can produce coarse image-model alignment and depend on scarce, costly image-to-3D correspondences.The related-work discussion also notes that optimization-based pseudo-labeling remains bounded by SMPLify's runtime and local-minimum accuracy.Source: §2 · p. 4
- Learned-gradient methods have been studied across domains: Andrychowicz et al. applied learned gradients to classification, while follow-up work advanced a partially learned gradient scheme for nonlinear tomographic inversion with simulated data.Source: §2 · p. 4
- This work combines learning and iterative model fitting: a network uses gradients and current parameters to generate updates that keep poses and shapes natural while taking larger steps.The cited inference pipeline starts from 2D detections, initializes parameters at zero, computes reprojection error and its gradient, then iteratively updates the estimate.Source: §2 · p. 5, Figure 2 · p. 5
3 Method
The method fits SMPL parameters to 2D keypoints by replacing conventional gradient steps with learned, state-aware updates. It trains these updates using MoCap-derived body meshes and reprojection-generated supervision.
Problem Setting
- SMPL maps pose and shape parameters to a triangulated mesh, while reconstruction parameters also include camera rotation, translation, and scale for projecting joints to 2D.The model uses a weak-perspective camera with orthographic projection.Source: §3 · pp. 5–6
Iterative Optimization with Explicit Regularization
- The inverse problem is ill-posed because model parameters outnumber 2D measurements; conventional regularized fitting, including SMPLify, is slow and prone to local minima.SMPLify uses several pose and shape priors and a staged quasi-Newton fitting procedure.Source: §3 · pp. 6–7
Learned Gradient Descent for Human Shape Recovery
- The Gradient Updating Network replaces standard gradient descent with parameter-specific updates that combine gradient and model state to learn priors and regularization.These adaptive updates aim to keep fitting on the natural pose-and-shape manifold, recover from local minima, and take larger steps.Source: §3 · p. 7
- At inference, a joint detector supplies 2D keypoints, and the network repeatedly updates SMPL parameters using the reprojection-loss gradient and current state, usually in fewer than four iterations.The initial parameters are zero-valued; each update is added to the current estimate.Source: §3 · pp. 7–8
Training
- Training uses MoCap-derived SMPL meshes, randomly sampled poses, shapes, and cameras, with reprojection-generated keypoints and parameter error as the training loss.Joint dropout bridges perfect projected keypoints and noisy detections with missing joints; no image data is used for training.Source: §3 · pp. 7, 9, Figure 3 · p. 8
4 Experiments
Experiments compare training-data requirements, dataset performance, ablations, convergence, and speed. The method performs strongly on 3DPW and EHF, and converges substantially faster than SMPLify.
4.1 Training Data Comparison
- Training uses 3D human meshes across poses and shapes, with MoCap data shared with SMPLify; the authors note that competing learning methods use substantially more annotated data.Those methods use additional datasets with 2D and 3D annotations; the comparison is framed around methods without paired image-to-3D training data.Source: §4.1 · p. 9
4.2 Test Datasets
- Evaluation spans indoor Human 3.6M, challenging outdoor 3DPW with a pure hold-out test set unused for training, and EHF’s 100 images with whole-body vertex-level annotations for real-world assessment.Human 3.6M reports MPJPE after Procrustes alignment using every fifth frame from frontal camera 3; 3DPW reports test-set 3D pose reconstruction error, while EHF enables vertex-to-vertex error.Source: §4.2 · pp. 9–10
4.3 Ablation and Iteration Studies
- Using all three update-network inputs performs best; accuracy improves as training iterations increase, then shows no further improvement after four iterations.The inputs are target 2D pose, current estimate, and unmapped gradient; the iteration study varies training updates from one to five.Source: §4.3 · p. 10, Table 1 · p. 10, Table 2 · p. 10
- 67.8 reconstruction error for direct 2D-to-3D lifting is 15% worse than the proposed method, which the authors interpret as evidence for gradient-based refinement over simple lifting.Direct lifting is non-iterative and does not use gradients as input.Source: §4.3 · p. 11
4.4 Comparison with Other Methods
- On H3.6M, the method achieves state-of-the-art performance among methods that output a human-body mesh; on EHF, it records 54.7 mm vertex-to-vertex error and outperforms SMPLify variants.H3.6M comparisons include methods that do not require image-to-3D annotations; EHF provides vertex-level evaluation.Source: Table 3 · p. 11, §4.4 · pp. 11, 13, Table 5 · p. 13
- 45% over SMPLify on 3DPW, with state-of-the-art performance even against regression methods using additional 3D supervision and pseudo-labels.The dataset is held out for testing and is presented as a better indicator of real-world performance.Source: Table 4 · p. 12, §4.4 · p. 13
4.5 Qualitative Results
- Qualitative comparisons show better alignment of details such as lower limbs, with consistent behavior across datasets.The cited examples compare iterative refinement with regression-based methods.Source: §4.5 · p. 13
4.6 Speed and Model Size.
- 120 ms average convergence versus SMPLify's 1 to 2 minutes, a 500x speed-up attributed to five versus 100 iterations and first-order rather than second-order derivatives.The gradient updating network has 4.3M parameters and 8M FLOPs.Source: §4.6 · p. 13
5 Conclusion
The conclusion presents learned gradient descent as a replacement for hand-tuned updates and reports improved convergence, speed, and accuracy over non-convex optimization methods using the same MoCap data.
5 Conclusion
- A deep network replaces the hand-tuned step size with per-parameter, state-dependent updates.Source: §5 · p. 13
- The method significantly improves convergence, speed, and accuracy over non-convex optimization methods such as SMPLify when using the same MoCap data.Source: §5 · pp. 13–14
- The iterative update scheme produces results that align better with the 2D observations than other methods.Source: Figure 5 · p. 14
- Qualitative results span Human 3.6M, 3DPW, EHF, and an internet dancing video, which the authors describe as generalization to random input sources.Source: Figure 6 · p. 14
- Improved optimization methods could provide more training data by fitting images with 2D annotations.Source: §5 · p. 14