Source-linked AI summary
DiffProxy: Multi-View Human Mesh Recovery via Diffusion-Generated Dense Proxies
Renke Wang, Zhenyu Zhang, Ying Tai, Jun Li, Jian Yang
TL;DR
Multi-view human mesh recovery is challenged by entangled errors in end-to-end prediction and limited surface constraints from sparse-keypoint fitting. DiffProxy uses a diffusion-based generator to predict dense correspondences from multi-view images, with synthetic training, hand refinement, and uncertainty-guided fitting. It achieves state-of-the-art performance across five diverse real-world benchmarks.
Problem
End-to-end HMR can produce entangled errors, while fitting-based methods rely on sparse keypoints that provide limited surface constraints.
Source: §1 · p. 2Method
Results
DiffProxy achieves state-of-the-art performance across five diverse real-world benchmarks when trained exclusively on synthetic data.
Source: §1 · p. 3, §6 · p. 15Takeaways & Limitations
Diffusion-generated dense proxies provide uniform visible-surface constraints that support precise fitting without dataset-specific tuning.
Source: §6 · p. 15Without hand refinement or test-time scaling, inference takes about 53 seconds per subject, and hand refinement and scaling add further cost.
Source: §5 · p. 14
Abstract
from arXiv · showhide
Precise human mesh recovery (HMR) from multi-view images remains challenging: end-to-end methods produce entangled errors hard to localize, while fitting-based methods rely on sparse keypoints that provide limited surface constraints. We observe that the true bottleneck lies in the quality of intermediate representations, and that dense pixel-to-surface correspondences can be effectively generated by repurposing pre-trained diffusion models with rich visual priors. We propose DiffProxy, a Stable-Diffusion-based framework trained on large-scale synthetic data with pixel-perfect annotations. A multi-conditional proxy generator predicts dense correspondences from multi-view images, providing uniform surface constraints that enable precise fitting. Hand refinement feeds enlarged hand crops alongside full-body images for fine-grained detail, while test-time scaling exploits diffusion stochasticity to estimate per-pixel uncertainty. Trained only on synthetic data, DiffProxy achieves state-of-the-art results on five diverse real-world benchmarks. Project page: https://wrk226.github.io/DiffProxy.html
1 Introduction
DiffProxy revisits fitting-based human mesh recovery by using diffusion-generated dense pixel-to-surface correspondences in place of sparse constraints. Its synthetic-data-trained pipeline adds hand refinement and uncertainty-guided fitting, and achieves state-of-the-art results across five diverse real-world benchmarks.
1 Introduction
- Dense correspondences provide more constraints per view than sparse keypoints, addressing limited surface coverage and the sensitivity of fitting to erroneous or occluded joints.Earlier dense-proxy approaches were hindered by noisy manual annotations and CNN detectors with poor out-of-distribution generalization.Source: §1 · p. 2
- The multi-view proxy generator predicts dense pixel-to-surface correspondences for fitting, while enlarged hand crops and stochastic predictions support hand refinement and per-pixel uncertainty weighting.Unreliable correspondences are down-weighted during fitting.Source: §1 · p. 3
- The approach trains on 105K synthetic samples and 844K images with pixel-perfect SMPL-X annotations, and uses Stable Diffusion 2.1 for synthetic-to-real transfer.The synthetic annotations are intended to eliminate label noise found in manual datasets.Source: §1 · p. 3
- Trained exclusively on synthetic data, DiffProxy achieves state-of-the-art results across five diverse real-world benchmarks.The paper attributes this to dense proxies generated using Stable Diffusion’s visual priors, alongside hand refinement and test-time scaling.Source: §1 · p. 3
2 Related Work
Prior HMR work includes fast end-to-end regression, fitting against sparse cues, and methods that predict dense correspondences. DiffProxy combines synthetic pixel-perfect supervision and diffusion priors with multi-view consistency to generate dense proxies for fitting.
End-to-end and fitting-based human mesh recovery
- End-to-end regression is fast but can entangle geometry and correspondence errors; fitting-based approaches instead optimize body models against detected cues, often relying on sparse keypoints.Multi-view keypoint fitting triangulates 2D detections, but remains sensitive to outliers and offers limited surface constraints.Source: §2 · pp. 3–4
Dense correspondence for mesh recovery
- Earlier dense-correspondence methods used noisy DensePose-COCO annotations or deterministic CNN detectors, while DiffProxy uses 105K synthetic multi-view samples with pixel-perfect labels and diffusion priors.DiffProxy also applies epipolar attention for multi-view consistency.Source: §2 · p. 4
Diffusion models for dense prediction
- Pre-trained Stable Diffusion models have been adapted for monocular depth and surface normals, and epipolar-constrained attention has been explored for cross-view interaction.DiffProxy applies these ideas to human pixel-to-surface correspondence prediction for fitting.Source: §2 · p. 4
Synthetic data for human mesh recovery
- Synthetic datasets with precise SMPL-X annotations can approach or match real-data performance; DiffProxy adds multi-view rendering, varied scenes, and dense annotations, with reported generalization to five real-world benchmarks.The described scene diversity includes realistic occlusions, varied lighting, and clothing simulation.Source: §2 · p. 4
3 Method
DiffProxy learns dense SMPL-X surface correspondences from synthetic multi-view data, then fits a mesh to those proxies using reliability-weighted reprojection. Its pipeline adds enlarged hand crops and stochastic proxy sampling to refine details and account for uncertain predictions.
3 Method
- The pipeline predicts initial body proxies, refines hand regions in a second pass, estimates per-pixel reliability from stochastic samples, then fits SMPL-X parameters by weighted reprojection.This combines proxy generation, hand refinement, test-time scaling, and fitting in one multi-view workflow.Source: Figure 2 · p. 5
3.1 Synthetic Data Preparation
- The synthetic dataset contains 105,487 multi-view samples and 843,896 rendered images, with pixel-perfect SMPL-X proxy annotations.Subjects are rendered from BEDLAM and SynBody sources using synthetic visual content and multiple camera views.Source: §3.1 · pp. 4–6
3.2 Human Proxy Generation
- Each SMPL-X proxy pairs semantic body-part labels with surface UV coordinates, forming a dense image representation of pixel-to-surface correspondences.Segmentation and UV components are encoded as three-channel images to match the pretrained VAE decoder.Source: §3.2 · p. 6
- The generator uses Stable Diffusion 2.1 with a frozen UNet, conditioning on text, T2I-Adapter features, and DINOv2 tokens for multi-view proxy prediction.Trainable cross-modality and epipolar attention modules support modality and view consistency; the UNet and DINOv2 remain frozen.Source: §3.2 · pp. 6–7
- Training uses a diffusion noise-prediction objective on VAE-encoded ground-truth proxies, with four views per sample combining full-body images and hand crops.The view budget uses two to four full-body views and fills remaining slots with randomly selected hand-region crops.Source: §3.2 · p. 7
- Hand refinement localizes hands from initial segmentation, enlarges their crops, and treats them as additional views with hand-specific prompts to improve finger fidelity.The second pass uses cross-view attention without modifying the network architecture.Source: §3.2 · p. 7
- Test-time scaling aggregates stochastic proxy samples and uses their disagreement to estimate per-pixel uncertainty, down-weighting unreliable predictions during optimization.The number of samples K provides a compute–accuracy trade-off without test-time adaptation of network weights.Source: §3.2 · pp. 7–8
3.3 Human Mesh Recovery
- Mesh fitting minimizes reliability-weighted reprojection error over foreground pixels, with a DPoser-X pose prior weighted by λ_prior = 0.1.Each proxy label and UV coordinate maps an image pixel to a 3D SMPL-X surface point for reprojection.Source: §3.3 · p. 8
4 Experiments
DiffProxy is evaluated on unseen real-world datasets using four views per dataset and achieves state-of-the-art performance across all six evaluation settings. Comparisons and ablations examine dense proxies, synthetic-data generalization, hand refinement, test-time scaling, and view count.
4.1 Implementation Details
- Training starts from Stable Diffusion 2.1 weights, updates attention modules and the T2I-Adapter, and freezes the UNet backbone and DINOv2 encoder.The proxy generator is trained with four views per sample; the VAE decoder is separately refined for proxy representations requiring high numerical precision.Source: §4.1 · p. 9
4.2 Datasets and Baselines
- Evaluation covers five unseen real-world datasets, with four views per dataset and temporal subsampling to reduce redundancy.The datasets span studio capture, human-object interaction, outdoor scenes, challenging poses, and loose clothing; 4D-DRESS partial tests random crops.Source: §4.2 · p. 9, §4.3 · pp. 9–10
4.3 Quantitative Results
- DiffProxy achieves state-of-the-art performance across all six evaluation settings.Table 1 compares five real-world datasets and includes 4D-DRESS partial; evaluation uses four views per dataset.Source: §4.3 · p. 10, Table 1 · p. 10
- On RICH, MPVPE is 35.1 mm for DiffProxy, 60.3 mm for EasyMoCap, and 81.1 mm for SMPLest-X.The base model without hand refinement or test-time scaling already surpasses end-to-end methods on most metrics using the same four-view input as multi-view baselines.Source: §4.3 · pp. 10–11
- 53.3 MPVPE on 3DHP for HeatFormer drops to 123.6 on MoYo and 327.0 on 4D-DRESS partial, while DiffProxy performs strongly across benchmarks without real training data.The synthetic training set contains 105K multiview samples.Source: §4.3 · p. 10
4.4 Qualitative Results
- Qualitative comparisons show tight image-mesh alignment across 3DHP, RICH, and 4D-DRESS partial, including cluttered scenes and heavy occlusion.On 3DHP, DiffProxy aligns more tightly despite lower benchmark scores; dense proxies also remain robust to RICH distractors and partial visibility.Source: Figure 5 · p. 11, §4.4 · pp. 11–12
4.5 Ablation Studies
- Hand refinement improves fitting and finger detail, while stochastic disagreement estimates uncertainty for down-weighting unreliable pixels; increasing K generally improves performance.Hands occupy less than 1% of full-body image pixels, and the default test-time scaling setting is K = 3.Source: §4.5 · p. 12
- More input views progressively improve recovery, modules each contribute to performance, and camera-free fitting remains competitive with moderate degradation.Fine-tuned DensePose improves on standard poses but remains fragmented on challenging poses; DiffProxy can nevertheless converge to depth-flipped local optima despite correct 2D appearance.Source: §4.5 · p. 13, Figure 8 · p. 13, Table 2 · p. 13
5 Limitation and Future Works
DiffProxy has fitting, speed, and input-view limitations. Its 2D reprojection objective can converge to depth-inverted poses, single-view results suffer from depth ambiguity, and refinement and test-time scaling add cost.
5 Limitation and Future Works
- Fitting can converge to depth-inverted poses despite correct 2D appearance; reliable results require multiple views, and hand refinement and test-time scaling add inference cost.Without those refinements, inference takes about 53 seconds per subject; the authors also identify single-view depth ambiguity as a limitation.Source: §5 · p. 14
6 Conclusion
DiffProxy treats dense pixel-to-surface prediction as an image-to-image translation task suited to diffusion models. Trained exclusively on synthetic data, it achieves state-of-the-art performance across five diverse real-world benchmarks.
6 Conclusion
- Trained exclusively on synthetic data without dataset-specific tuning, DiffProxy achieves state-of-the-art performance across five diverse real-world benchmarks.The authors describe this as evidence that the dense proxy route is effective for precision-oriented tasks.Source: §6 · p. 15
- Large-scale synthetic data with pixel-perfect annotations and Stable Diffusion visual priors address noisy manual labels and limited generalization in prior dense-proxy approaches.The authors frame dense correspondence prediction as image-to-image translation well-suited to modern diffusion models.Source: §6 · p. 15
- Hand refinement adds fine-grained detail, while test-time scaling estimates uncertainty to guide fitting with dense proxies that provide uniform surface constraints.These components accompany fitting for precise mesh recovery.Source: §6 · p. 15