PaperlayerView exact revision on arXiv

Source-linked AI summary

DreamHMR: From Video Generation to Human Motion Modeling via Structural Focusing

Zhengjie Xu, Ruihao Gong, Xianglong Liu

arXiv:2610.20124v1cs.CVcs.AI

TL;DR

Occlusion and partial visibility make complete body-and-hand recovery from video difficult, while video-generation representations are trained for synthesis rather than explicit human geometry. DreamHMR adapts those representations with complete human renderings and evidence-gated local refinement. Across standard benchmarks and held-out occlusion sequences, it reports lower reconstruction errors, including the stated body and hand MPJPE reductions.

  • Problem

    Occlusion and partial visibility leave incomplete, uneven visual evidence, while video-generation representations target visual synthesis rather than explicit human pose and surface geometry.

    Source: §1 · p. 1

  • Method

    DreamHMR adapts video-generation representations with complete human renderings for geometry supervision, then uses evidence-gated local features to refine body-and-hand estimates.

    Source: §1 · p. 2

  • Results

    4.2%, 4.7%, and 6.1% lower body MPJPE on 3DPW, EMDB-1, and RICH, respectively; on held-out BEDLAM and EgoBody occlusion sequences, overall body and hand MPJPE fall by 7.0% and 10.0%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Across four video backbones, DreamHMR improves reconstruction under occlusion and attains the lowest errors on the evaluated HMR benchmarks.

    Source: §5 · p. 10

Abstract

from arXiv · show

Human mesh recovery (HMR) provides a geometric foundation for understanding human motion, but accurately reconstructing both body motion and fine hand articulation from videos remains challenging, especially in real-world scenes where occlusion and partial visibility are common. Video generation pretraining offers rich spatial and temporal priors for this task, but its focus on visual synthesis leaves a gap between these representations and explicit human geometry. We present DreamHMR, a general framework that adapts video generation priors for complete body-and-hand reconstruction. It follows a global-to-local strategy, first grounding video representations in human geometry and then refining body and hand detail with reliable local evidence. Global Body Representation Transfer trains the video backbone to predict complete human renderings, turning its native prediction task into a means of learning surface correspondence and posed geometry. Local Joint Feature Enhancement combines contextual estimates with reliable local evidence to recover fine body and hand detail, while limiting interference from occluders and background. Experiments on five datasets with four video backbones demonstrate strong reconstruction performance across backbone families. Under a unified evaluation protocol, our best model outperforms the best existing methods on all reported metrics of 3DPW, EMDB-1, and RICH, reducing body MPJPE by 4.2%, 4.7%, and 6.1%, respectively. In more challenging occlusion settings on held-out BEDLAM and EgoBody sequences, DreamHMR reduces overall body and hand MPJPE by 7.0% and 10.0%, respectively. Our code will be made publicly available.

1 INTRODUCTION

Complete body-and-hand motion recovery remains difficult when real-world videos contain occlusion and partial visibility. DreamHMR adapts video-generation representations with complete human geometry supervision and selective local refinement, and reports gains across benchmarks and occlusion settings.

1 INTRODUCTION

  • Occlusion and partial visibility leave incomplete, uneven evidence across body parts and frames, making complete body-and-hand motion difficult to recover from real-world video.Local joint features can come from an occluder or background rather than the target person.Source: §1 · p. 1
  • Video-generation models capture spatial appearance and temporal dependencies, but their synthesis objective does not directly target human pose or surface geometry.DreamHMR adapts those representations to human structure.Source: §1 · p. 2
  • DreamHMR adapts video-generation priors for complete body-and-hand recovery using complete SMPL-X renderings and evidence-gated local refinement.Canonical surface colors encode correspondence across poses, shading conveys posed geometry, and supervised evidence scores weight local updates to limit occluder or background influence.Source: §1 · p. 2
  • 4.2%, 4.7%, and 6.1% lower body MPJPE on 3DPW, EMDB-1, and RICH, respectively, under a unified protocol; occlusion-setting body and hand MPJPE fall by 7.0% and 10.0%.Experiments span five datasets and four video backbones, with gains extending across backbone families.Source: §1 · p. 2

2 RELATED WORK

Prior work studies image- and video-based mesh recovery, generative representations for reconstruction, and recovery under occlusion. DreamHMR uses a shared video-generation representation with complete rendered human structure for body-and-hand recovery.

2 RELATED WORK

  • Human mesh recovery estimates parametric bodies from images, while video models add temporal context; whole-body and hand-specific methods address articulated body-and-hand recovery at different levels of detail.The passage also describes methods using prompted temporal transformers, motion distributions, and larger training data or model capacity.Source: §2 · p. 2
  • Image diffusion features support correspondence estimation, while video diffusion features have been used for motion and geometry tasks; DPMesh and ADHMR apply diffusion to occluded mesh recovery.DreamHMR adapts video-generation features with complete human renderings to encode surface correspondence and posed geometry.Source: §2 · p. 2
  • Occlusion methods use part-guided aggregation, part-mesh fusion, structured human and joint tokens, or local image-aligned hand features; nearby features may instead describe background or another person.DreamHMR learns supervised evidence scores to weight local updates.Source: §2 · p. 3

3 METHOD

DreamHMR predicts complete SMPL-X motion from video-generation features, adapting them with complete rendered human structure and refining anatomical states with local image evidence. Its pipeline combines global context, evidence-weighted joint features, and temporal recovery.

3.1 OVERALL FRAMEWORK

  • DreamHMR combines Global Body Representation Transfer, which adapts video features with complete rendered human structure, and Local Joint Feature Enhancement, which refines anatomical states with local image evidence.Given target-person RGB, observation masks, and frozen-tracker boxes, it predicts complete SMPL-X motion, shape, and presence.Source: §3 · p. 3, §3.1 · p. 3

3.2 GLOBAL BODY REPRESENTATION TRANSFER

  • Complete geometry supervision adapts the video backbone to predict complete projected human structure across the full image canvas, including regions outside the observation mask.Canonical template coordinates provide colors identifying the same surface location across poses; current normals and fixed-light shading add pose-dependent geometry, and the rendered target is composited with source RGB across frames.Source: §3.2 · pp. 3–5

3.3 LOCAL JOINT FEATURE ENHANCEMENT

  • The decoder uses part states to summarize complete body-part context and joint descriptors to locate joints and retrieve nearby image evidence.Four part queries and 52 joint queries extract features; local sampling averages valid features around predicted locations, and evidence scores weight the local corrections before they are fused into anatomical states.Source: §3.3 · pp. 5–6
  • Evidence scores are supervised with labels marking whether a projected joint lies within both the observation region and the target person's visible instance mask.Balanced BCE averages losses over supported joints inside the observation region, unsupported joints inside it, and joints outside it; reconstruction gradients also train the evidence head through the gated update.Source: §3.3 · p. 5

3.4 TRAINING AND INFERENCE

  • Training first adapts the DiT with rendered-video flow matching, then jointly trains the backbone and structured decoder for parameter recovery; inference encodes RGB once and decodes SMPL-X parameters and presence.Recovery supervision covers pose, position, shape, joint localization, temporal velocity, and presence, while shape is shared across the window.Source: §3.4 · p. 6, §3.3 · p. 6

4 EXPERIMENTS

Experiments evaluate DreamHMR on standard HMR benchmarks and held-out occlusion settings, then analyze rendering supervision, local evidence mechanisms, and temporal context. Results show improvements across benchmarks and video-backbone families, including in occluded body-and-hand reconstruction.

4.1 EXPERIMENTAL SETUP

  • The experiments span five datasets and four video backbones, with standard HMR evaluation and held-out occlusion evaluation alongside ablations.Occlusion evaluation and ablations use separate BEDLAM and EgoBody sequence holdouts disjoint from training and validation.Source: §4 · p. 6, §4.1 · p. 6

4.2 EVALUATION ON WIDELY USED HMR BENCHMARKS

  • Wan-5B improves every reported HMR benchmark metric, reducing MPJPE by 2.3 mm on 3DPW, 2.9 mm on EMDB-1, and 3.2 mm on RICH.PVE reductions are 1.3, 3.1, and 3.5 mm, respectively.Source: §4.2 · p. 6, Table 1 · p. 7

4.3 BODY-AND-HAND RECONSTRUCTION UNDER OCCLUSION

  • All four DreamHMR backbones improve on the strongest external overall body result; Cosmos-2B ranks second, while both Wan models give the strongest joint body-and-hand results.Wan-1.3B is used for the subsequent design analyses.Source: §4.3 · p. 7
  • Wan-5B reaches 107.1/55.1 mm body/hand MPJPE under occlusion, beating 3DB-DINOv3 by 8.1/6.1 mm.It also has the lowest error in every source-specific column, including EgoBody’s real interaction videos; Figure 3 shows a close-up where DreamHMR keeps fingers wrapped around the bowl and food while SMPLest-X lays them against the torso.Source: §4.3 · pp. 6–7, §4.2 · p. 6, Figure 3 · p. 7

4.4 ABLATION STUDIES

  • Replacing RGB supervision with canonical coloring plus shading reduces occlusion body/hand MPJPE from 120.0/68.4 to 106.3/57.8 mm and body PVE from 140.6 to 121.8 mm.The combined target also yields the lowest Full-input errors: 53.8/30.7 mm MPJPE and 61.3 mm PVE; canonical coloring and shading outperform either alone and shading with a skeleton.Source: Table 3 · p. 8, §4.4 · p. 8, Figure 4 · p. 8
  • Structured decoding reduces occlusion body/hand MPJPE from 124.4/65.0 to 116.9/60.5 mm, and local sampling further reduces it from 119.3/63.9 to 116.9/60.5 mm.The ablation compares local updates absent, ungated, or weighted by predicted evidence scores.Source: §4.4 · pp. 8–9, Table 4 · p. 9
  • Evidence supervision plus gating reduces occlusion body/hand MPJPE from 113.1/59.6 to 106.3/57.8 mm; adding supervision with gating reduces it from 110.7/59.0 to 106.3/57.8 mm.The full model predicts evidence scores to weight local corrections and supervises those scores with binary evidence labels.Source: §4.4 · p. 9

4.5 TEMPORAL CONTEXT UNDER OCCLUSION

  • Removing neighboring RGB frames increases center-frame body/hand MPJPE by 12.71/9.27 mm under occlusion, versus 3.26/3.58 mm without additional masking.This diagnostic retains original tracks and timestamps and scores only the center frame, so its errors are not directly comparable with full-window ablations.Source: Table 5 · p. 9, §4.5 · p. 9

5 CONCLUSION

DreamHMR adapts video-generation representations for complete body-and-hand recovery under occlusion. Complete renderings transfer geometry, while evidence-gated local features refine body-part estimates; the method improves reconstruction across backbones and evaluated benchmarks.

5 CONCLUSION

  • Complete human renderings transfer surface correspondence and posed geometry into video representations, while evidence-gated local features refine contextual body-part estimates.The framework combines global geometry supervision with local refinement for complete body-and-hand recovery under occlusion.Source: §5 · p. 10
  • Across four video backbones, DreamHMR improves reconstruction under occlusion and attains the lowest errors on the evaluated HMR benchmarks.Source: §5 · p. 10
  • DreamHMR adapts video-generation representations for complete body-and-hand recovery under occlusion.Source: §5 · p. 10

AI USE STATEMENT

The authors report using generative AI for implementation, translation, data processing, and manuscript-related tasks, while excluding it from theoretical and methodological research decisions. They also report human review of AI-assisted outputs and retain responsibility for the final work.

AI USE STATEMENT

  • Generative AI assisted method implementation, translation, and dataset cleaning and reformatting, but not theory development, hypothesis work, methodology, experiments, or results interpretation.Source: AI USE STATEMENT · p. 10
  • The authors also used LLMs for figures, code, research artifacts, readability, literature identification, and information search.Source: AI USE STATEMENT · p. 10
  • The authors report checking AI-assisted code and data processing, verifying figures, reviewing translations and edits, and taking responsibility for the final content.Source: AI USE STATEMENT · p. 10

A IMPLEMENTATION DETAILS

The implementation adapts several video backbones through a shared decoder and trains them in two stages, then recovers structured body-and-hand parameters with temporal refinement and supervised losses.

A IMPLEMENTATION DETAILS

  • Four video backbones encode observed RGB using frozen VAEs and provide final Transformer features before native velocity heads; linear projection plus LayerNorm maps them to the shared 512-D decoder space.LTX features are resampled along time by nearest-neighbor interpolation to match the 21 decoder endpoints, while retaining their spatial grid.Source: A IMPLEMENTATION DETAILS · p. 14
  • The structured decoder samples 3×3 neighborhoods around predicted joint locations and uses evidence scores to weight local corrections before joint-to-part fusion.Its part and joint query branches use two attention layers, while the evidence head is an MLP with a sigmoid output.Source: A IMPLEMENTATION DETAILS · p. 14
  • Part-specific heads predict rotations, camera-frame pelvis position, and presence at 21 endpoints; interpolated outputs receive residual temporal refinement, while shared shape is predicted separately.The final projection of the temporal block is initialized to zero.Source: A IMPLEMENTATION DETAILS · p. 14
  • Training first uses rendered-video flow matching, then jointly trains the backbone and structured decoder for parameter recovery while freezing the velocity head.Training samples full, static, and dynamic observations in a 2:1:1 ratio and uses the terminal checkpoint.Source: A IMPLEMENTATION DETAILS · p. 14
  • The recovery objective combines rotation, joint, root, and shape losses, with supervised-part averaging and availability masks governing loss aggregation.Rotation, relative-joint, localization, and relative-velocity losses average within body and hand groups before equally averaging supervised parts.Source: A IMPLEMENTATION DETAILS · pp. 14–15
  • Additional supervision covers descriptor locations, temporal velocities, presence, and evidence support, with valid annotations and frame pairs determining which terms contribute.Support supervision balances supported joints inside the observation region, unsupported joints inside it, and joints outside it.Source: A IMPLEMENTATION DETAILS · pp. 15–16

B OBSERVATION CONSTRUCTION AND EVALUATION

The evaluation protocol constructs full, static, and moving partial observations, defines how tracks and annotations supply supervision, and separates occlusion holdouts from training and validation. Rendering details specify how complete, camera-aligned human targets are formed from canonical surface colors and pose-dependent shading.

B OBSERVATION CONSTRUCTION AND EVALUATION

  • Static observations comprise eight fixed templates, while dynamic observations move local rectangles from four initial positions by up to 0.35w or 0.35h; pixels outside regions are set to neutral gray.Figure 5 illustrates the construction.Source: B OBSERVATION CONSTRUCTION AND… · p. 16
  • A frozen detector and tracker provide target tracks; presence and joint evidence labels depend on visible target pixels, observation-region overlap, and aligned visible-instance masks.Complete geometry is supervised wherever its annotations are available.Source: B OBSERVATION CONSTRUCTION AND… · p. 17
  • Occlusion and ablation holdouts each contain 4,080 windows from disjoint BEDLAM and EgoBody sequences, separated from training and validation by identity, session, motion, and scene.The public benchmark evaluation uses official 3DPW, EMDB-1, and RICH test sets with original RGB inputs.Source: B OBSERVATION CONSTRUCTION AND… · p. 17
  • Compared methods share observed RGB and target tracks, use released inference models without test-time flipping or extra smoothing, and are evaluated with common joint mappings and specified alignment protocols.Occlusion body and hand MPJPE use pelvis- and wrist-relative joints, respectively, with equal left/right hand weighting.Source: B OBSERVATION CONSTRUCTION AND… · p. 17

C RENDERING DETAILS

  • The renderer forms a complete camera-aligned human video from annotated SMPL-X pose and shape, using the input calibration and a canvas aligned with the RGB frame.The camera-frame mesh places the pelvis at its annotated position.Source: C RENDERING DETAILS · p. 18
  • Canonical surface colors identify the same template location across poses, while shading from current mesh normals represents pose-dependent orientation and brightness.Rasterization resolves visibility, and the rendered mesh replaces source pixels only where the posed human is visible; coverage is independent of the observation mask.Source: C RENDERING DETAILS · pp. 18–19

D.1 EFFECT OF BACKBONE INITIALIZATION

The ablations compare video-backbone initialization choices and test evidence gating under static and dynamic observations. Pretrained Wan initialization and evidence gating each reduce reconstruction errors in the reported settings.

D.1 EFFECT OF BACKBONE INITIALIZATION

  • 13.53/8.28 mm at 1.3B and 15.35/12.97 mm at 5B are the body/hand MPJPE reductions under occlusion from pretrained versus randomly initialized DiTs.The paired Wan runs use the same task-training settings after 10,000 steps; Table 9 covers all 81 frames.Source: Table 9 · p. 19, §D.1 · p. 20
  • 3.66/3.51 mm are the body/hand MPJPE reductions under occlusion for pretrained Wan-1.3B relative to the DINOv3-H+ reference.The comparison varies architecture, pretraining, and temporal encoding as well as representation choice.Source: §D.1 · p. 20

D.2 EVIDENCE GATING WITH STATIC AND DYNAMIC OBSERVATIONS

  • Evidence gating reduces body MPJPE by 12.2 mm with static regions and 9.0 mm with dynamic regions, while hand MPJPE improves by 1.6 and 3.8 mm, respectively.Table 10 compares gated supervised evidence scores with unit local weights on the Wan-1.3B ablation holdout.Source: §D.2 · p. 20, Table 10 · p. 20

D.3 ADDITIONAL QUALITATIVE COMPARISONS

The qualitative comparisons show DreamHMR reconstructions over time and across varied scenes. In a bench-dip sequence, DreamHMR preserves hand and foot details that both baselines miss.

D.3 ADDITIONAL QUALITATIVE COMPARISONS

  • Across seven frames over two seconds, DreamHMR keeps hands gripping the bench edge and both feet inside shoes, while both baselines leave the shoes uncovered.The sequence uses a fixed crop and scale throughout.Source: §D.3 · p. 20, Figure 6 · p. 21
  • Figure 7 compares four DreamHMR video-backbone configurations with Hand4Whole++ and SMPLest-X across eight videos featuring support, walking, reaching, and foreground occlusions.RGB and ground truth are shown for reference.Source: §D.3 · p. 21, Figure 7 · p. 22

Related papers

Loading 2610.20124v1…