PaperlayerView exact revision on arXiv

Source-linked AI summary

FastHMR: Accelerating Human Mesh Recovery via Token and Layer Merging with Diffusion Decoding

Soroush Mehraban, Andrea Iaboni, Babak Taati

arXiv:2510.10868v1cs.CV

TL;DR

Transformer-based HMR can require substantial computation and memory. FastHMR combines layer and background-token merging with a diffusion decoder, achieving up to 2.3× speed-up while reducing joint-position and acceleration errors.

  • Problem

    Transformer-based HMR requires substantial computation and memory, limiting use in real-time applications and on resource-constrained hardware.

    Source: §1 · p. 1

  • Method

    FastHMR merges transformer layers and background tokens, then uses a diffusion decoder with temporal cues and a pose prior to refine predictions.

    Source: §1 · pp. 1–2, §5 · p. 8

  • Results

    Across 3DPW and EMDB, FastHMR achieves up to 2.3× speed-up on a single GPU while reducing joint-position and acceleration errors.

    Source: §5 · p. 8

  • Takeaways & Limitations

    FastHMR uses one denoising step and one sample, unlike probabilistic models that require iterative denoising and multiple samples.

    Source: §4.3 · p. 6

    FastHMR remains vulnerable when masks classify useful regions such as chairs as background, since those tokens are aggressively merged.

    Source: §4.4 · p. 8

Abstract

from arXiv · show

Recent transformer-based models for 3D Human Mesh Recovery (HMR) have achieved strong performance but often suffer from high computational cost and complexity due to deep transformer architectures and redundant tokens. In this paper, we introduce two HMR-specific merging strategies: Error-Constrained Layer Merging (ECLM) and Mask-guided Token Merging (Mask-ToMe). ECLM selectively merges transformer layers that have minimal impact on the Mean Per Joint Position Error (MPJPE), while Mask-ToMe focuses on merging background tokens that contribute little to the final prediction. To further address the potential performance drop caused by merging, we propose a diffusion-based decoder that incorporates temporal context and leverages pose priors learned from large-scale motion capture datasets. Experiments across multiple benchmarks demonstrate that our method achieves up to 2.3x speed-up while slightly improving performance over the baseline.

1. Introduction

Transformer-based HMR is computationally demanding, in part because it contains redundant layers and background tokens. FastHMR merges both and adds a diffusion decoder; the framework reports up to 2.3× faster inference with slightly improved estimation error.

1. Introduction

  • Transformer-based HMR requires substantial compute and memory, limiting real-time use and deployment on resource-constrained hardware.The passage identifies self-attention’s quadratic complexity as a key source of demand as sequence length grows.Source: §1 · p. 1
  • Redundancy arises across consecutive transformer layers with correlated representations and across background tokens that contribute little to pose or shape estimates.These are the inter-layer and spatial redundancies that motivate the two merging strategies.Source: §1 · p. 1
  • ECLM merges consecutive layers below an error threshold, while Mask-ToMe uses person-background segmentation to merge background tokens and retain human-centric and context information.A diffusion decoder conditions on merged-backbone per-frame features, using temporal context and a VAE pose prior learned from motion-capture data.Source: §1 · pp. 1–2
  • Up to 2.3× faster inference than its transformer baseline, with a slight improvement in estimation error, is reported for FastHMR.The framework jointly uses layer and token merging, then adds a diffusion decoder to offset accuracy loss.Source: §1 · p. 2

2. Related Works

Related work contrasts deterministic mesh estimation with probabilistic diffusion approaches, and surveys several routes to efficient HMR. FastHMR is positioned as post-hoc backbone compression paired with a pose-prior decoder.

2. Related Works

  • Deterministic HMR includes optimization-based SMPL fitting [3] [33] [49] and feed-forward image- or video-based models, while diffusion and score-based methods model uncertainty.Video-based variants add temporal encoders such as 1-D CNNs [20], GRUs [22], or recurrent ViTs.Source: §2 · p. 2
  • Diffusion-based methods include HMDiff [13], ScoreHMR [42], ScoreHypo, and DiffMesh [52], which is described as the first to attempt using temporal information.The methods use distinct conditioning, refinement, or ranking strategies for plausible mesh estimates.Source: §2 · p. 2
  • Efficient HMR spans token pruning [11], lightweight attention [6] [53], and hardware co-design; FastHMR instead merges layers and tokens post-hoc and adds a diffusion decoder.The passage characterizes FastHMR as applicable to any transformer backbone and reports up to 2.3× speed-ups with slight accuracy gains on standard GPUs.Source: §2 · p. 2

3. Method

FastHMR reduces HMR cost by merging layers with limited MPJPE impact and selectively merging background tokens, then uses a motion-prior diffusion decoder conditioned on frame features to recover pose.

3.1. Preliminaries

  • Noisy pose latents are formed by forward diffusion, while backward diffusion denoises Gaussian noise conditioned on video-frame features to recover pose latents.The reparameterization trick and Markov process allow sampling the noisy latent at an arbitrary timestep.Source: §3.1 · pp. 2–3

3.2. Token and Layer Merging

  • ECLM iteratively merges consecutive transformer layers from the end when the resulting MPJPE increase stays below a threshold.The method tests merged candidates against the baseline MPJPE and retains merges according to threshold τ.Source: §3.2 · p. 3, §3.1 · p. 3
  • Mask-ToMe matches tokens across alternating sets and averages the most similar pairs, assigning person tokens negative-infinity scores to prevent their merging.Merging is applied to the first l transformer layers, with a fixed number of tokens retained; the final layer can merge fewer tokens to meet that constraint.Source: §3.2 · pp. 3–4, Figure 3 · p. 3

3.3. Diffusion Decoder

  • A motion VAE learns a lower-dimensional pose-sequence representation from motion data, reconstructing pose sequences with MSE and KL losses.Each pose represents SMPL parameters as joint locations and body-part twist rotations.Source: §3.3 · pp. 4–5
  • A conditional latent diffusion model uses per-frame video features to denoise pose latents, which the VAE decoder converts into pose parameters.The denoiser concatenates frame-feature tokens with noisy latent pose tokens; the method retains linear decoders for shape and camera translation and replaces the pose decoder.Source: §3.3 · p. 5

4. Experiments

FastHMR evaluates its merging and diffusion components on 3DPW and EMDB, using a single RTX 3090 GPU for throughput measurements. It reaches higher throughput with lower reported errors than its baselines, while ablations examine threshold, latent size, training data and objective, qualitative failures, and memory use.

4.1. Datasets and Metrics

  • For human mesh recovery, the study evaluates MPJPE, PA-MPJPE, and PVE on 3DPW and EMDB 1; it also evaluates VAE reconstruction on AMASS.The VAE is trained using AMASS and training splits of Human3.6M and MPI-INF-3DHP; the diffusion model uses Human3.6M, MPI-INF-3DHP, and BEDLAM.Source: §4.1 · p. 5

4.2. Implementation Details

  • At τ = 0.1 mm, ECLM merges 6 CameraHMR layers and 4 HMR2.0 layers; Mask-ToMe retains 90 tokens and merges 40 across the first 3 layers.The diffusion decoder processes 243-frame clips, resampling shorter or longer videos to that length and back.Source: §4.2 · p. 5

4.3. Performance Comparison

  • FastHMR-HMR2.0 achieves 150.0 fps (2.3× faster than HMR2.0), while FastHMR-CameraHMR reaches 103.4 fps (1.9× faster than CameraHMR), reducing acceleration error roughly three-fold and slightly improving estimation errors.FastHMR uses one denoising step and one sample.Source: §4.3 · p. 6

4.4. Ablation Study

  • Larger VAE latents lower AMASS reconstruction error from 6.6 mm to 3.6 mm, while 3DPW recovery error remains around 62 mm; adding BEDLAM improves all three reported metrics.The latent-size ablation suggests that the decoder does not fully benefit from increased latent expressiveness; the BEDLAM comparison reports consistent gains without architectural changes.Source: §4.4 · p. 7, Table 4 · p. 7
  • V-prediction lowers 3DPW MPJPE from 181.8 mm to 62.5 mm and PA-MPJPE from 97.0 mm to 39.1 mm; the mixed objective yields 62.2 mm MPJPE.The mixed objective has nearly the same PA-MPJPE as v-prediction alone.Source: §4.4 · p. 7, Table 5 · p. 7
  • FastHMR raises throughput from 54.6 to 103.4 fps and lowers MPJPE from 73.3 to 71.6 mm versus CameraHMR, while memory remains comparable at 2,787 versus 2,898 MB.Excessive merging can distort pose cues: chair tokens may cause an implausible seated pose, while the diffusion decoder corrects a wrist pulled toward the torso using temporal context. In crowded scenes, retaining M′ = 90 tokens can force merging subject tokens and increase error; low light can remove usable image despite an accurate mask.Source: §4.4 · p. 8

5. Conclusion

FastHMR combines layer and background-token merging with a one-step diffusion decoder in a learned latent space. On 3DPW and EMDB, it achieves up to 2.3× speed-up while reducing joint-position and acceleration errors, though unreliable masks or dominant background cues remain challenges.

5. Conclusion

  • FastHMR combines ECLM and Mask-ToMe with a one-step latent-space diffusion decoder that refines predictions using temporal and pose priors.ECLM merges layers with little output effect, while Mask-ToMe merges tokens outside the person region.Source: §5 · p. 8
  • On 3DPW and EMDB, FastHMR achieves up to 2.3× speed-up while reducing joint-position and acceleration errors; unreliable masks or dominant background cues remain challenges.The conclusion identifies adaptive merging and joint segmentation training as future work.Source: §5 · p. 8

A. Datasets

The paper draws on real-world, motion-capture, markerless, and synthetic datasets for training and evaluation, with annotations varying by dataset.

A. Datasets

  • 3DPW and EMDB provide test data for mesh recovery, while BEDLAM, Human3.6M, MPI-INF-3DHP, and AMASS supply training or motion-capture data.3DPW includes in-the-wild scenes and SMPL annotations; EMDB provides SMPL parameters and separate camera-coordinate and trajectory test splits.Source: A. Datasets · p. 12

B. Additional Ablation Studies

The ablations examine pose representation, diffusion decoding, token masking, and model efficiency. They report benefits from pose decomposition and person-mask-guided merging, alongside qualitative robustness in challenging videos.

Effect of swing-twist decomposition

  • 33 mm: swing-twist decomposition reduces MPJPE on 3DPW with a single-step diffusion model.The authors attribute this to separating joint location from twist rotation, reducing denoising difficulty and reliance on shape parameters.Source: B. Additional Ablation Studies · p. 12, Table 7 · p. 12

Using diffusion alone

  • 1.9 mm on 3DPW and +1 mm on EMDB: removing merging while retaining diffusion lowers MPJPE on the tuning set but raises it on the external set.The authors say diffusion can recover the loss associated with Mask-ToMe and that Mask-ToMe supports generalizability by merging background tokens.Source: B. Additional Ablation Studies · p. 12, Table 8 · p. 12

Effect of Segmentor

  • In the proposed FastHMR framework, YOLO11n-seg is used during inference because the compared alternative requires significantly more computational resources; the excerpt does not identify that alternative by name.Source: B. Additional Ablation Studies · p. 13, Table 9 · p. 13

Effect of masking ratio

  • In Mask-ToMe, larger merging ratios generally increase MPJPE error on 3DPW and EMDB; nl = 100 produces the worst results, while ratios of 10 or 20 also cause higher-than-expected errors.Attention maps reveal that corner background tokens are repurposed as register tokens, with important information gradually shifting into them as layers progress.Source: B. Additional Ablation Studies · p. 13

Excluding person mask in Mask-ToMe

  • 14.1 mm: removing the person mask increases MPJPE, and replacing the MLP decoder with diffusion does not recover the loss.The authors attribute the loss to merging person tokens and removing information critical for mesh recovery.Source: B. Additional Ablation Studies · p. 13, Table 11 · p. 13

Parameters and MACs

  • ECLM reduces parameters and MACs for HMR2.0 and CameraHMR, while adding Mask-ToMe yields substantial computational savings.The diffusion decoder is described as lightweight, with relatively few parameters and MACs compared with the backbones.Source: B. Additional Ablation Studies · p. 13, Table 12 · p. 13

In-the-wild Evaluation

C. Additional Hyperparameters

The supplied material identifies tables for diffusion-training, denoiser, and VAE-pretraining hyperparameters, and a figure comparing CameraHMR with FastHMR-CameraHMR.

Diffusion Hyperparameters

Denoiser Hyperparameters

VAE Hyperparameters

Qualitative comparison

  • Figure 9 presents a qualitative comparison between CameraHMR and FastHMR-CameraHMR.Source: Figure 9 · p. 14

D. Additional Qualitative Comparison

Across four frames, CameraHMR and FastHMR-CameraHMR can appear similar depending on viewpoint. FastHMR is described as more robust to occluded body parts because of its temporal awareness.

Qualitative comparison

Viewpoint

Occlusion

Related papers

Loading 2510.10868v1…