PaperlayerView exact revision on arXiv

Source-linked AI summary

MHE-Former: Multi-Hypothesis Transformers via Entropy Maximization for 3D Mesh Recovery

Boshu Jia, Rongyu Chen, Linlin Yang, Zihao Liu, Yingjie Chen, Zhongqun Zhang, Zhulin Tao, Shaohui Lin, Xiaoyu Wu, Libiao Jin, Baochang Zhang, Angela Yao

arXiv:2609.10743v1cs.CV

TL;DR

Ambiguous monocular hand and body mesh recovery is poorly served by deterministic single predictions, especially under occlusion. The paper introduces entropy-maximized MHE-Former for diverse hypothesis exploration and context-aware selection using evidence, VLM reasoning, and natural-language intent, reporting state-of-the-art accuracy and diversity plus practical user-study support.

  • Problem

    Monocular mesh recovery is ill-posed under occlusion and ambiguity, while deterministic methods regress single overconfident solutions.

  • Method

    The paper introduces an exploration–exploitation framework with entropy-maximized MHE-Former and context-aware hypothesis selection using additional evidence, VLM reasoning, and natural-language intent.

  • Results

    Experiments report state-of-the-art accuracy and diversity across hand and body benchmarks, while user studies and qualitative results demonstrate practical utility for hypothesis selection.

  • Takeaways & Limitations

    The framework combines uncertainty-aware multi-hypothesis generation with flexible context-guided decision-making for ambiguous mesh recovery.

Abstract

from arXiv · show

Monocular 3D hand and body mesh recovery often suffers from severe occlusion and ambiguity. Traditional deterministic methods typically regress a single optimal solution, leading to overconfident predictions. In this paper, we introduce an exploration--exploitation paradigm for ambiguous mesh recovery with multi-hypothesis learning and selection. Specifically, during exploration, based on our probabilistic formulation and entropy maximization, we propose a novel multi-hypothesis method referred to as MHE-Former. It is a Transformer-based multi-hypothesis framework, ensuring high training efficiency and label friendliness while generating plausible and diverse hypotheses. During exploitation, we propose Hypothesis Selection, a context-aware process for multiple predictions. Especially leveraging VLM's powerful visual understanding and reasoning capabilities, it allows users to choose the most plausible and desired estimate with additional evidence and natural language intent. Extensive experiments demonstrate that our framework achieves state-of-the-art performance in accuracy and diversity across multiple datasets. The user preference study further shows the practicality of our hypothesis selection process.

I. INTRODUCTION

Monocular mesh recovery is ambiguous, yet deterministic methods typically produce overconfident single solutions. MHE-Former addresses this with entropy-maximized exploration of diverse hypotheses followed by context-aware selection using evidence or natural-language intent.

  • Deterministic monocular reconstruction commonly maps ambiguous observations to one solution, producing overconfident predictions under occlusion and depth ambiguity.
  • The entropy-maximized formulation uses only 2D visible keypoints as weak supervision while preserving visible-joint consistency and exploring diverse occluded-part solutions.
  • The proposed exploration–exploitation paradigm first generates multiple plausible hypotheses, then selects the most suitable result using additional information or instructions.
  • MHE-Former combines a Transformer architecture, a dual-branch MH-Decoder, normalizing-flow pose exploration, and entropy maximization for plausible and diverse hypotheses.
  • Hypothesis selection uses extra evidence, multi-view renderings, VLM reasoning, and natural-language instructions to identify accurate, plausible, or task-consistent estimates.
  • Experiments on hand and body benchmarks report state-of-the-art accuracy and diversity, while user studies and qualitative results support the selection process's practical utility.

II. RELATED WORKS

Related work spans deterministic mesh recovery, multi-hypothesis generative modeling, and hypothesis selection. Existing approaches model diverse solutions but remain limited in representing the underlying distribution or balancing exploration with exploitation.

  • Human Hand & Body Mesh Recovery: Transformer-based methods using ViT-Pose backbones and large-scale datasets have become predominant for monocular hand and body mesh recovery.
  • Human Hand & Body Mesh Recovery: Monocular recovery remains ill-posed because occlusion and ambiguity make forcing a single mean-like solution overconfident for plausible low-likelihood poses.
  • Multi-Hypothesis Learning: Multi-hypothesis methods use stochastic sampling, regression heads, VAEs, diffusion, and normalizing flows to generate diverse feasible predictions.
  • Multi-Hypothesis Learning: Existing generative methods can fit pose distributions and generate multiple hypotheses, but often depend on training-set diversity and do not fully model the underlying distribution.
  • Hypothesis Selection: Hypothesis selection identifies an optimal conditioned hypothesis using external knowledge, with prior work employing learned aggregation, scoring networks, or consensus checks.

III. PRELIMINARIES

The preliminaries define parametric human models, weakly supervised 3D pose and shape estimation, and normalizing flows for modeling complex pose distributions.

  • A. Parametric Models: MANO and SMPL parameterize human bodies and hands using pose θ and shape β, which determine meshes and joint coordinates.The pose and shape parameters are expressed through rotation or PCA-based representations.
  • B. Weakly-Supervised Estimation: Weak supervision estimates MANO or SMPL parameters from images and projects the resulting 3D pose back to observed 2D joints.The objective combines 2D reconstruction with pose-prior and shape-regularization terms.
  • C. Normalizing Flow: Normalizing flows model complex, multimodal distributions through a sequence of invertible mappings between data X and a base variable Z.Sampling maps base-distribution samples forward to data space, while likelihood evaluation uses the reverse flow.
  • C. Normalizing Flow: Masked Autoregressive Flow uses preceding dimensions to predict scale and shift parameters, yielding a tractable triangular Jacobian.This structure supports exact likelihood estimation and maximum-likelihood optimization.

D. Principle of Maximum Entropy

The maximum-entropy principle models ambiguous pose and shape recovery by matching visible observations while avoiding unsupported assumptions about incomplete regions.

  • D. Principle of Maximum Entropy: Entropy quantifies the uncertainty of a random variable, providing the uncertainty measure used in the maximum-entropy formulation.The section introduces entropy as a property of a random variable X over its domain.
  • D. Principle of Maximum Entropy: Maximum entropy selects the highest-entropy distribution compatible with complete observations, such as visible joints.For ambiguous or incomplete observations, the distribution remains as unbiased as possible.
  • D. Principle of Maximum Entropy: Entropy maximization is used to remove inadvertent assumptions and encourage exploration of the full set of prototypes.This motivation connects the principle to efficient, self-supervised, and semi-supervised learning.
  • D. Principle of Maximum Entropy: The exploration stage models pose recovery from RGB images and visible 2D keypoints while treating shape β and camera c as deterministic estimates.The modeled uncertainty is concentrated on the pose parameter θ conditioned on image, keypoints, camera, and shape.
  • D. Principle of Maximum Entropy: The learned model distribution pϕ(θ|I, j, c, β) is trained against the data distribution using Kullback–Leibler divergence.The conditional pose distribution is defined over pose parameters given the image and associated observations.

A. Data Distribution

The paper models ambiguous pose recovery probabilistically, combining projection consistency with pose priors and entropy maximization to generate diverse feasible hypotheses. MHE-Former implements this through deterministic camera and shape prediction plus stochastic pose generation conditioned on image features.

  • Data Distribution: MHE-Former models pose uncertainty with p(θ|I) proportional to projection likelihood times pose prior, separating reconstruction accuracy from feasibility.The likelihood enforces projection consistency, while the prior encodes general pose plausibility.
  • Data Distribution: Entropy maximization encourages diverse hypotheses while reconstruction and prior terms keep them consistent with observations and feasible.This avoids requiring one-to-many labels or direct comparisons among sampled keypoints.
  • Architecture: MHE-Former uses a ViT encoder and dual-branch MH-Decoder to predict shared camera and shape parameters alongside sampled pose hypotheses.The stochastic branch estimates pϕ(θ|I), while the deterministic branch predicts c and β.
  • Architecture: Masked cross-attention lets each joint token retrieve relevant image evidence rather than receiving only globally injected context.The joint token acts as query and image embeddings provide keys and values for conditional generation.
  • Architecture: The stochastic decoder transforms independent latent samples through attention flow blocks into diverse yet plausible pose hypotheses.Reconstructed meshes share camera and shape parameters across pose hypotheses.

B. Training with Multiple Hypotheses

Training combines visible-joint reconstruction, pose and shape priors, and an entropy objective. This objective uses weak supervision while balancing evidence fitting, feasibility, and hypothesis diversity.

  • Training Objective: The reconstruction loss supervises only visible 2D joints, using a visibility indicator to compare predicted and ground-truth locations.This implements weak supervision from visible keypoints rather than requiring complete mesh labels.
  • Training Objective: The training objective combines reconstruction, adversarial prior, entropy, and shape-regularization terms with tunable trade-off weights.Each component corresponds to a term in the probabilistic objective.
  • Training Objective: The entropy loss is computed from pose samples generated by the normalizing flow and mapped back to latent space for likelihood evaluation.The reverse flow enables Monte Carlo estimation of the entropy-related term.

C. Hypothesis Selection

Hypothesis Selection exploits generated predictions using evidence, interaction priors, and natural-language intent. A VLM filters for plausibility and then selects the hypothesis that best matches the user’s request.

  • Selection Framework: Hypothesis Selection identifies a suitable prediction using input evidence, interaction priors, and natural-language intent.The process is context-aware because these three sources guide selection among ambiguous outputs.
  • Evidence-driven Optimization: Evidence-driven optimization selects samples that best satisfy additional constraints, such as grasp configuration or multi-view labels.The constraints refine the model distribution using task-specific evidence.
  • Interactive Selection: The VLM first filters rendered hypotheses for feasibility and visible-region alignment before interactive selection.Filtering scores hypotheses across multiple criteria and retains the top-k candidates.
  • Interactive Selection: Natural-language instructions guide final selection after plausibility filtering, with few-shot examples and generated rationales supporting the VLM decision.The VLM analyzes the evidence, justifies its choice, and returns the selected hypothesis index.
  • Interactive Selection: Interactive selection can incorporate arbitrary contextual information to guide hypothesis choice through VLM reasoning.The paper identifies this flexibility as useful for intricate multi-hypothesis selection challenges.

A. Experimental Details

The experiments evaluate ambiguity handling, weak supervision, accuracy, consistency, and diversity across hand and body mesh recovery settings. Toy studies examine how entropy and feasibility priors shape the learned modes.

  • Toy Experiments: Under weak supervision, the proposed probabilistic method recovers all four toy depth-ambiguity modes, whereas MDN fits one mode without complete strong supervision.The deterministic model also learns only one mode and can predict incorrect modes after small input perturbations.
  • Experimental Settings: MHE-Former is evaluated on ambiguous and standard body datasets plus HO3D and ARHD hand datasets featuring occlusion and manual masking.Comparisons include deterministic and multi-hypothesis baselines such as ProHMR, MDN, CVAE, VMarker-Pro, and CtF-MHE.
  • Metrics: The evaluation uses BH for best-case accuracy, AH for visible-joint consistency, and PJD and RD for diversity concentrated in uncertain joints.PJD measures per-joint variation, while RD compares diversity in occluded and visible parts.
  • Toy Experiments: Lower entropy weighting misses modes, while higher weighting disperses predictions and harms fitting accuracy.The experiment demonstrates a trade-off between multimode coverage and reconstruction fidelity.
  • Toy Experiments: Feasibility priors determine the solution space: imposing sin α_k ≥ 0 for the top keypoint reduces the learned modes from four to two.Reducing prior loss instead favors evidence fitting but can produce less feasible poses.

B. Toy Experiments

Toy and benchmark experiments show that MHE-Former recovers diverse feasible modes under weak or visible-keypoint supervision while preserving consistency with observed parts. Across hand and body settings, it achieves strong accuracy and diversity, including state-of-the-art results on HO3D-v3.

  • Toy Experiments: Under weak supervision, MHE-Former recovers all ambiguous modes, whereas MDN fits only one mode without complete strong supervision.The deterministic model also learns only one of four modes and is sensitive to small input perturbations.
  • Toy Experiments: The entropy term balances reconstruction and diversity: reducing its weight misses modes, while increasing it weakens agreement with observed evidence.The prior term further constrains the feasible solution space from which diverse predictions are explored.
  • Interaction Scenarios: MHE-Former achieves lower joint and mesh reconstruction errors and lower RD than related methods on HO3D-v3, even with weak 2D supervision.The comparison is reported for hand-object interaction scenarios.
  • Interaction Scenarios: On HO3D, generated hypotheses maintain higher visible-part consistency while providing wider diversity for occluded fingertips.The skeleton distributions and VHRC analysis compare visible accuracy with invisible-part diversity.
  • Ambiguous Scenarios: On ambiguous hand and body recovery benchmarks, MHE-Former achieves the best results under matched visible-keypoint supervision and generalizes to an out-of-domain dataset.It also attains a lower BH result than deterministic and multi-hypothesis methods on standard H36M.

D. Hypothesis Selection

Hypothesis selection uses additional evidence and VLM reasoning to filter or choose among generated predictions. The experiments evaluate evidence-driven optimization, interactive selection, and training choices affecting accuracy, diversity, and efficiency.

  • Evidence-Driven Optimization: Evidence-driven optimization selects a single hypothesis using additional constraints, and MHE-Former achieves the best PA-MPJPE on H36M in this downstream setting.The optimized model outputs one hypothesis after observing additional evidence.
  • Interactive Selection: VLM-based selection performs plausibility filtering and interactive choice for hand and body estimates using 2D JPE and interaction plausibility loss.The framework also evaluates user preference for plausibility and instruction consistency.
  • Visible Weak Supervision: Adding occluded-keypoint supervision does not significantly improve accuracy and can increase RD, harming diversity for occluded keypoints.This ablation supports training with visible weak supervision rather than manually added occluded labels.
  • Number of Hypotheses: Increasing the number of generated hypotheses lowers BH while AH 2D JPE and RD remain unchanged, indicating preserved visible consistency and diversity with relatively few samples.The trends are evaluated across different hypothesis sample sizes.
  • Strategy and Structure: LoRA fine-tuning balances diversity, necessary accuracy, and training efficiency, whereas full fine-tuning increases diversity but substantially reduces accuracy and raises training costs.The comparison concerns training strategies using a pretrained ViT encoder.

VII. CONCLUSION

The paper presents an exploration–exploitation framework for ambiguous hand and body mesh recovery. MHE-Former explores plausible distributions, while VLM-assisted hypothesis selection uses evidence and semantic reasoning to improve practical utility.

  • Conclusion: MHE-Former combines probabilistic entropy maximization with a Transformer framework to explore multiple plausible mesh hypotheses under ambiguity.The conclusion frames this as the exploration stage of the proposed paradigm.
  • Conclusion: Hypothesis selection exploits additional evidence and VLM semantic reasoning to connect multi-hypothesis generation with practical use.The paper reports quantitative experiments and user studies supporting the framework's effectiveness.
Loading 2609.10743v1…