PaperlayerView exact revision on arXiv

Source-linked AI summary

Training-Free Dense Hand Contact Estimation with Multi-Modal Large Language Models

Daniel Sungho Jung, Kyoung Mu Lee

arXiv:2605.05886v1cs.CV

TL;DR

Dense hand contact estimation requires semantic understanding alongside fine-grained 3D geometric reasoning, but MLLMs struggle to encode hand geometry and vertex-level contact. ContactPrompt addresses this with structured hand representations and multi-stage reasoning, achieving superior performance to supervised approaches without task-specific training.

  • Problem

    MLLM-based dense hand contact estimation remains limited by ineffective encoding of 3D hand geometry and difficulty capturing fine-grained vertex-level contact.

  • Method

    ContactPrompt uses detailed hand-part segmentation, part-wise vertex grids, and multi-stage structured contact reasoning with part conditioning.

  • Results

    ContactPrompt achieves superior dense hand contact estimation performance compared with previous supervised approaches without training on dense hand-contact datasets.

  • Takeaways & Limitations

    The approach shows that MLLMs can perform dense hand contact estimation through structured reasoning without task-specific training.

  • Takeaways & Limitations

    ContactPrompt relies on computationally expensive MLLMs and external APIs whose changing behavior can increase cost and limit reproducibility.

Abstract

from arXiv · show

Dense hand contact estimation requires both high-level semantic understanding and fine-grained geometric reasoning of human interaction to accurately localize contact regions. Recently, multi-modal large language models (MLLMs) have demonstrated strong capabilities in understanding visual semantics, enabled by vision-language priors learned from large-scale data. However, leveraging MLLMs for dense hand contact estimation remains underexplored. There are two major challenges in applying MLLMs to dense hand contact estimation. First, encoding explicit 3D hand geometry is difficult, as MLLMs primarily operate on vision and language modalities. Second, capturing fine-grained vertex-level contact remains challenging, as MLLMs tend to focus on high-level semantics rather than detailed geometric reasoning. To address these challenges, we propose ContactPrompt, a training-free and zero-shot approach for dense hand contact estimation using MLLMs. To effectively encode 3D hand geometry, we introduce a detailed hand-part segmentation and a part-wise vertex-grid representation that provides structured, localized geometric information. To enable accurate and efficient dense contact prediction, we develop a multi-stage structured contact reasoning with part conditioning, progressively bridging global semantics and fine-grained geometry. Therefore, our method effectively leverages the reasoning capabilities of MLLMs while enabling precise dense hand contact estimation. Surprisingly, the proposed approach outperforms previous supervised methods trained on large-scale dense contact datasets without requiring any training. The codes will be released.

1 Introduction

ContactPrompt addresses the underexplored use of MLLMs for dense hand contact estimation by combining high-level semantic reasoning with fine-grained geometric reasoning. It does so through structured hand-geometry representations and multi-stage contact reasoning, achieving accurate estimation without training and outperforming supervised methods trained on large-scale datasets.

  • Motivation: Human hand interactions are shaped by language-expressible semantic intentions, motivating dense contact estimation that leverages semantic meaning.Examples include holding a cup or pressing a button.
  • Motivation: MLLMs provide powerful language-based reasoning over predominantly visual inputs, but their use for 3D reasoning remains underexplored.Prior work has used MLLMs for high-level semantic guidance and improved generalization in vision tasks.
  • Challenges: Dense hand contact estimation faces difficulty encoding 3D hand geometry because MLLMs primarily process vision and language modalities.Raw MANO mesh data may be converted into text, causing geometry to be processed as textual input.
  • Method: ContactPrompt introduces structured geometry-to-language representations, including detailed hand-part segmentation and part-wise vertex grids, to support MLLM reasoning.The framework is designed to connect high-level semantic reasoning with fine-grained geometric reasoning.
  • Results: ContactPrompt enables accurate and efficient dense hand contact estimation in a training-free, zero-shot manner and outperforms supervised methods trained on large-scale datasets.Its contact prediction uses multi-stage structured reasoning with part conditioning.

2 Related works

Prior dense hand-contact methods largely depend on task-specific supervision and specialized architectures, while MLLM-based dense hand-contact estimation remains underexplored. Related MLLM research instead uses structured representations, geometric cues, and prompting for broader 3D reasoning tasks.

  • Dense hand contact estimation: Most dense hand-contact methods rely on task-specific datasets with dense labels or distance-thresholded contact derived from human and scene geometry.These datasets underpin existing approaches to dense hand contact estimation.
  • Dense hand contact estimation: POSA models contact probability conditioned on 3D body pose using a cVAE, while BSTRO estimates dense body–scene contact on SMPL-X vertices with a Transformer.The cited passage presents POSA and BSTRO as representative architectures for contact estimation.
  • Dense hand contact estimation: GECO predicts contact only at the part level and focuses on full-body contact, whereas HACO remains constrained by task-specific supervision and generalization limitations.These limitations motivate approaches that support fine-grained vertex-level hand-contact estimation with MLLMs.
  • Dense hand contact estimation: MLLM-based dense hand-contact estimation remains underexplored, and ContactPrompt formulates it as structured reasoning to enable fine-grained vertex-level prediction.The passage contrasts this formulation with prior contact-estimation advances.
  • Prompting for 3D reasoning with MLLMs: Recent MLLM work for 3D reasoning uses object coordinates, hierarchical scene graphs, atlases, structured roadmaps, and explicit geometric priors or cues.The cited examples include Transcribe3D, SG-Nav, CE3D, TSTMotion, and 3DAxisPrompt.

3 Method

ContactPrompt formulates dense hand contact estimation as a multi-stage structured reasoning problem over the 778-vertex MANO mesh. It combines detailed functional hand-part segmentation, part-wise vertex grids, and staged MLLM reasoning to connect global interaction semantics with focused vertex-level prediction.

  • Problem formulation: ContactPrompt predicts binary contact labels over the 778-vertex MANO hand mesh through progressively structured reasoning rather than direct image-to-contact regression.The stages connect global semantic reasoning with fine-grained geometric prediction.
  • Hand-part segmentation: The hand is segmented into detailed, functionally aligned parts based on surface orientation and finer regions such as finger bases and palm centers.The decomposition includes palmar, dorsal, palmar radial, and palmar ulnar regions and differs from DIGIT’s segmentation.
  • Part-wise vertex grids: Each segmented part is represented as an ordered vertex grid whose rows run from fingertip to wrist and whose within-row vertices run left to right.The grid specification provides each part’s row count and row lengths to the MLLM, with visual dots, lines, and row connections indicating ordering.
  • Multi-stage structured reasoning: The MLLM reasons in three stages: free-form interaction description, contact-part prediction, and dense vertex-grid prediction for selected parts.The free-form prompt addresses pose, viewpoint, object interaction, occlusion, and physically plausible contact regions before part and dense prediction.
  • Part conditioning: Part conditioning restricts dense prediction to vertices belonging to predicted contact parts and assigns vertices outside this active set as non-contact.This reduces the effective prediction size from all V vertices to V′ = |Vactive|, lowering output tokens and improving dense-stage inference efficiency.

4 Implementation details

The implementation uses GPT-5.5 through the OpenAI API for training-free, zero-shot inference. Images are passed as base64 JPEGs, textual prompts directly, and the reasoning stages permit limited retries.

  • 4 Implementation details: GPT-5.5 serves as the base MLLM through the OpenAI API for training-free, zero-shot inference.
  • 4 Implementation details: The input RGB image and visual prompts are encoded as base64 JPEGs before being passed to the MLLM.
  • 4 Implementation details: Textual prompts are provided directly without additional preprocessing.
  • 4 Implementation details: The contact reasoning pipeline permits a fixed number of retries for each stage, with up to 2 retries at the part stage.The supplied passage truncates the retry limit for the dense stage.

5 Experiments

Experiments on the 92-sample MOW evaluation split assess dense contact accuracy and MLLM efficiency. Ablations and comparisons show that structured hand representations, multi-stage reasoning, and part conditioning improve performance, with ContactPrompt outperforming prior methods without task-specific training.

  • Evaluation Setup: The evaluation uses 92 MOW samples with HACO-derived dense contact annotations from ground-truth 3D hand and object meshes.Precision, recall, and F1-score are computed at MANO-vertex level, alongside output-token count and per-sample inference cost.
  • Ablation Studies: Detailed hand-part segmentation improves precision by 17.1%, recall by 53.0%, and F1-score by 35.2% over DIGIT’s coarse segmentation.It also reduces output tokens by 32.4%, attributed to structured, semantically meaningful hand decomposition.
  • Ablation Studies: The part-wise vertex grid improves recall by 55.7% and F1-score by 21.8% compared with removing explicit within-part spatial structure.The ablation collapses each part’s multi-row structure into a single row, limiting spatial reasoning.
  • Ablation Studies: The full free-form, part, and dense reasoning pipeline achieves the best F1-score of 0.526 among partial configurations.Using only the dense stage yields low precision (0.382) and extremely high recall, indicating overly confident predictions.
  • Ablation Studies: Part conditioning improves precision by 10.5% and F1-score by 0.8% while reducing output tokens by 20.7% versus no part conditioning.The unconditioned variant has higher recall but produces overly broad, less precise contacts.
  • Comparative Results: ContactPrompt achieves the best F1-score of 0.526 and highest recall of 0.710 against POSA, BSTRO, DECO, and HACO without task-specific training.GPT-5.5 matches these best results; Claude Sonnet 4.6 reaches an F1-score of 0.488, while Claude Opus 4.7 achieves the highest precision of 0.506.

6 Limitations and societal impacts

ContactPrompt is limited by the computational cost of MLLMs and reliance on external APIs, which can increase expenses and reduce reproducibility. It may support AR/VR and robotics applications but could also be misused for human monitoring.

  • Limitations: MLLMs are typically more computationally expensive than task-specific models, limiting ContactPrompt’s efficiency.The passage identifies reliance on MLLMs as a primary limitation.
  • Limitations: External MLLM APIs may incur costs and limit reproducibility because model behavior can change.API dependence creates both financial and reproducibility concerns.
  • Societal impacts: ContactPrompt could benefit hand-interaction applications such as AR/VR and robotics, but may be misused for human monitoring.The societal impacts include both beneficial applications and potential misuse.

7 Conclusion

ContactPrompt is a training-free, zero-shot approach for dense hand contact estimation with multimodal large language models. It combines structured 3D hand geometry representations with multi-stage, part-conditioned reasoning and achieves superior performance compared to previous supervised methods.

  • Approach: ContactPrompt performs dense hand contact estimation with multimodal large language models in a training-free, zero-shot setting.The approach is designed to leverage MLLMs without additional training.
  • 3D Geometry Encoding: Detailed hand-part segmentation and a part-wise vertex-grid representation encode 3D hand geometry for MLLMs.These representations provide structured geometric information for the estimation task.
  • Contact Reasoning: A multi-stage structured contact reasoning framework with part conditioning enables accurate and efficient dense contact prediction.The framework combines staged reasoning with explicit hand-part conditioning.
  • Performance: ContactPrompt achieves superior performance compared to previous supervised methods.The conclusion reports this comparison while emphasizing that ContactPrompt requires no training.

A Appendix

The appendix supplies technical details omitted from the main manuscript, including the full prompts used in each stage of ContactPrompt’s multi-stage structured contact reasoning and discussion of prompt design.

  • A Appendix: The appendix presents full text prompts for stages 0, 1, and 2, followed by discussions of text prompt design.These materials expand the technical description of ContactPrompt’s multi-stage structured contact reasoning.

A.1 Full text prompt of stage 0

Stage 0 uses free-form reasoning to describe hand-object interaction after explicitly analyzing viewing geometry, visibility, grasp structure, and physically plausible contact. The prompt requires free-form text output only.

  • Stage 0 Text Prompt: Free-Form Reasoning: The prompt first asks the model to describe the hand-object interaction visible in the image.
  • Stage 0 Text Prompt: Free-Form Reasoning: It requires reasoning about hand orientation, camera viewpoint, visible hand surfaces, occlusion, object position, and the distinction between appearance and true contact.
  • Stage 0 Text Prompt: Free-Form Reasoning: The prompt specifies analyzing hand laterality, grasp type, supporting fingers and surfaces, contact-region size, and physically plausible occluded support regions.
  • Stage 0 Text Prompt: Free-Form Reasoning (continued): It directs sequential finger reasoning, comparison of finger length and object height, and precise localization of contact on proximal, middle, or distal palm segments.
  • Stage 0 Text Prompt: Free-Form Reasoning: The model must output free-form text only.

A.2 Full text prompt of stage 1

Stage 1 uses a structured text prompt to predict hand parts contacting an object from interaction imagery and multi-view part segmentation. It combines viewpoint, occlusion, grasp, object-placement, and threshold-aware reasoning before emitting a constrained JSON list.

  • Stage 1 Text Prompt: Part-level Contact Prediction: Stage 1 predicts which hand parts contact the object using an interaction image and a multi-view rendering of hand-part segmentation.The prompt specifies part-level contact prediction for the current sample.
  • Rules: The prompt requires reasoning about camera viewpoint, visible and occluded regions, foreshortening, and physically supported contact rather than selecting only visible surfaces.Viewpoint categories include palmar, dorsal, radial, ulnar, and oblique mixed angles.
  • Rules: Contact extent is constrained by object-to-hand distance and grasp configuration, including wrapping, pinching, webspace, finger bases, palm regions, and adjacent stabilizing parts.For pen- or pencil-like objects, the prompt prioritizes webspace and finger-base regions before thenar involvement.
  • Rules: The output must be exactly one JSON object, using only provided part names and including clear, partial, subtle, stabilizing, side, grazing, or occluded contact.The required format is {"contact_parts":[part_a_name, part_b_name, ...]}; an empty list is disallowed unless the hand certainly has no contact.
  • Rules: For thin finger regions, contact on one surface orientation can justify anatomically corresponding orientations when threshold-based overlap is plausible, but this prior is not applied aggressively to broad palms.The orientation spread may occur across palmar-dorsal or radial-ulnar counterparts of the same finger segment or fingertip.

A.3 Full text prompt of stage 2 · A.4 Discussions on text prompt design

Stage 2 uses structured prompts for dense binary contact prediction on selected hand parts, combining multi-view segmentation and vertex-grid inputs with viewpoint, visibility, and physical-plausibility reasoning. The prompt design progressively bridges semantic interaction understanding and fine-grained geometric prediction across stages.

  • A.3 Full text prompt of stage 2: Stage 2 predicts dense binary contact only for the selected contact parts of the current sample.The prompt requests exactly one JSON object containing only selected parts, with binary values and grids matching the specified dimensions.
  • A.3 Full text prompt of stage 2: The stage 2 input combines a hand-object interaction image with multi-view hand-part segmentation and part-wise vertex-grid renderings.The segmentation views include part indices and vertex-grid structure.
  • A.3 Full text prompt of stage 2: Before prediction, the prompt requires reasoning about the right-hand viewpoint and whether selected parts are visible, partially visible, self-occluded, or object-occluded.Viewpoint categories include palmar, dorsal, radial, ulnar, and oblique mixed angles.
  • A.3 Full text prompt of stage 2: The prompt encourages physically plausible, spatially coherent contact by preserving continuity across adjacent parts while avoiding fragmentation or unjustified broad-region flooding.It also constrains overall contact size when the object center is far from the hand center and spreads contact within parts for pen- or pencil-like objects.
  • A.3 Full text prompt of stage 2: For thin fingers, the prompt models threshold-based 3D contact propagation across anatomically corresponding orientations when holding, pressing, or stabilizing an object.This orientation-spread prior is applied mainly to thin finger regions and more cautiously to broad palms.
  • A.4 Discussions on text prompt design: The text prompts progressively bridge high-level semantics and fine-grained geometry: Stage 0 reasons globally, Stage 1 narrows prediction to meaningful parts with geometric priors, and Stage 2 performs structured dense prediction.The progression explicitly incorporates viewpoint, occlusion, physical plausibility, orientation consistency, and threshold-based contact propagation.
Loading 2605.05886v1…