TactoHand: Learning Geometry-Based Dense Hand-Surface Interaction Fields from Video

Motivation

• The paper treats contact, proximity, and force as complementary vertex-wise quantities describing interaction location, separation, and mechanical effect.The shared representation is intended to support interaction reconstruction, action understanding, and embodied learning.Source: §1 · p. 1

Data and supervision

• The training collection harmonizes 24 datasets spanning around 50 million frames, with geometry-derived contact and proximity labels and motion-derived force labels for eligible sequences.Object motion constrains aggregate force and torque but does not uniquely determine their distribution across hand vertices.Source: §1 · pp. 1–2

Force supervision

• An SDF-aware formulation applies geometric, frictional, and temporal constraints to regularize vertex-force estimates.Source: §1 · p. 2

Approach

• TactoHand uses a temporal transformer and three learned queries to predict contact, proximity, and 3D force fields on corresponding MANO vertices.Predicted vertex forces are aggregated into object-level wrenches for differentiable rigid-body supervision.Source: §1 · p. 2, Figure 1 · p. 2

Evaluation

• On HOCap, contact Macro-F1 is 50.5 for TactoHand versus 18.0 for HACO; physics supervision reduces rollout errors, and contact and proximity fields improve downstream reconstruction.The evaluation spans interaction-field prediction and hand–object reconstruction.Source: §1 · p. 2

Hand-surface Contact and Proximity

• Earlier methods separately model dense contact, hand–object distance, pressure, or force; TactoHand aligns contact and proximity on MANO vertices across hand–object, hand–hand, and hand–scene data.Source: §2 · pp. 2–3

Visual Prediction of Pressure and Force

• Visual force methods use instrumented surfaces or gloves, while TactoHand derives three-component vertex-force targets from object motion for sequences with suitable trajectories.Source: §2 · p. 3

Motion and Physical Consistency

• Prior motion-consistency methods match simulated forces to observed motion, while TactoHand regularizes vertex-force targets because motion constrains only the aggregate wrench.Source: §2 · p. 3

Interaction Cues for Downstream Tasks

• Prior work uses contact and distance constraints in hand–object reconstruction, and this paper tests predicted contact and proximity as fixed constraints in existing pipelines.Source: §2 · p. 3

Problem Formulation

• Each hand uses the fixed MANO topology of 778 vertices, indexed by hand side and vertex, to represent the predicted interaction fields.Source: §3 · p. 3
• The fields comprise contact probability, proximity that increases with closeness, and raw vertex-local force components predicted from RGB.Converting gated force components to camera-frame forces additionally requires ground-truth posed MANO geometry.Source: §3 · p. 3
• Force losses apply only to eligible hand–object samples, while contact and proximity supervision is retained for hand–hand and hand–scene samples when annotations are valid.Source: §3 · p. 3

4.1 DATA SOURCES

• The collection contains 282,874 sequences and 49.02 million frames from 24 sources; geometry-based contact and proximity labels cover all datasets, while motion-based force labels are derived for 15 hand–object datasets.Training excludes GigaHands force targets and all HOCap data.Source: §4.1 · p. 3

4.2 CONTACT AND PROXIMITY LABELING

• Proximity targets equal one on and inside the counterpart surface, decrease with outside separation, and remain constant beyond the distance cutoff.Source: §4.2 · p. 4
• Contact labels use distance thresholds and normal alignment, while inside-mesh inclusion retains potential contacts displaced into the object by pose error.The rule bypasses the normal test for on-surface vertices and within a 3 mm band.Source: §4.2 · p. 4, Figure 2 · p. 4

4.3 MOTION-DERIVED FORCE LABELS

• Observed object motion determines a target wrench, but not a unique hand-surface force distribution; the method allocates forces under geometric, frictional, and temporal constraints.Source: §4.3 · pp. 4–5
• Rigid-object target wrenches come from smoothed pose trajectories and Newton–Euler inverse dynamics, using a 1 kg reference mass and scaled inertia.Candidate hand vertices lie within 20 mm and obey local friction-cone constraints; an upward environment force accounts for support surfaces.Source: §4.3 · pp. 4–5
• The allocation constraints make hand and environment forces reproduce the motion-derived target wrench, while magnitude and temporal terms regularize feasible solutions.Articulated sequences receive vertex-force supervision but are excluded from the rigid rollout loss.Source: §4.3 · p. 5

5.1 VIDEO ENCODING AND INTERACTION-FIELD DECODING

• A pretrained ViT-L DINOv3 backbone encodes each frame; a temporal transformer aggregates clip information and three task queries decode fields for both hands at each 10 Hz timestamp.Architecture and input settings are provided in Appendix C.1.Source: §5.1 · p. 6

5.2 DENSE CONTACT AND PROXIMITY

• Contact probability factors into hand-level engagement and conditional vertex contact; engagement also gates vertex forces, while a separate proximity head uses masked mean squared error.Binary cross-entropy supervises engagement, and HACO’s vertex-level class-balanced loss supervises conditional contact on engaged frames.Source: §5.2 · p. 6

5.3 VERTEX FORCES AND ANALYTIC WRENCH AGGREGATION

• Vertex-force components are predicted along two local tangents and the outward normal, reconstructed in camera coordinates, supervised at image timestamps, and analytically aggregated into a hand wrench.The local basis is defined by ground-truth posed MANO geometry; the point-force loss balances active and near-zero targets.Source: §5.3 · p. 6

5.4 MOTION-CONSISTENT ROLLOUT

• Each 10 Hz visual interval uses predicted forces for its image-aligned 30 Hz step and detached motion-derived reference forces for the two intermediate steps.The rollout objective is restricted to eligible rigid sequences for which the aggregated hand force can be attributed to the target object.Source: §5.4 · pp. 6–7, Figure 4 · p. 7
• Differentiable integration combines the hand wrench with gravity and estimated environment-support terms, allowing motion-level errors to propagate back to predicted vertex forces.The rollout compares propagated and observed object position and orientation.Source: §5.4 · p. 6

5.5 TRAINING OBJECTIVE

• The training objective combines contact, proximity, point-force, and rollout losses, activating force losses only when their annotations and sequence eligibility support them.Geometry-only sequences use contact and proximity supervision; eligible rigid sequences additionally activate rollout loss, and padded repeated frames are excluded from force and rollout losses.Source: §5.5 · p. 7

6.1 DENSE PROXIMITY AND CONTACT COMPARISON

• TactoHand lowers full-range proximity ADE across all four benchmarks; the temporal variant leads on ARCTIC, DexYCB, and HOCap, while the single-frame variant leads on MOW.ADE-100 separates accuracy within the shared distance range from methods’ different full ranges.Source: §6.1 · p. 7, Table 1 · p. 8
• The temporal model improves contact Macro-F1 over HACO on DexYCB, HOCap, and ARCTIC, with the largest gain on HOCap, from 18.0 to 50.5.On MOW, TactoHand matches HACO at Macro-F1 34.5 and improves Micro-F1 from 41.5 to 44.3; Figure 5 shows representative missed and spurious regions.Source: §6.1 · pp. 7–8, Table 1 · p. 8, Figure 5 · p. 8

6.2 FORCE AND ROLLOUT ABLATION

• For the temporal model, physics supervision lowers rollout ADE from 5.7 to 3.3 cm and active-force MAE from 126.4 to 122.7 mN.The ablation compares single-frame and temporal variants with and without rollout loss while holding other training settings fixed.Source: §6.2 · pp. 8–9, Table 2 · p. 9

6.3 DOWNSTREAM HAND–OBJECT RECONSTRUCTION REFINEMENT

• On DexYCB, HOCap, and MOW, TactoHand improves PA-VPE in all six method–dataset pairs and yields the lowest hand–object Chamfer distance in each.The controlled refinement replaces the original contact objective with terms derived from fixed TactoHand contact and proximity predictions.Source: Table 3 · p. 9, §6.3 · p. 9

• Geometry-derived contact depends on source geometry and alignment, while motion-derived vertex forces are non-unique estimates that depend on inertia, friction, and environmental support assumptions.Motion-level evaluation is limited to short-horizon, reference-assisted rigid-body rollouts.Source: §7 · p. 9
• The authors identify measured forces and articulated rollout supervision as future directions, alongside reducing reliance on auxiliary physical inputs.Source: §7 · p. 9
• The reported rollout evidence is bounded to short-horizon, reference-assisted rigid-body evaluation.Source: §7 · p. 9

• TactoHand predicts contact, proximity, and 3D force as dense interaction fields on the hand surface.Source: §8 · p. 9
• In the reported evaluations, TactoHand improves contact and proximity estimation, and those fields improve controlled hand–object reconstruction.Source: §8 · p. 9
• Physics supervision reduces trajectory error in the reference-assisted rigid-body evaluation.The paper presents the shared hand-surface representation as connecting visual prediction, object motion, and hand–object reconstruction.Source: §8 · p. 9

• The main text defines the hand-surface fields, label construction, model, and evaluation.Source: REPRODUCIBILITY STATEMENT · p. 10
• Appendices A–D document data sources and labels, additional results, model and training settings, and evaluation protocols.Source: REPRODUCIBILITY STATEMENT · p. 10
• The authors state that they will release code and derived annotations to support reproduction of the labels and trained model.Source: REPRODUCIBILITY STATEMENT · p. 10

AI USE STATEMENT

• Generative AI tools assisted with retrieval and discovery of related work.The authors also used them to draft sections and polish the writing.Source: AI USE STATEMENT · p. 10
• Generative AI tools assisted with drafting sections of the paper and polishing its writing.Source: AI USE STATEMENT · p. 10
• The authors take responsibility for the final content, including its claims, citations, figures, and results.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

• Force labels are estimates, not measurements, and non-unique local force fields can explain one motion-derived wrench; they should not be treated as ground truth in safety-critical settings.The authors record validity flags and residuals, and do not substitute zero-force targets when allocation is infeasible.Source: ETHICS STATEMENT · p. 10

A.1 DATASET INVENTORY

• Table 4 inventories 24 curated sources and their label types; contact and proximity come from canonical-MANO geometry, while motion-derived force labels are grouped by rigid or articulated object motion.Collection counts are converted-frame counts across camera views, not counts of full dataset releases or training samples.Source: §A.1 · p. 15, Table 4 · p. 15

A.2 GEOMETRIC CONTACT-LABEL COMPARISON

• Contact labels use source-specific distance thresholds plus normal filtering and inside-mesh inclusion, which removes back-facing false positives and recovers penetrating vertices missed by thresholding alone.For open or unreliable surfaces, the method retains unsigned distance and skips inside-mesh inclusion.Source: §A.2 · p. 16, Figure 6 · p. 16

A.3.2 RIGID-OBJECT FORCE ALLOCATION AND VALIDATION

• Rigid-object force targets come from smoothed 30 Hz trajectories and inverse dynamics, then a constrained allocation distributes the target wrench across candidate hand vertices and environment support.The allocation uses geometric candidates, friction constraints, and temporal penalties; fitted wrench residuals and one-step replay assess validity.Source: §A.3.2 · pp. 17–18

A.3.3 ARTICULATED-OBJECT FORCE LABELS

• Articulated-object force targets subtract generalized gravity from inverse-dynamics effort, then allocate contact forces through a part-aware map while leaving unexplained joint effort as non-hand contribution.Valid articulated vertex forces supervise fields, but do not use the rigid rollout loss.Source: §A.3.3 · p. 18

B ADDITIONAL RESULTS AND VISUALIZATIONS

• Across all six method–dataset pairs, TactoHand improves joint errors and hand–object F-scores over the base and HACO variants.Table 5 reports the full reconstruction results; Figures 7 and 8 illustrate HOCap alignment and isolate the contact and proximity terms’ effects on individual EasyHOI examples.Source: B ADDITIONAL RESULTS AND VISUALIZATIONS · p. 18, Table 5 · p. 19

C.1 NETWORK ARCHITECTURE

• A six-layer temporal transformer processes per-frame visual features, and three learned queries decode force, contact, and proximity fields for both hands at each image timestamp.The frame encoder is a trainable DINOv3 ViT-L; single-frame variants remove cross-frame attention while retaining the shared modules and output shapes.Source: §C.1 · pp. 19–20

C.2 FORCE COORDINATES, TEMPORAL SAMPLING, AND SUPERVISION

• The rollout trains on predicted forces for the first force interval in each visual interval and detached reference forces for the next two; invalid labels and padded boundary frames are excluded from force and rollout losses.Predicted forces are expressed using a posed-MANO local basis before camera-frame supervision and wrench aggregation.Source: §C.2 · p. 20

C.3 TRAINING CONFIGURATION

• The final-stage variants share training settings, with dataset sampling weights based on available 32-frame windows and dataset-specific multipliers.Runs initialize from earlier checkpoints but reset optimizer, scheduler, and epoch state; physics variants ramp rollout-loss weight to 10^-4 by epoch six.Source: §C.3 · pp. 20–21, Table 6 · p. 21

D.1 CONTACT AND PROXIMITY COMPARISONS

• Contact and proximity comparisons use nominal dataset splits, with HOCap excluded from training and treated as cross-dataset transfer.The protocol re-scores released checkpoints on geometry-derived annotations; MOW repeats each image across the 32-frame temporal input because no additional temporal observations are available.Source: §C.3 · p. 21, §D.1 · p. 21

D.2 DOWNSTREAM RECONSTRUCTION PROTOCOL

• Downstream reconstruction compares base, HACO, and TactoHand variants under shared model, initialization, image, and optimization settings within each reconstruction method.The predicted contact and proximity values remain fixed during refinement, where proximity and contact-normal terms replace the original contact objective while other reconstruction terms remain.Source: §D.2 · p. 22

D.3 FORCE AND ROLLOUT EVALUATION

• Force metrics compare predictions with motion-derived reference vectors at valid image timestamps; vertex MAE includes zero-force targets, while active MAE and force angle use targets above 0.05 N.The four single-frame/temporal and physics-off/on variants are evaluated on DexYCB test data.Source: §D.3 · p. 23
• Rollout metrics use a 0.4 s reference-assisted horizon, and reported errors describe only non-divergent windows within the recorded sequence.Each visual interval uses predicted forces for its first force interval and detached reference forces for the next two; windows are excluded for non-finite states or speeds above 100 m/s or 1000 rad/s.Source: §D.3 · p. 23