Source-linked AI summary
Joint-Conditioned Stereo Surface Reasoning for Interaction Field Estimation
Yanlin Jin, Yifan Yang, Bowen Yang, Kai Zhu
TL;DR
Hand–object interaction fields require locating each joint’s nearest object-surface point despite small, occluded image regions and depth ambiguity. JSSR combines temporal-stereo predictions with calibrated candidate search, shared hand-level surface evidence, and gated geometric correction. Full JSSR improves mean ADE over the temporal baseline, and the system ranked third on the challenge leaderboard with its fine-tuned ensemble submission.
Problem
Predicting interaction fields requires resolving each joint’s nearest object-surface point despite small image regions, mutual occlusion, and depth ambiguity.
Source: §1 · p. 1Method
JSSR jointly predicts 3D joints, a direct field, and per-view endpoint evidence, then refines joint-conditioned endpoints using calibrated stereo search, hand-shared candidates, and a learned residual gate.
Source: §1 · p. 1, §2.1 · p. 2, §2.2 · pp. 2–3Results
Full JSSR reduces mean ADE from Temporal7’s 16.194 mm to 15.740 mm, and the fine-tuned 50/50 two-expert ensemble scored 32.61 for third place.
Source: §3 · p. 4Takeaways & Limitations
The results show that the complete stereo-reasoning system improves mean ADE over the temporal baseline, while the best leaderboard score used object-specific fine-tuning and an expert ensemble.
Source: §3 · p. 4Historical leaderboard scores use different training settings and are not scores of the epoch-8 ablation checkpoints.
Source: §3 · p. 4
Abstract
from arXiv · showhide
Predicting hand--object interaction fields requires locating the nearest object-surface point for each hand joint, often from small and partially occluded image regions. We view this task as joint-conditioned surface-endpoint estimation: each joint has its own nearest endpoint, while endpoints from the same hand can draw on shared local surface evidence. This structure motivates Joint-Conditioned Stereo Surface Reasoning (JSSR). A temporal-stereo network jointly predicts 3D joints, a direct interaction field, and per-view endpoint evidence. Calibrated candidate search evaluates endpoint hypotheses using joint-specific image compatibility and cross-view correspondence. A hand-shared candidate support lets joints draw on common surface evidence, and a learned residual gate controls the geometric correction when observations are ambiguous. Our system built on this method ranked third on the SHOW3D Interaction Field Challenge leaderboard.
1 Introduction
Interaction fields require finding each hand joint’s nearest object-surface point despite small image regions, occlusion, and depth ambiguity. JSSR uses calibrated stereo evidence and shared surface hypotheses to refine joint-specific endpoints while retaining direct predictions as a fallback.
1 Introduction
- SHOW3D represents each visible hand with 21 vectors from its joints to nearest object-surface points, which are difficult to predict under occlusion and depth ambiguity.Source: §1 · p. 1
- A field endpoint is projectable across calibrated views, while joints on one hand can share surface hypotheses but must retain joint-specific nearest-point selection.Source: §1 · p. 1
- JSSR combines temporal-stereo predictions, calibrated search, hand-shared support, and a learned residual gate that applies geometric correction when evidence is reliable.Source: §1 · pp. 1–2, Figure 1 · p. 2
2 Method
The method predicts joint-conditioned endpoints from temporal stereo, evaluates calibrated candidates using image and cross-view evidence, and applies surface-based field corrections through an evidence-dependent gate.
2.1 Network Overview
- A shared DINOv3 encoder processes seven synchronized stereo pairs, and a Transformer decoder assigns one query to each of the 42 joints.Hand-aware temporal aggregation summarizes neighboring observations; the queries predict joints, direct fields, and per-view endpoint evidence.Source: §2.1 · p. 2
2.2 Joint-Conditioned Stereo Surface Reasoning
- K = 32 candidate depths are sampled along each calibrated ray, then scored using joint-conditioned image compatibility and stereo feature agreement.A reliability-weighted D4RT descriptor adds an auxiliary logit that reorders the fixed candidates.Source: §2.2 · pp. 2–3
2.3 Evidence-Gated Field Refinement
- Joint-specific surface estimates correct a direct–epipolar parent only through a learned gate, which is set to zero for invalid candidates.The gate predicts how far to move toward the surface field and is supervised by the interpolation coefficient that best approaches ground truth.Source: §2.3 · p. 3
3 Experiments
The experiments compare fixed epoch-8, clip-held-out ablations and historical challenge submissions. Full JSSR improves mean ADE over the temporal baseline, while the best leaderboard score came from fine-tuning and an expert ensemble.
3 Experiments
- Evaluation uses 72,307 training and 12,201 evaluation targets from contiguous held-out clips with no temporal overlap, after removing 535 geometrically inconsistent annotations.All DINOv3 variants train independently for eight epochs, and results use epoch 8 without model-specific selection.Source: §3 · p. 3
- Historical leaderboard scores use different training settings and are not scores of the epoch-8 ablation checkpoints.The Temporal7 submission uses D4RT, and the endpoint submission includes epipolar search.Source: §3 · p. 4
- 16.194, 15.925, and 15.740 mm: mean ADE for Temporal7, Two-view Endpoints, and Full JSSR; Full JSSR improves on Temporal7 by 0.454 mm (2.80%).The best single-model challenge submission scored 33.59; object-specific fine-tuning and a fixed 50/50 two-expert ensemble scored 32.61 for third place.Source: §3 · p. 4, Table 1 · p. 4
A Shared Local Surface Support
The method pools local 3D surface hypotheses into a shared support for each hand, while each joint selects its own endpoint from that support.
A Shared Local Surface Support
- All 21 joint queries for a hand select from one local 3D candidate support, a soft surface constraint that does not reconstruct a mesh or require CAD or object pose.Source: A Shared Local Surface Support · p. 5
A.1 Candidate construction
Candidate construction projects each parent endpoint into camera 0 and builds a hand-level set of surface hypotheses around it. The shared set is filtered to hypotheses valid in both calibrated views.
A.1 Candidate construction
- 21 parent endpoints each generate five image-plane offsets and nine log-spaced depth hypotheses, yielding 945 candidates.The depth multipliers span [exp(−0.25), exp(0.25)].Source: §A.1 · p. 5
- The spatial search radius adapts to projected endpoint spread and is clamped to [0.04, 0.08] in normalized image coordinates.Source: §A.1 · p. 5
- After construction, candidates form a hand-level set available to every joint query; invalid or out-of-view hypotheses are masked after projection into both views.Source: §A.1 · p. 5
A.2 Learned stereo candidate score
The learned stereo candidate score combines sampled two-view appearance with explicit projective geometry. It evaluates shared hand-level candidates using both joint-specific compatibility and a candidate-quality score shared across joints.
A.2 Learned stereo candidate score
- Candidate features combine two-view appearance with image position, disparity, and relative depth, using mean, absolute difference, and elementwise product cues.An MLP forms the candidate representation, with depth expressed relative to the hand's median parent-endpoint depth.Source: §A.2 · p. 5
- Each joint's candidate logit combines joint-specific compatibility with a learned candidate-quality scalar shared across joints.Source: §A.2 · pp. 5–6
- Calibration determines where evidence is sampled, while the MLP learns how stereo appearance and projective geometry indicate a plausible local surface point.Source: §A.2 · p. 6
B Training the Surface Distribution and Gate
The surface distribution is trained from field-distance targets, and the estimated surface field adjusts rather than replaces the parent prediction. A learned gate controls that correction, while shared support couples endpoint choices without forcing joints to select the same point.
B Training the Surface Distribution and Gate
- Distances from candidate endpoints to the ground-truth field define a soft target distribution for the KL loss, alongside endpoint supervision of the expected surface field.The field error and σ are expressed in millimeters.Source: B Training the Surface Distribution… · p. 6
- The surface estimate is applied as a residual correction to the parent prediction rather than replacing it.Source: B Training the Surface Distribution… · p. 6
- A zero-initialized tanh gate uses the joint representation, pooled surface feature, selection entropy, and expected candidate score to control the correction.The gate is forced to zero when no valid candidate exists and trained toward the best convex interpolation coefficient.Source: B Training the Surface Distribution… · p. 6
- Shared support couples endpoint distributions through one candidate set and stereo scoring function, but does not force joints to choose the same point.Here, “surface” means shared stereo-validated endpoint support, not an explicitly reconstructed object surface.Source: B Training the Surface Distribution… · p. 6