Source-linked AI summary
Single-Query Person-Centric Bimanual Hand-Object Interaction Detection
Jonghyun Kim, Junho Roh, Yubin Yoon, Hyotae Lee, Jongkuk Park, Taehwan Hwang, Jaechul Kim, Jungho Lee
TL;DR
Hand-centric interaction methods leave the pairing of two hands with the same person implicit, a concern in multi-person scenes. The paper uses one query to predict a person’s body, hands, and interaction targets, with part-aware attention and query-based target selection. Experiments report improved tuple-level performance over direct object regression and gains from pose learning and part-aware references.
Problem
Hand-centric methods do not explicitly couple left and right hands under a shared person, making ownership ambiguous in multi-person scenes.
Source: §1 · pp. 2–3Method
A single query predicts a person’s body, left/right hands, and their interaction targets, using part-aware references and hand-to-query target selection.
Source: §1 · p. 2Results
Experiments report improved tuple-level performance from relation-based target selection, stronger hand reasoning with joint pose learning, and better localization with part-aware references.
Source: §7 · p. 15Takeaways & Limitations
Structured tuple evaluation measures hand-state accuracy and ownership-consistent interactions under increasingly strict localization requirements.
Source: §7 · p. 15Performance drops in scenes with ≥5 people, where severe occlusion, hand overlap, and small visible hands remain challenges.
Source: §6.3 · p. 15
Abstract
from arXiv · showhide
Understanding person-level bi-manual interactions requires not only detecting hands, but also identifying which two hands belong to the same person and what each hand interacts with. Existing hand--object interaction methods are mostly hand-centric: they treat each hand as an independent instance, which can lead to ambiguous ownership in multi-person scenes. We propose a person-centric formulation in which a single query predicts a structured output for one person, including the human box, body pose, hand boxes and states, and interaction targets. We introduce part-aware deformable attention to allocate attention across human, hand, and pose-specific reference regions, enabling one query to capture the full person structure. We further unify detection and interaction reasoning with a hand-to-query relationship matrix, where each hand selects its interaction target from the detected query set plus a learnable off token, directly recovering the target's box and class without separate object regression. We build a COCO-based dataset with person-centric bi-manual interaction annotations and define structured metrics for evaluating hand states and complete hand--object tuples. Experiments with a transformer-based detector show that our formulation improves person-level bi-manual interaction parsing and provides an effective unified framework for joint detection, pose estimation, and hand reasoning.
1 Introduction
Hand-centric methods do not explicitly pair left and right hands under a shared person, leaving person-level ownership ambiguous in multi-person scenes. The paper proposes a single-query, person-centric framework that jointly predicts a person’s body, hands, and interaction targets, supported by a COCO-based dataset and structured evaluation.
1 Introduction
- Hand-centric methods treat hands independently, leaving left/right ownership under one person ambiguous in multi-person scenes.This reflects a mismatch between independent-hand prediction and person-level interpretation.Source: §1 · pp. 2–3
- A single person query predicts the human box, body pose, left/right hand boxes and states, and hand-specific interaction targets.The human instance is the primary prediction unit, with hands attached as explicit slots.Source: §1 · pp. 2, 4
- Part-aware deformable attention uses human, hand, and pose reference regions to capture person context and fine hand details without task-specific queries.The design builds on deformable transformers.Source: §1 · p. 2
- For each hand, a relationship matrix selects a target from detected queries plus a learnable off token, retrieving its box and class without separate object regression.Queries can represent either a human or an object.Source: §1 · pp. 2–4
- The COCO-based dataset combines human boxes, body keypoints, hand boxes and contact labels, and object boxes and classes; tuple metrics penalize ownership inconsistencies.The annotations support joint detection, pose estimation, and hand–object reasoning.Source: §1 · pp. 3–4, Figure 1 · p. 3
- Experiments with a transformer-based detector show person-centric bi-manual parsing can be accurate and scalable.The paper presents this as validation of the proposed formulation.Source: §1 · p. 3
2 Related Work
Prior work includes hand-centric interaction datasets and methods, human–object interaction detection, entity-set relation prediction, and transformer detectors. This work differs by structuring two hands under a person and evaluating complete person–hand–object tuples.
Hand–object interaction understanding
- Hand-centric datasets such as 100DOH and Hands23 provide per-hand interaction signals but do not make person-level bi-manual ownership a first-class target.The paper attaches left/right hands as structured slots to each human instance.Source: §2 · p. 4
Human–object interaction detection and relation prediction
- HOI methods model human–object pairs and predicates, while scene-graph methods predict entity relations; this work instead selects hand-specific targets within a person-centric structure.The relation matrix operates over detected queries rather than forming a general predicate graph.Source: §2 · p. 4
DETR-style detectors and evaluation protocols
- Rather than proposing a new detector family, the paper adds person-centric representation and part-aware references to transformer detection, with tuple metrics that capture ownership inconsistencies.Existing hand-interaction metrics do not directly penalize cross-person assignment or left/right slot confusion.Source: §2 · p. 5
3 Method
The method makes each person the prediction unit, with one query producing person, pose, hand, and interaction outputs. Part-aware attention supports this unified representation, while relation-based target selection links each hand to detected entities or an off option.
3.1 Task and Person-Centric Output Representation
- A single query represents a person with a class and box, body keypoints, left/right hand boxes and embeddings, and hand-contact outputs.For person queries, interaction output can use direct object-box regression or relation-based target selection; hand- and pose-related outputs from non-person queries are not supervised.Source: §3.1 · pp. 5–6
3.2 Part-Aware Deformable Attention with Multiple Reference Points
- Part-aware deformable attention uses human, hand, and joint reference regions so one query can combine instance context with local part cues without separate part queries.The design distributes decoder attention across references and uses head-wise query structure for global-to-local interactive learning.Source: §3.2 · pp. 6–7
3.3 Joint Detection and Hand–Object Interaction via Relation Matrix
- The relation formulation makes each hand select a detected query or learnable off option, retrieving a selected target’s box and class without explicit object-box regression.Off represents no contact; selecting the person’s own query denotes self-contact, while another query denotes contact with another entity.Source: §3.3 · p. 8
3.4 Training and Inference
- The RT-DETR-style model is trained end-to-end with bipartite matching and detection, hand-box, pose, and hand–object relation supervision; inference retrieves targets from selected queries.At inference, top-K queries are retained by detection confidence, and person queries output pose, hand boxes, and interaction targets.Source: §3.4 · p. 8
4 Dataset
The dataset combines COCO person, object, and keypoint annotations with Hands23 hand annotations to create person-centric bi-manual interaction records. Its construction aligns hands and contact targets to COCO entities, with a manually annotated validation split and verification of training labels.
4.1 Overview
- COCO supplies diverse multi-person scenes, object boxes, and human keypoints, while Hands23 hand annotations are converted into person-centric hand slots and interaction targets.The resulting dataset supports joint detection, body-pose estimation, and person-centric hand-interaction learning.Source: §4.1 · pp. 8–9
4.2 Person–Hand Pair Construction
- Hands23 hands are matched to COCO people by wrist-guided geometric rules, with ambiguous multiple matches assigned to the person with the nearest wrist-to-hand-center distance.The pipeline injects matched boxes into left/right hand slots and uses image, local-crop, and relative-depth cues to verify or correct ambiguous mappings.Source: §4.2 · p. 9
4.3 Constructing Interaction Targets
- Hand-contact labels are mapped to COCO entities by IoU: object and other-person contacts select the highest-IoU matching entity, self-contact selects the current person, and no-contact selects off.Unmatched object contacts receive a generic pseudo-target; samples without a valid other-person match are discarded.Source: §4.3 · p. 10
4.4 Validation Set Construction
- A human-verified validation split uses approximately 2K COCO val2017 images containing people, with hand boxes and contact labels in the person-centric schema.Interaction targets use the same assignment rules as the training data.Source: §4.4 · p. 10
4.5 Annotation reliability.
- On a 770-sample audit, verification reduced annotation error from 2.5% to 2.1%; two annotators achieved 98.4% agreement on 500 samples.Uncertain cases are excluded under a no-guessing policy; remaining errors mainly involve visually ambiguous hands, occlusion, and difficult depth configurations.Source: §4.5 · pp. 10–11
5 Evaluation
Evaluation matches predicted and ground-truth persons one-to-one using person-box IoU, then measures hand-state accuracy at increasing levels of interaction localization requirements.
5 Evaluation
- All three metrics use one-to-one IoU-based person matching and retain pairs with person-box IoU above τp = 0.5.The matching follows the cited assignment protocol, with the threshold following standard detection overlap criteria.Source: §5 · p. 11
- The soft metric measures hand-state accuracy over matched persons, counting annotated left and right hands.The hand-availability mask excludes hands without annotations.Source: §5 · p. 11
- The medium metric additionally requires correct interacted-object localization for hold states, while the hard metric also requires correct hand localization.The default object-box and hand-box IoU thresholds are both 0.5.Source: §5 · p. 11
6 Experiments
Experiments use the COCO-based person-centric dataset, with ContactHands providing an additional hand-state-only evaluation. Ablations favor relation-based target selection, pose supervision, and part-aware references for interaction reasoning, while performance drops in highly crowded scenes.
6.1 Experimental Setup
- Training aligns COCO human instances and keypoints with hand annotations and adds converted 100DOH data with pseudo labels; ContactHands evaluation reports only Accsoft.The validation split uses manually annotated COCO val2017 images for hand-interaction evaluation.Source: §6.1 · pp. 11–12
6.2 Ablation Studies
- Accmid rises from 44.9 to 60.8 (+15.9) and Acchard from 12.6 to 28.9 (+16.3) with Relation versus DirectBox, despite similar detection mAP (49.1 vs. 48.8).COCO Accsoft falls from 81.5 to 80.0, while ContactHands Accsoft rises from 80.7 to 81.7.Source: §6.2 · p. 12, Table 2 · p. 13
- With pose, Relation Accmid rises from 60.8 to 64.6 (+3.8) and Acchard from 28.9 to 31.3 (+2.4), while detection mAP falls from 48.8 to 47.1.Relation+Pose also yields the best overall Accsoft on COCO (83.8) and ContactHands (82.2).Source: §6.2 · p. 13, Table 2 · p. 13
- Adding 100DOH raises COCO Accmid from 64.6 to 66.2 (+1.6) and Acchard from 31.3 to 32.7 (+1.4), but ContactHands Accsoft falls from 82.2 to 81.2.The authors attribute the stronger tuple-metric gains to 100DOH’s explicit interacted-object boxes; its supervision is less aligned with ContactHands’ hand-state-only evaluation.Source: §6.2 · pp. 13–14, Table 3 · p. 13
- Part-aware references raise Relation+Pose Acchard from 10.3 to 31.3 (+21.0); without them, pose AP drops to about 42 and strict hand metrics degrade.Detection mAP can be slightly higher with shared references, but the authors identify part-aware allocation as essential for reliable interactions under strict localization criteria.Source: §6.2 · p. 14, Table 4 · p. 14
6.3 Crowding Analysis
- Performance drops in COCO val2017 images with at least five people; severe occlusion, hand overlap, and small visible hands remain challenges.The analysis stratifies the final model by the number of people per image.Source: §6.3 · p. 15, Table 5 · p. 15
7 Conclusion
The conclusion presents a person-centric framework that jointly predicts human, pose, hand, and interaction outputs, supported by part-aware attention and relation-based target selection. Experiments show gains from relation modeling, pose learning, part-aware references, and additional 100DOH supervision; broader domains and severe occlusion remain future-work areas.
7 Conclusion
- A single query represents one person and jointly supports detection, body pose estimation, and bi-manual interaction reasoning.Part-aware deformable attention supplies multiple reference regions, while the relation head selects hand targets from detected queries with an off option.Source: §7 · p. 15
- Experiments report stronger tuple-level performance from relation-based target selection, improved hand reasoning with pose learning, and further gains from 100DOH supervision.Part-aware reference allocation is reported as critical for accurate pose and hand localization, especially under the strictest evaluation criterion.Source: §7 · p. 15
- Future work includes broader domains and interaction types, better robustness under severe occlusion and crowded scenes, and interaction semantics beyond contact-level reasoning.Source: §7 · p. 15