Source-linked AI summary
Understanding Human Hands in Contact at Internet Scale
Dandan Shan, Jiaqi Geng, Michelle Shu, David F. Fouhey
TL;DR
The paper addresses the challenge of extracting rich hand state from diverse Internet videos, where prior methods often rely on constrained settings. It introduces a contact-aware hand detector trained on large-scale video and annotated frames, and reports strong detection generalization while enabling mesh reconstruction applications.
Problem
Internet videos contain hands across highly varied viewpoints, contexts, and scales, challenging methods developed for in-lab or pre-localized settings.
Method
The paper trains a Faster-RCNN-based model to predict hand boxes, side, contact state, and contacted-object boxes using 100DOH and a 100K annotated frame dataset.
Results
90% VOC AP is reported for hand detection, with cross-dataset generalization, while the system supplies inputs for deploying hand-mesh reconstruction on Internet videos.
Takeaways & Limitations
The model and data provide a foundation for hand-contact understanding, mesh reconstruction, and proof-of-concept learning about human-object interaction in consumer videos.
Takeaways & Limitations
Common failures involve precise contact-state classification, associating hands with objects amid multiple interactions, and determining hand side for egocentric views.
Abstract
from arXiv · showhide
Hands are the central means by which humans manipulate their world and being able to reliably extract hand state information from Internet videos of humans engaged in their hands has the potential to pave the way to systems that can learn from petabytes of video data. This paper proposes steps towards this by inferring a rich representation of hands engaged in interaction method that includes: hand location, side, contact state, and a box around the object in contact. To support this effort, we gather a large-scale dataset of hands in contact with objects consisting of 131 days of footage as well as a 100K annotated hand-contact video frame dataset. The learned model on this dataset can serve as a foundation for hand-contact understanding in videos. We quantitatively evaluate it both on its own and in service of predicting and learning from 3D meshes of human hands.
1. Introduction
The paper targets hand-interaction understanding in diverse Internet videos, where prior methods largely assume constrained, in-lab settings. It introduces a rich hand-state representation, large-scale data, and downstream applications including reconstruction.
- Motivation: Internet videos challenge hand analysis because viewpoint, context, and hand scale vary widely.Examples range from fifty-pixel hands in cooking videos to thousand-pixel closeups in DIY videos.
- Representation: The model predicts each hand’s box, left/right side, contact state, and the contacted object’s box from a single RGB image.Contact states include none, self, other person, non-portable object, and portable object.
- Applications: The system combines hand-state prediction with existing reconstruction methods to detect contact, reconstruct hands in 3D, and identify touched objects.A self-supervised method assesses mesh quality for learning from Internet reconstructions.
- Results: 90% VOC AP is reported for hand detectors, with good cross-dataset generalization and occasional superiority to same-dataset training and testing.The authors also demonstrate mesh reconstruction and a proof-of-concept mapping objects to 3D hand meshes.
2. Related Work
The work positions contact-centered hand understanding between scalable image or video detection and richer but more constrained sensing approaches. It extends prior interaction datasets and supplies inputs for mesh reconstruction from Internet videos.
- Human-object interaction: Prior human-object interaction work commonly recognizes verb-noun pairs, whereas this paper represents physical contact and interaction.The approach is technically most related to Gkioxari et al. but proposes a different output representation.
- Video datasets: Relative to VLOG, the paper expands interaction data to broader diversity and annotates contact more thoroughly; it is also complementary to AVA’s below-activity representation.The paper builds on ideas and part of VLOG’s data.
- Hand analysis: The dataset and approach balance scalability with annotation richness between image-based detection datasets and specialized contact-sensing environments.The authors expect complete hand understanding may require multiple complementary approaches.
- Dataset comparison: Table 1 compares 100DOH with existing human-object interaction datasets, emphasizing its large unlabeled video volume for unsupervised learning.Only a small fraction is labeled compared with more densely annotated datasets.
- Mesh reconstruction: The approach provides hand locations and sides required by mesh methods, enabling their large-scale deployment to Internet videos.It also includes a self-supervised system for detecting reconstruction failures.
3. Dataset
The dataset combines massive unlabeled YouTube interaction videos with a 100K labeled frame subset, gathered implicitly across diverse everyday activities. Filtering and annotation produce rich hand-contact supervision for scalable modeling.
- Video collection: Implicit gathering uses generic queries to find candidate videos, followed by filtering for hands, interaction, and non-cartoon content.The process begins with an overcomplete candidate set and removes irrelevant videos.
- Video collection: 13.2K queries across 11 categories produced approximately 6.5M YouTube responses, estimated at 86 years of footage, before filtering.The categories include cooking, DIY, gardening, housework, packing, and repairs.
- Video collection: Thumbnail models predict hand presence, human-judged interaction, and cartoons, enabling rapid large-scale filtering before frame-level annotation.Subsequent annotations are independent of the video-level filtering mechanism.
- Annotation: Each annotated hand receives a box, side, contact state, and a box around the contacted object, including unnamed objects.The contact taxonomy covers no contact, self, other person, portable object, and non-portable object.
- Dataset properties: The image dataset fills a gap by providing object-contact and side information at scale, with hands appearing across varied image sizes.The dataset comparison reports broader hand-scale coverage than existing hand datasets.
4. Finding Hands & Objects in Interaction
The system extends Faster-RCNN to detect hands and contacted objects, infer hand side and contact state, and associate each hand with an object. Its outputs support mesh reconstruction, reconstruction-failure detection, and downstream grasp-learning experiments.
- Hand and object detection: The base system predicts hand and object boxes, hand side, contact state, and hand–object links from a single image.Contact states are none, self, person, portable object, and non-portable object.
- Hand and object detection: Faster-RCNN is augmented with auxiliary per-region predictions and losses while retaining its standard detection outputs.The model uses shared ROI-pooled features for side, contact-state, and association predictions.
- Hand and object detection: The association module predicts a direction and magnitude from each hand to a contacted object, allowing multiple hands to share one object.The formulation does not support one hand contacting multiple objects, which the authors leave for future work.
- Hand and object detection: A greedy post-processing step converts soft detections, contact scores, and association vectors into discrete hand–object correspondences.It suppresses unlikely detections and associates confident hands with nearby object centers according to the predicted relationship.
- Applications to reconstruction: The system supplies hand locations and side information to an existing MANO-based method for large-scale 3D hand-mesh reconstruction.The paper also estimates whether reconstructions are likely correct using joint consistency across rotated image copies.
5. Experiments
Experiments evaluate hand detection, full hand-state prediction, and downstream mesh reconstruction across datasets, scales, data sizes, and human judgments. Results show strong cross-dataset detection and useful reconstruction support, while complete hand-state prediction remains difficult.
- Hand Bounding-Box: 100DOH-trained detectors retain 92.9% of same-dataset mAP at worst when evaluated across datasets.VLOG and VGGHands generalize less well, while VIVA and EgoHands generalize poorly beyond egocentric views.
- Hand Bounding-Box: 49.5% recall is the effective ceiling for OpenPose hand detection, while Faster R-CNN reaches 99.7% precision at that recall.The comparison supports using dedicated hand detectors alongside full-body pose estimation.
- Hand Bounding-Box: 0.08% and 0.11% AP from median-box baselines show that hands are widely distributed across images and vary in size.The baselines use median boxes for all hands and separately for left and right hands.
- Full Hand State: The full system usually recognizes hands and sides across varied scales and contexts, but complete hand-state prediction remains harder and leaves substantial room for improvement.Reported failure modes include precise contact state, hand-object association with multiple people or objects, and hand side in egocentric views.
- Full Hand State: Tripling training data from 15K to 45K yields large gains, whereas doubling from 45K to 90K gives more incremental improvements.All-output correctness still suffers a 7% AP hit when using less than the full training set.
- Hand State for Reconstruction: 57.8% of workers judged meshes reconstructed with the detected hand side correct, versus 29.1% for the opposite side.The evaluation used 3,861 detections reconstructed with both side choices, producing 7,722 meshes assessed by crowdworkers.
- Hand State for Reconstruction: 90% AUROC identifies plausible hand reconstructions using self-consistency labels, compared with 89% for Gaussian Naive Bayes and 60% for an unlabeled Gaussian likelihood baseline.The comparison indicates that the positive/negative labels matter more than the choice between the proposed MLP and Gaussian Naive Bayes.
- Future Prediction: Crowdworker judgments compare reconstructed grasps against random training-set hands, with 60% of cases producing a conclusive result.The test examines whether the system extracts a plausible grasp signal despite many possible correct grasps.
6. Conclusion
The paper presents a hand-contact representation, a large-scale dataset, and applications to reconstruction and human-object interaction. It frames these resources as a basis for further learning from Internet video.
- Conclusion: The paper contributes hand-contact information, a large-scale training dataset, and demonstrated applications of the technique.The authors describe this as an initial step toward learning from large-scale Internet video.
- Conclusion: The proposed representation is intended to help the field explore human-object interaction in general consumer videos.The conclusion characterizes the presented work as only an initial exploration of what can be learned from Internet video.