Source-linked AI summary
Functional Hand Type Prior for 3D Hand Pose Estimation and Action Recognition from Egocentric View Monocular Videos
Wonseok Roh, Seung Hyun Lee, Won Jeong Ryoo, Jakyung Lee, Gyeongrok Oh, Sooyeon Hwang, Hyung-gun Chi, Sangpil Kim
TL;DR
Egocentric action recognition can struggle to perceive dynamic hand movements from geometric or physical information alone. The paper introduces a functional hand-type taxonomy and a framework that uses per-frame hand-type semantics as a prior. It reports state-of-the-art results for 3D hand pose estimation and action recognition on FPHA and H2O.
Problem
Egocentric methods relying on geometric or physical information can struggle to perceive dynamic hand movements and distinguish semantic differences in hand–object interactions.
Source: §1 · p. 2Method
The paper defines a function-based hand-type taxonomy, annotates existing datasets, and uses per-frame hand-type semantics as a prior in its hand action-recognition framework.
Source: §1 · pp. 2–3Results
The method reports state-of-the-art performance on 3D hand pose estimation and egocentric action recognition, with experiments on FPHA and H2O.
Source: §1 · p. 3Takeaways & Limitations
The paper concludes that hand-type knowledge can be used as a semantic indicator for perceiving egocentric hand actions.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Current methods for egocentric view action recognition often face challenges in perceiving dynamic hand movements relying solely on geometrical or physical information. In this work, we effectively address this problem by gaining insights into the correlation between functional hand configurations and objects, which improves the detailed interpretation of real-world scenarios. To this end, we introduce a practical taxonomy of hand types based on the functioning perspective and utilize it for per-frame hand type labeling on existing datasets. We also propose a novel hand action recognition framework considering semantic details of the hand type as prior. This approach boosts the network's understanding of the continuous hand interaction throughout the action sequence. Our whole pipeline consists of three main modules: (1) Feature Extraction, (2) Egocentric Knowledge Module, which estimates 3D hand pose, object category, and hand type leveraging short-term cues, and (2) Egocentric Action Module, which aggregates per-frame knowledge, including text embeddings of hand type, over a longer time. In our extensive experiments with large-scale benchmarks, FPHA and H2O, our model outperforms current state-of-the-art methods, demonstrating its superior performance.
1 Introduction
Egocentric hand-action recognition must capture semantic differences in hand–object interactions that geometry alone can miss. The paper introduces a function-based hand-type taxonomy and uses per-frame hand-type semantics in an action-recognition framework, reporting state-of-the-art results on 3D hand pose estimation and action recognition.
1 Introduction
- Geometric hand positions and high-level object labels can leave dynamic hand movements difficult to interpret, including actions with the same goal but different hand–object interactions.The paper gives “Open Juice Bottle” and “Open Liquid Soap” as examples.Source: §1 · p. 2
- Functional hand-type labels address a gap in object-property taxonomies by representing how the hand operates and its role in an activity.The paper provides hand-type annotations for existing datasets.Source: §1 · pp. 2–3
- The framework uses per-frame hand-type semantics alongside 3D hand pose and object category, and reports state-of-the-art results for 3D hand pose estimation and egocentric action recognition.It is evaluated on FPHA and H2O.Source: §1 · p. 3
- The action-recognition framework uses hand-type semantics in each frame to represent continuous interactions among hands and objects across an action sequence.The authors describe this as the first use of hand-type semantic details for hand action recognition.Source: §1 · p. 3
2 Related Work
Prior work uses hand pose and temporal models to represent hand actions, while grasp taxonomies commonly organize types around object properties. This paper’s related-work discussion points to function-based categorization as a way to represent hand roles in activities.
3D Hand Pose Estimation & Hand Action Recognition
- Action-recognition approaches combine spatial hand information with temporal modeling, including transformer architectures that model relationships across input sequences.The paper adopts hierarchical transformers to learn spatial information and short-term geometric and long-term semantic cues.Source: §2 · p. 3
Hand Type Taxonomy
- The taxonomy figure distinguishes static grasp types from dynamic Control types intended to describe more real-life hand behaviors.The figure categorizes hand types by functionality.Source: Figure 2 · p. 4
- Prior grasp taxonomies describe object shape, hand surfaces, and hand shape; Napier et al. distinguish prehensile from non-prehensile movements and power from precision grasps, while this paper redefines types by functionality.Source: §2 · p. 4
3 Function-Based Hand Type Taxonomy
The taxonomy groups hand types by function and role, distinguishing independent hand types from those interacting with people or objects. It further separates holding from manipulation and annotates the FPHA and H2O datasets under this scheme.
3 Function-Based Hand Type Taxonomy
- The taxonomy is designed to capture hand function and role in everyday activities, supplementing object-characteristic taxonomies that primarily describe grasp manner and appearance.The authors identify limitations in traditional taxonomies for explaining varied everyday hand-action scenarios.Source: §3 · pp. 4–5
- The taxonomy divides hand types into Isolated and Interaction, then organizes Interaction into Grasp and Control according to hand function.Grasp covers holding an object over time; Control covers manipulating it over time.Source: §3 · p. 5
- Within Grasp, hand types divide by appearance into Power, Precision, and Intermediate, with Circular and Prismatic distinctions based on interacting-object shape.Control types are separately classified as Deformable or Non-Deformable according to whether the object deforms.Source: §3 · p. 5
- The authors annotate every frame in FPHA for the right hand and in H2O for both hands using the new taxonomy.FPHA provides information only for the right hand, while H2O provides data for both hands.Source: §3 · p. 5
4 Learning Functional Hand Type
This section adds functional hand type as semantic context to pose and object cues, targeting continuous interactions that geometry-only approaches struggle to capture. The framework predicts per-frame pose, object category, and hand type before aggregating their features for action recognition.
4 Learning Functional Hand Type
- Functional hand type supplies semantic cues about hand-object relationships and the continuous interaction between primary and assistive hands or objects.The section motivates this cue because geometry and physical information alone leave challenges in perceiving dynamic hand movements.Source: §4 · p. 5, §4.2 · pp. 6–7
4.1 Overview
- The framework combines per-frame pose, object, and hand-type information, with short-term estimation feeding a longer-span action module.It is built on a hierarchical temporal transformer framework.Source: §4.1 · p. 5, Figure 3 · p. 6
4.2 Egocentric Knowledge Module
- The knowledge module processes consecutive video segments in parallel, decoding per-frame 2D joints and depth while predicting hand-type and object categories.Camera intrinsics are used to infer camera-space 3D joint positions from the pose estimates.Source: §4.2 · pp. 6–7
4.3 Boosting the Action Recognition with Hand Type Prior
- Predicted hand-type labels are mapped to text descriptions and encoded by CLIP as semantic features for action recognition.Examples include “Hand Clench” and “Tip Pinch.”Source: §4.3 · p. 7
4.4 Egocentric Action Module
- The action module aligns and concatenates temporal, pose, object, and hand-type features before a Global Transformer and action-classification head.The concatenated features are reduced to the Global Transformer’s token dimension.Source: §4.4 · p. 7
4.5 Training
- End-to-end training combines action, pose, object, and hand-type losses, weighted by their respective hyperparameters.The pose, object, and hand-type losses are averaged over frames across the video segments.Source: §4.5 · p. 8
5 Experiments
The experiments evaluate hand action recognition and 3D hand pose estimation on FPHA and H2O, with ablations testing hand-type cues and qualitative pose comparisons.
5.2 Comparison with the State-of-the-Arts
- Action accuracy reaches 95.13% on FPHA and 89.67% on H2O, and the method estimates 3D hand pose more precisely than HTT.Table 1 reports RGB-video action accuracy; Table 2 covers root-aligned pose estimation.Source: Table 1 · p. 8, Table 2 · p. 9, §5.2 · p. 9
5.1 Experimental Setups
- FPHA and H2O provide supervision and evaluation labels for hand pose, action, and object category; hand types are annotated for all frames.Action recognition uses classification accuracy, while pose estimation uses PCK, AUC, and MEPE in camera and root-aligned spaces.Source: §5.1 · p. 8
5.3 Ablation Study
- Text embeddings of hand type improve accuracy over predicted hand-type vectors, and hand-type cues enhance action accuracy on FPHA and H2O.The predicted probability vector performs better than using hand type only as an auxiliary task.Source: §5.3 · p. 9, Table 3 · p. 9
5.4 Qualitative Analysis
- Estimated poses are shown alongside ground truth, and H2O root-aligned PCK curves compare the model with HTT.The authors describe the estimates as comparable to ground truth and report reasonable results for both hands.Source: Figure 4 · p. 10, §5.4 · p. 10
6 Conclusion
The conclusion presents a temporal-transformer method that uses functional hand-type knowledge for egocentric hand action recognition.
6 Conclusion
- The paper introduces a functionality-based hand-type taxonomy, annotates large-scale benchmarks, and reports outstanding performance for its approach.It frames hand-type semantics as a critical indicator for perceiving egocentric hand actions.Source: §6 · p. 10
Overview
The supplementary material expands on experiments, training, annotations, datasets, and hand-type distributions.
Supplementary material
- The supplement provides additional experimental results and visualizations for the main framework.These are presented in Section A.Source: Overview · p. 15
- Training implementation details are described in a dedicated section.These details appear in Section B.Source: Overview · p. 15
- The supplement elaborates on hand-type annotation, FPHA and H2O, and distributions under the functional taxonomy.These topics are covered in Sections C and D.Source: Overview · p. 15
A. Additional Experimental Results
Additional qualitative results compare estimated hand poses with ground truth and HTT on FPHA and H2O.
A. Additional Experimental Results
- On both FPHA and H2O, the visualized gap between the model's estimated poses and ground truth is smaller than HTT's.The comparison is qualitative, with estimates projected in 3D and onto corresponding 2D image frames.Source: A. Additional Experimental Results · p. 16, Figure 1 · p. 16
- The figure uses blue for the model's estimates, magenta for HTT, and green for ground truth.It visualizes hand poses in 3D and their projections onto image frames.Source: Figure 1 · p. 16
- H2O provides labels for both hands, whereas FPHA includes labels only for the right hand.Source: Figure 1 · p. 16
B. Implementation Details
The implementation uses a fixed video-feature training setup, while annotation assigns function-sensitive hand types frame by frame and tracks their changes through actions.
B. Implementation Details
- The model processes K RGB frames resized to 480 × 270 through segmented ResNet-18 features and a transformer, using 512-dimensional features and tokens.Training used Adam for 45 epochs, with a learning rate of 3×10−5 reduced tenfold every 15 epochs, and augmentation following.Source: B. Implementation Details · p. 16
C. Annotation Details
- Hand types distinguish function beyond grasp appearance, including “Poking” without deformation and “Extended Index Curl” applying pressure to deform an object.“Index Finger” describes holding an object while extending the index finger.Source: C. Annotation Details · p. 17
- Annotators label every frame using hand appearance and function, checking action-specific types first and assigning “Invisible” when the hand is occluded or out of frame.For unclear cases, they compare other subjects performing the same action and select the closest matching type.Source: C. Annotation Details · p. 17
- Annotations change with hand function over time: placing a box shifts from “Large Diameter” through “Relaxed Hand” to “Fist,” while High Five distinguishes “Open Hand” from “Dynamic Flatten.”The High Five hand shape remains the same; the label differs according to interaction with the other person’s hand.Source: C. Annotation Details · p. 17
D. Analysis of Datasets and Hand Type Distributions
The section presents a 31-type function-based taxonomy and examines its coverage and distribution in FPHA and H2O, across hand types and action labels.
D. Analysis of Datasets and Hand Type Distributions
- The taxonomy defines 31 functional hand types, grouping them first as Isolated or Interaction, then dividing Interaction into Grasp and Control and refining by appearance and degree of object manipulation.The labels were applied across all frames of both datasets.Source: D. Analysis of Datasets and Hand Type… · p. 18
- In the hand-type distributions, FPHA had 29 of 31 types identified, versus H2O’s 24 of 31, and operated more diverse hand types across more action scenarios.FPHA’s most common types were Large Diameter, Disk Grip, and Thumb Tucked; H2O’s were Large Diameter, Relaxed Hand, and Parallel Extension.Source: D. Analysis of Datasets and Hand Type… · p. 18
- The taxonomy’s radial diagram presents the hand-type categories, while FPHA and H2O distribution plots encode frame counts by hand type and frequency across action labels.The histogram axes are hand type and frame count; scatter plots map action labels against hand types, with circle size indicating frequency within action clips.Source: Figure 4 · p. 18, Figure 5 · p. 21
- FPHA has more monotonous hand-type distributions across actions than H2O, which the authors relate to FPHA’s right-hand-only labels and repeated simple actions.FPHA uses 29 types across 45 action scenarios; H2O uses 24 across 36 and is described as more balanced.Source: D. Analysis of Datasets and Hand Type… · p. 18
- The figure examples distinguish “Index Finger,” “Poking,” and “Extended Index Curl” by their functions despite their shared use of the index finger.The labels represent holding while extending the finger, poking without deforming an object, and applying pressure to deform it.Source: Figure 2 · p. 19
- H2O’s hand-type distribution figure separates left- and right-hand frame counts and action-label distributions, using circle size to show within-video frequency.The histograms map hand type to number of frames; the scatter plots map actions to hand types.Source: Figure 6 · p. 22