PaperlayerView exact revision on arXiv

Source-linked AI summary

VERA-Hand: A Verified Multidimensional Benchmark for 2D Hand Pose Estimation in Third-Person Dynamic Gesture Videos

Gibran Benitez-Garcia, Hiroki Takahashi

arXiv:2610.32517v1cs.CV

TL;DR

Frame-wise localization evaluations do not capture missed or spurious hands, handedness errors, and temporal instability, limiting the characterization of hand-pose reliability. VERA-Hand evaluates six estimators using verified 2D annotations and complementary metrics, finding that rankings change substantially across dimensions rather than one estimator dominating.

  • Problem

    Frame-wise localization metrics are computed over matched predictions and do not account for missed or spurious hands, handedness errors, or temporal stability.

    Source: §1 · p. 2

  • Method

    VERA-Hand provides manually verified frame-level 2D annotations for 10,690 hands and evaluates six estimators across four dimensions, including joint detection-localization Pose-F1.

    Source: §1 · pp. 2–3

  • Results

    No single pipeline dominates across evaluation axes: rankings change substantially between dimensions, and localization alone gives an incomplete account of reliability.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Reliability comparisons are more actionable when they report the dimensions relevant to an application rather than collapsing performance into one score.

    Source: §5 · p. 8

    The benchmark covers only third-person dynamic gestures from IPN Hand, so its observed rankings should not be assumed to transfer to egocentric, close-range, or other domains.

    Source: §5 · p. 8

Abstract

from arXiv · show

Hand landmarks serve as input for gesture recognition, sign language understanding, and touchless human-computer interaction. Yet hand pose estimators are typically evaluated with frame-wise localization metrics such as PCK or MPJPE, which are computed over matched predictions and overlook missed or spurious hands, handedness errors, and temporal instability. To address these limitations, we present VERA-Hand, a benchmark that evaluates estimators using manually verified, frame-level 2D annotations for 10,690 hands across four complementary dimensions (detection, localization, identification, and temporal stability), as well as a joint Pose-F1 metric that requires both correct detection and accurate localization. Evaluating six estimators spanning hand-specific, whole-body, reconstruction-based, and foundation-model paradigms, we find that no estimator dominates: rankings reorder sharply across dimensions. VERA-Hand also exposes failures that localization-only and downstream evaluations may not reveal, including cross-hand overlap in over 85% of frames for a recent foundation model. A post-hoc cleanup case study demonstrates how the benchmark can validate corrective interventions, while stratified analyses show that robustness varies across motion-blur classes and hand regions. We release the verified annotations, protocol, and code at https://gibranbenitez.github.io/VERA_Hand/

1. Introduction

Frame-wise localization metrics do not capture missed or spurious hands, handedness errors, or temporal stability, despite the importance of hand landmarks to downstream applications. VERA-Hand addresses this gap with verified annotations and a multidimensional evaluation, finding that estimator rankings vary by dimension and that targeted cleanup can change them.

1. Introduction

  • Localization metrics evaluate matched predictions, leaving missed and spurious hands, handedness identification, and temporal stability insufficiently characterized.This limitation matters in continuous gestures, where blur, scale changes, face occlusion, frame exits, and absent hands occur.Source: §1 · p. 2
  • To the authors’ knowledge, no prior resource combines fully verified frame-level 2D annotations with unified evaluation across four dimensions in third-person continuous dynamic gestures.The four dimensions are detection, localization, identification, and temporal stability.Source: §1 · p. 2
  • VERA-Hand provides verified annotations for 10,690 hands and evaluates six estimators across detection, localization, identification, and temporal stability, with Pose-F1 and stratified analyses.The stratifications examine motion-blur classes and hand regions.Source: §1 · pp. 2–3
  • WiLoR leads in detection but ranks second-last in localization, while MP-hands leads in localization despite substantially higher identification error.The reversals show that no estimator dominates across dimensions, so localization alone gives an incomplete account of reliability.Source: §1 · p. 2
  • Targeted post-hoc cleanup improves affected estimators and reorders detection and Pose-F1 rankings, with only minor changes in pipelines lacking the targeted failures.The cleanup addresses cross-hand overlap, duplicate predictions, and detection gaps.Source: §1 · pp. 2–3

2. Related Work

Prior hand-pose resources vary in annotation verification, exhaustiveness, and supported evaluation dimensions. VERA-Hand complements existing work with verified third-person gesture annotations and a unified evaluation protocol.

2.1. Hand pose datasets and annotation resources

  • Existing datasets span varied acquisition settings, but complete frame-level manual verification and exhaustive all-hand annotation remain uncommon.Third-person gesture corpora may release estimator-generated landmarks without human verification, leaving no prior resource known to support all four axes under complete verification.Source: §2.1 · p. 3

2.2. Hand pose estimator paradigms

  • Hand-specific, whole-body, reconstruction-based, and foundation-model pipelines differ in how they detect and represent hands, yet their recovery, identification, and temporal behavior is less systematically characterized.These systems are commonly used as complete pipelines while comparisons often focus on keypoint localization.Source: §2.2 · p. 3

2.3. Evaluating and benchmarking pose estimators

  • Prior comparisons use downstream tasks, controlled perturbations, or spatial benchmarks, while VERA-Hand complements them with exhaustive verified 2D annotations and a unified four-axis protocol.Figure 2 depicts annotation initialization and correction alongside examples of detection, localization, and identification evaluation.Source: §2.3 · p. 3, Figure 2 · p. 4

3. Benchmark Construction and Protocol

VERA-Hand combines manually verified frame-level 2D hand annotations with a protocol for evaluating complete pose-estimation pipelines across complementary reliability dimensions. Its construction, annotation audit, matching rules, joint metric, stratifications, and off-the-shelf setup define the scope of the comparison.

3. Benchmark Construction and Protocol

  • VERA-Hand combines manually verified, frame-level 2D hand pose annotations for third-person continuous dynamic-gesture videos with a multidimensional evaluation protocol.The protocol evaluates detection, localization, identification, and temporal stability.Source: §3 · p. 4

3.1. Source videos and benchmark composition

  • Annotations use the IPN Hand test split, sampled at category-dependent rates, so supervised metrics use annotated frames while temporal metrics use full videos.The benchmark has 10,690 hands on 10,768 frames from 13 subjects and 52 videos; each visible hand receives a left/right label and 21 keypoints, with occluded keypoints annotated and out-of-frame points marked separately.Source: §3.1 · p. 4

3.2. Ground-truth quality and annotation bias

  • An independent audit found that 268 of 3,500 sampled frames required corrections, and all identified issues were corrected in the released annotations.Manual correction substantially changed the initialization on the bias-free subset; results are reported on that subset unless stated otherwise.Source: §3.2 · pp. 4–5

3.3. Evaluation axes and matching

  • Detection measures recovery without regard to label, localization scores keypoints on matched hands, identification measures unambiguous label swaps, and temporal metrics characterize prediction dynamics without temporal ground truth.Matching uses hand-box IoU; detection is label-agnostic, while localization and Pose-F1 require same-label matching.Source: §3.3 · p. 5

3.4. Pose-F1: Joint Detection and Localization

  • Pose-F1 jointly evaluates detection and localization by counting a prediction as a true positive only when it matches a same-label ground-truth hand and meets the PCK threshold.The main paper reports α=0.10 and β=0.5; unmatched predictions and ground-truth hands are counted as false positives and false negatives.Source: §3.4 · p. 5

3.5. Stratified reporting

  • Stratified reporting uses hand-region motion and sharpness signals to classify blur, and reports localization separately for the wrist, palm joints, fingertips, and all 21 keypoints.Blur thresholds were calibrated by visual inspection; region-wise results indicate which anatomical areas differentiate estimators.Source: §3.5 · pp. 5–6

3.6. Estimators and evaluation setup

  • The comparison uses six off-the-shelf estimators with official defaults and no dataset-specific tuning, maps native outputs to a common 21-keypoint topology, and evaluates cleanup separately.Table 2 summarizes the estimators, paradigms, default detectors, and native keypoints before mapping.Source: §3.6 · p. 6, Table 2 · p. 6

4. Results and Analysis

Estimator rankings reverse across reliability dimensions, and targeted cleanup improves the metrics affected by specific failures. Stratified results further show that robustness varies with blur, hand region, and estimator.

4.1. No single estimator dominates

  • Four estimators account for five metric-wise best results, with Pose-F1 led by RTMW-wb (0.477).WiLoR leads Det-F1 (0.978) and Jjerk (1.181), while MP-hands leads PCK (0.573).Source: §4.1 · p. 6, Table 3 · p. 6

4.2. Per-dimension findings

  • RTMW-wb leads Pose-F1 (0.477) by combining strong detection (0.932) with competitive localization (0.56), despite leading neither axis individually.MP-hands’ localization advantage yields Pose-F1 of 0.389, while WiLoR drops from first in detection to second in Pose-F1.Source: §4.2 · p. 7
  • MP-hands and RTMPhand have label-swap rates of 4.41% and 3.51%, while WiLoR has the lowest jitter (Jjerk: 1.18) despite weak localization.Low jitter can reflect limited predicted motion, so the paper cautions against interpreting temporal smoothness as correctness in isolation.Source: §4.2 · p. 7

4.3. Stratified analysis: blur and region

  • Pose-F1 declines from sharp to blurry for all estimators; RTMW-wb leads all three blur classes, but falls from 0.57 to 0.39.MP-hands drops from 0.49 on sharp instances to 0.27 under blur.Source: §4.3 · p. 7, Figure 3 · p. 7
  • Right-hand wrist PCK ranges from 0.16 to 0.57 across estimators, compared with 0.52 to 0.68 at the fingertips.Sapiens2 is weakest at the wrist (0.16), while WiLoR is weakest at the palm (0.36); PCK is computed over correctly matched predictions.Source: §4.3 · p. 7, Figure 3 · p. 7

4.4. Cross-hand overlap and post-hoc cleanup

  • Sapiens2 predicts an opposite-label second hand over the real hand in 85.44% of benchmark frames, more than an order of magnitude above any other estimator.The overlap occurs mostly when only one hand is present and nearly disappears when both hands are visible.Source: §4.4 · pp. 7–8
  • Cleanup reduces Sapiens2 cross-hand overlap from 85.44% to 0.05% and raises its Det-F1 from 0.666 to 0.94, moving it from last to third.MP-hands’ Pose-F1 rises from 0.389 to 0.442, overtaking WiLoR (0.419), while RTMW-wb remains highest (0.477).Source: Table 4 · p. 7, §4.4 · p. 8

5. Discussion and Limitations

The discussion frames estimator choice as application-dependent and positions verified component-level evaluation as complementary to downstream evaluation. It cautions that the findings are bounded by the dataset and by evaluation of complete released pipelines.

Discussion

  • The relevant reliability dimension depends on the application: identity consistency and precise keypoint localization may favor different pipelines.The paper argues that reporting complementary axes makes these trade-offs actionable rather than collapsing them into one score.Source: §5 · p. 8
  • Verified 2D references can reveal pose-estimator failures that downstream evaluation may overlook, so VERA-Hand complements rather than replaces task-level evaluation.The cleanup case study illustrates assessing targeted behavior while monitoring potential regressions elsewhere.Source: §5 · p. 8

Limitations

  • The benchmark covers only third-person dynamic gestures from IPN Hand, so its rankings should not be assumed to transfer to egocentric, close-range, or other domains.The dataset is strongly right-hand-dominant, with only 255 left hands, making left-hand estimates less conclusive.Source: §5 · p. 8
  • Results compare complete released pipelines, not isolated pose heads; they do not identify the best detector–pose combinations.The main benchmark evaluates only the Sapiens2-1B checkpoint with official weights.Source: §5 · p. 8

6. Conclusion

VERA-Hand provides a manually verified, frame-level benchmark and multidimensional protocol for evaluating hand pose estimators. Its comparisons reveal axis-dependent rankings and estimator-specific failures, while a cleanup case study shows the protocol can assess targeted interventions.

6. Conclusion

  • VERA-Hand combines verified frame-level 2D annotations with evaluation of detection, localization, identification, temporal stability, and joint Pose-F1.The comparison covers six off-the-shelf estimators.Source: §6 · p. 8
  • No single pipeline dominates across evaluation axes; rankings change substantially between dimensions, making localization alone an incomplete account of reliability.The benchmark also exposes estimator-specific failures such as cross-hand overlap.Source: §6 · p. 9
  • The post-hoc cleanup case study shows that the protocol can assess targeted corrective interventions and monitor their effects across reliability axes.The annotations, protocol, and code are released to support reproducible multidimensional evaluation.Source: §6 · p. 9

Supplementary Material

The supplementary material documents methods and extended analyses supporting the main paper. It covers annotation, estimator reproducibility, quantitative results, cleanup, and qualitative comparisons.

Supplementary Material

  • The supplement reports the annotation audit and describes how the audited sample is characterized.This material appears in Section A.Source: Supplementary Material · p. 11
  • It documents estimator configurations and keypoint conversions needed for reproducibility.These details appear in Section B.Source: Supplementary Material · p. 11
  • Extended analyses cover blur, hand regions, temporal stability, annotation bias, benchmark-size sensitivity, Sapiens2 overlap, cleanup ablations, and qualitative comparisons.These analyses are distributed across Sections C–F.Source: Supplementary Material · p. 11

A. Annotation Audit

The supplement audits annotation reliability, documents estimator pipelines, and extends quantitative analyses across blur, hand regions, temporal stability, annotation bias, and benchmark size. Results show estimator- and metric-dependent robustness, while audit corrections and initialization bias qualify the comparisons.

A. Annotation Audit

  • Across 3,500 audited frames, 268 (7.7%) required correction, mostly local keypoint displacements; five missed hands were added, with no handedness or gross-pose errors found.The audit deliberately covered all rare left-only and bimanual frames rather than sampling proportionally.Source: A. Annotation Audit · p. 11

B. Estimator Setup and Keypoint Conversion

  • RTMP-hand lacks native handedness labels, so its detections are assigned sides using compatible wrist anchors or, when unavailable, predicted hand geometry.The keypoint outputs are mapped to the common 21-keypoint MediaPipe topology; Sapiens2 and WiLoR require additional transformations.Source: B. Estimator Setup and Keypoint… · pp. 11–12

C.1. Motion blur

  • Detection is nearly blur-invariant for MP-holistic, Sapiens2-1B, and WiLoR, while RTMP-hand drops by 0.19 and MP-hands by 0.13; PCK and Pose-F1 degrade for all estimators.Identification is largely unaffected, and no pipeline remains uniformly robust across metrics.Source: §C.1 · p. 12

C.2. Hand region

  • Wrist PCK ranges from 0.16 to 0.57 across estimators, the clearest regional separation; Sapiens2-1B is weaker at the wrist but competitive over palm and whole hand.WiLoR’s lowest regional score is over the palm.Source: §C.2 · pp. 12–13, Table 7 · p. 13

C.3. Temporal stability

  • Temporal stability is measured on the right hand; the steadiest estimators also predict the least motion, so low motion energy can reflect a low-motion prior rather than tracking accuracy.Fingertips show higher jitter than base joints, and cleanup reduces gaps by approximately 37% to 67% while slightly increasing Jjerk.Source: §C.3 · p. 13, Table 8 · p. 13, Table 9 · p. 13

C.4. Annotation bias

  • On the bias-free subset, 76% of keypoints moved by more than α=0.02, with median normalized displacement 0.06 versus 0.008 for biased hands; MP-holistic’s localization improves when evaluated on the full set.Approximately 1,381 missing hands were added and 1,387 spurious detections removed, supporting use of the bias-free subset for the main supervised comparison.Source: §C.4 · p. 13, Table 11 · p. 14

C.5. Benchmark size sensitivity

The benchmark-size analysis recomputes supervised metrics on deterministic, nested per-video frame subsets. Estimator ordering remains stable down to 10%, but rare failures can disappear at that size.

C.5. Benchmark size sensitivity

  • Subsets are selected deterministically within each video by retaining chronologically equispaced annotated frames, including each clip’s first and last annotated frame.This preserves the proportions of frames across videos and subjects while distributing samples across each clip.Source: §C.5 · p. 14
  • Estimator ordering is preserved on all four metrics down to 10% of the benchmark, including Sapiens2 cross-hand overlap, which remains above 82% at every fraction.Detection varies by less than 0.01 across fractions; localization and Pose-F1 fluctuate more at the smallest fractions.Source: §C.5 · p. 14, Table 12 · p. 14
  • WiLoR’s cross-hand overlap is 0.07% on the full benchmark but reaches zero at 10%, where too few frames remain for such a rare event to appear.This makes measurement of infrequent failures the size-sensitive exception.Source: §C.5 · p. 14

D. Sapiens Family Analysis

Sapiens2-1B’s cross-hand overlap substantially reduces detection performance despite its stronger localization and identification results. The analysis links this failure mainly to opposite-handed duplicate predictions on single-hand frames, and cleanup largely restores detection without changing localization.

D. Sapiens Family Analysis

  • Sapiens2-1B has substantially better localization and a lower label-swap rate than Sapiens1-1B, yet its raw Det-F1 is well below Sapiens1-1B’s.Opposite-handed phantom predictions reduce its precision.Source: D. Sapiens Family Analysis · p. 14
  • Cross-hand overlap is rare in two-hand frames but dominant for Sapiens2-1B in single-hand frames, where it usually duplicates the visible hand with the opposite handedness.A full-body prediction prior favoring bilateral outputs may contribute to this pattern; VERA-Hand’s predominance of single-hand frames makes it measurable.Source: D. Sapiens Family Analysis · p. 14
  • Removing cross-hand overlap raises Sapiens2-1B Det-F1 from 0.666 to 0.929 while PCK and MPJPE remain effectively unchanged.The temporal stage adds a smaller detection gain and removes the remaining duplicate-hand predictions.Source: D. Sapiens Family Analysis · pp. 14–15
  • In qualitative examples, Sapiens2-1B localizes keypoints better but sometimes overlays a second prediction on the only visible hand, while Sapiens1-1B can miss a heavily blurred hand.The examples also show Sapiens2-1B correctly predicting both hands in one frame where Sapiens1-1B overlaps the right hand and misses the blurred left hand.Source: Figure 6 · p. 15

E. Post-hoc Cleanup: Method and Ablation

The post-hoc cleanup uses only predicted hands, applying spatial overlap resolution before temporal consistency rules. Its temporal components are complementary: together they outperform individual variants and a majority-vote baseline on detection and Pose-F1.

E. Post-hoc Cleanup: Method and Ablation

  • The cleanup uses no ground truth, image content, or localization metric, and applies spatial processing before temporal consistency checks.Its spatial stage resolves overlapping opposite-handed predictions within a frame; its temporal stage enforces short-term consistency across consecutive frames.Source: E. Post-hoc Cleanup: Method and Ablation · p. 15
  • Opposite-handed predictions are treated as overlapping when bounding-box IoU is at least 0.80 and normalized center distance is at most 0.15; the lower-confidence hand is removed.The center distance is normalized by the larger box diagonal, and the retained hand has the higher mean keypoint confidence.Source: E. Post-hoc Cleanup: Method and Ablation · p. 15
  • The temporal stage deduplicates labels, removes non-persistent predictions, and recovers supported gaps by interpolating keypoints, subject to motion and gap-length limits.It uses a seven-frame window: gap recovery requires at least four supporting neighbors on both temporal directions, fills gaps of at most three frames, and skips large endpoint motion.Source: E. Post-hoc Cleanup: Method and Ablation · p. 15
  • On MP-hands, combining all three temporal components outperforms every individual variant on Det-F1 and Pose-F1, while majority vote remains below the full stage on all three metrics.Gap recovery provides the main individual detection gain but reduces PCK; phantom removal improves PCK and Pose-F1 at a small detection cost, while deduplication changes scores little.Source: E. Post-hoc Cleanup: Method and Ablation · p. 15
  • The paper applies spatial cleanup first, then temporal cleanup; the spatial stage drives Sapiens2-1B’s recovery, while the temporal stage reduces tracking gaps and increases jitter.Both stages together yield the raw-versus-clean comparison in Table 4.Source: E. Post-hoc Cleanup: Method and Ablation · p. 15

F. Qualitative Results

Qualitative comparisons show that estimator failures are pipeline-dependent and that localization-only comparisons omit important detection and identification errors. The examples align with ranking reversals and highlight blur, face overlap, and partial exits from the frame as common difficult cases.

F. Qualitative Results

  • A frame that defeats one estimator is often handled correctly by another, and no estimator is consistently weakest across the qualitative examples.This pattern is the qualitative counterpart of the reported ranking reversals.Source: F. Qualitative Results · p. 16, Figure 7 · p. 17, Figure 8 · p. 18
  • Motion blur, hands overlapping the face, and hands partly leaving the frame account for most failures shown in Figure 8.Figure 7 instead shows all hands recovered, with residual deviations confined to individual keypoints, typically fingertips or wrists.Source: F. Qualitative Results · p. 16
  • Localization-only comparisons capture keypoint deviations on recovered hands but miss absent hands, spurious predictions, grossly wrong poses, and predictions on frames with no hands.In the no-hand example, two pipelines still predict a hand; those frames contribute no localization measurement and are scored only on detection.Source: F. Qualitative Results · p. 16
Loading 2610.32517v1…