PaperlayerView exact revision on arXiv

Source-linked AI summary

IPN Hand: A Video Dataset and Benchmark for Real-Time Continuous Hand Gesture Recognition

Gibran Benitez-Garcia, Jesus Olivares-Mercado, Gabriel Sanchez-Perez, Keiji Yanai

arXiv:2005.02134v2cs.CV

TL;DR

Public continuous-HGR datasets do not cover transition-free gestures and natural hand movements as non-gesture actions, despite the need for online systems to handle these real-world cases. IPN Hand introduces a varied touchless-screen dataset and evaluates 3D-CNNs and RGB-derived modalities. RGB-seg is reported as a suitable real-time alternative, with an accuracy advantage over RGB at comparable inference time.

  • Problem

    Public hand-gesture datasets do not cover continuous gestures without transition states or natural hand movements as non-gesture actions, which matter for online HGR applications.

    Source: §I · p. 1

  • Method

    The paper introduces IPN Hand and evaluates three 3D-CNN methods for isolated and continuous real-time HGR, including RGB fusion with semantic segmentation or optical flow.

    Source: §I · p. 2

  • Results

    Around 10% improvement in Levenshtein accuracy is reported for ResNet-50 RGB-seg over ResNeXt-101 RGB at comparable inference time.

    Source: 1) Experimental results · p. 7

  • Takeaways & Limitations

    The authors conclude that RGB-seg is a suitable multimodal alternative for real-time continuous HGR and that IPN Hand can serve as a benchmark.

    Source: §V · p. 7

Abstract

from arXiv · show

In this paper, we introduce a new benchmark dataset named IPN Hand with sufficient size, variety, and real-world elements able to train and evaluate deep neural networks. This dataset contains more than 4,000 gesture samples and 800,000 RGB frames from 50 distinct subjects. We design 13 different static and dynamic gestures focused on interaction with touchless screens. We especially consider the scenario when continuous gestures are performed without transition states, and when subjects perform natural movements with their hands as non-gesture actions. Gestures were collected from about 30 diverse scenes, with real-world variation in background and illumination. With our dataset, the performance of three 3D-CNN models is evaluated on the tasks of isolated and continuous real-time HGR. Furthermore, we analyze the possibility of increasing the recognition accuracy by adding multiple modalities derived from RGB frames, i.e., optical flow and semantic segmentation, while keeping the real-time performance of the 3D-CNN model. Our empirical study also provides a comparison with the publicly available nvGesture (NVIDIA) dataset. The experimental results show that the state-of-the-art ResNext-101 model decreases about 30% accuracy when using our real-world dataset, demonstrating that the IPN Hand dataset can be used as a benchmark, and may help the community to step forward in the continuous HGR. Our dataset and pre-trained models used in the evaluation are publicly available at https://github.com/GibranBenitez/IPN-hand.

I. INTRODUCTION

Continuous HGR must handle gestures without neutral transitions and natural hand movements, but existing public datasets omit these scenarios. IPN Hand targets that gap with a varied touchless-screen benchmark evaluated using 3D-CNNs and RGB-derived modalities.

I. INTRODUCTION

  • Continuous HGR must handle gestures without transition states, natural hand movements resembling gestures, variable gesture duration, and lag between a gesture and its classification.A touchless-screen example is opening a picture and zooming in with consecutive gestures, without returning the hand to a neutral position.Source: §I · p. 1
  • Existing public continuous-HGR datasets do not cover gestures without transition states or natural hand movements as non-gesture actions.The authors note that some automotive-interface datasets partially include these issues, but are not publicly available.Source: §I · p. 1
  • IPN Hand combines touchless-screen gestures with transition-free sequences, natural hand movements as non-gestures, and varied scenes and illumination.It contains more than four thousand RGB gesture samples, 800 thousand frames, 50 subjects, and 13 static and dynamic gesture classes.Source: §I · p. 2
  • The study evaluates three 3D-CNN methods for isolated and continuous real-time HGR and tests RGB fusion with segmentation or optical flow.It also compares results with the publicly available nvGesture dataset.Source: §I · p. 2

II. RELATED WORKS

Continuous-HGR datasets vary in scale, gesture domain, and treatment of non-gesture frames. Prior systems use multi-stream or hierarchical 3D-CNN approaches, and this work evaluates a hierarchical approach with RGB-derived modalities for real-time use.

A. Datasets for continuous HGR

  • Public continuous-HGR datasets differ in scale, classes, sensors, and gesture domain; IPN Hand is reported to have 21 instances per video across 28 scenes.ChaLearn ConGD has 249 classes and more than 40 thousand instances; nvGesture has 25 gesture types and 1,532 instances, while EgoGesture has 83 classes and more than 20 thousand instances.Source: A. Datasets for continuous HGR · p. 2
  • Non-gesture examples differ across datasets, while IPN Hand includes natural hand movements, making gesture-versus-non-gesture discrimination more challenging.The cited comparison describes neutral transition states in ChaLearn ConGD, hands at the steering wheel in nvGesture, and backgrounds with hands out of view in EgoGesture.Source: A. Datasets for continuous HGR · p. 2, Figure 2 · p. 2

B. Continuous HGR

  • Prior continuous-HGR systems include multi-stream RGB-and-depth models, temporal dilated 3D-CNNs, and hierarchical 3D-CNNs for spotting and classification.The paper follows the hierarchical structure and evaluates RGB-derived multimodality for real-time performance, using data-level fusion to limit computational cost.Source: B. Continuous HGR · p. 3

III. THE IPN HAND DATASET

IPN Hand videos were collected across varied recording scenes, with participants performing a sequence of gestures in each recorded session.

III. THE IPN HAND DATASET

  • Videos were recorded at 640 × 480 resolution and 30 fps using participants’ own PCs or laptops, producing 28 scenes and variation in camera distance.Three prepared scenes included clutter and variable illumination for participants without cameras meeting the recording specifications.Source: A. Data collection · p. 3
  • Each participant was given randomized gesture-name lists for four videos and performed 21 gestures with three random breaks in a recorded session.The scene examples show different subjects performing the same two-finger pointing gesture.Source: A. Data collection · p. 3, Figure 3 · p. 3

B. Dataset characteristics

The dataset defines 13 touchless-screen gestures, collected from a student cohort, and includes scenes with extreme lighting and cluttered or dynamic backgrounds.

B. Dataset characteristics

  • The 13 touchless-screen gestures comprise two static pointer gestures and 11 actions for manipulating interfaces.The actions include clicks, directional throws, double-clicks, zoom-in, and zoom-out; natural hand movements also appear as non-gesture states.Source: B. Dataset characteristics · p. 3

2) Subjects:

  • The 50 participants were students aged 18–25, with a mean age of 21.2 and a gender breakdown of 16 females and 34 males.All participants attended Instituto Politécnico from Mexico.Source: 2) Subjects · p. 3

3) Extreme illumination and clutter backgrounds:

  • The collection includes strong sunlight, almost no artificial light, cluttered static backgrounds, and dynamic backgrounds with people walking through view.These conditions were included to evaluate baseline robustness to illumination changes and cluttered backgrounds.Source: 3) Extreme illumination and clutter… · pp. 3–4

C. Dataset statics

IPN Hand comprises 4,218 gesture instances and 800,491 RGB frames from 50 subjects across 13 gesture classes and 28 scenes. Its duration statistics show substantial variation in how quickly different subjects perform the same gesture.

C. Dataset statics

  • The dataset contains 4,218 gesture instances and 800,491 RGB frames from 50 subjects performing 13 gesture classes across 28 scenes.Figure 5 distributes videos by scene ID, with scene IDs on the horizontal axis and video counts on the vertical axis.Source: C. Dataset statics · p. 4, Figure 5 · p. 4
  • Each video records a semi-random sequence of gestures, with manually labeled start and end frames for supervised continuous recognition.The 200 videos each record a session of 21 gestures; gesture durations range from 9 to 650 frames.Source: C. Dataset statics · p. 4
  • Duration statistics compare public continuous-gesture datasets, using normalized standard deviation within each class to characterize gesture-speed variation.The paper averages the class-wise nstd_k values across the dataset.Source: C. Dataset statics · pp. 4–5
  • 0.43 is IPN Hand’s largest duration nstd_k, above EgoGesture, indicating greater speed variation among subjects performing the same gesture.The comparison is reported in Table III.Source: C. Dataset statics · p. 5, Table III · p. 5
  • IPN Hand’s statistics characterize a dataset containing continuous gestures without transition states, natural non-gesture hand movements, and varied gesture durations.The paper identifies these as real-world issues covered by the dataset.Source: C. Dataset statics · p. 5

IV. BENCHMARK EVALUATION

The benchmark evaluates three state-of-the-art 3D-CNN models on isolated and continuous hand gesture recognition. For continuous recognition, it uses a two-model hierarchy to detect and classify gestures in a video stream.

IV. BENCHMARK EVALUATION

  • Three state-of-the-art 3D-CNN models are evaluated as baselines on IPN Hand for isolated and continuous hand gesture recognition.Source: §IV · p. 5
  • The continuous HGR approach uses a two-model hierarchical architecture to detect and classify an incoming stream of frames.Source: §IV · p. 5
  • The benchmark covers both isolated and continuous HGR tasks using the new IPN Hand dataset.Source: §IV · p. 5

A. Hierarchical 3D-CNNs for continuous HGR

Continuous recognition uses a hierarchical detector-classifier pipeline, while the experimental setup holds out subjects for testing. Implementation details describe the compared networks, modality processing, and pretraining choices.

A. Hierarchical 3D-CNNs for continuous HGR

  • A sliding-window detector activates a classifier when it recognizes a gesture, and post-processing permits one final activation per performed gesture.The detector runs at every stride, whereas the classifier window is activated conditionally, so inference time is not always the sum of both models.Source: A. Hierarchical 3D-CNNs for continuous… · p. 5

B. Experimental setup

  • Subjects were split into 37 for training and 13 for testing, yielding 148 training videos and 52 testing videos.The respective gesture-instance counts were 3,117 and 1,101.Source: B. Experimental setup · p. 5

1) Implementation details:

  • Classification compares C3D, 3D ResNet-50, and 3D ResNeXt-101; the shallow detector uses ResLight, and all models train with manually segmented gestures.The isolated-task results and multimodality processing comparisons are presented in Tables IV and V, respectively.Source: 1) Implementation details · p. 5, Table IV · p. 6, Table V · p. 6
  • HarDNet segmentation masks and SPyNet optical flow provide additional modalities, with the latter computed from each input frame using an open-source pretrained model.The 3D-CNN models were pretrained on Jester, HarDNet on ImageNet, and ResLight was trained from scratch; inference speed was measured on a GTX 1080ti system.Source: 1) Implementation details · pp. 5–6

C. Isolated HGR task

The isolated HGR task predicts a class label for each manually segmented gesture sample. Performance is assessed using classification accuracy and a confusion matrix.

C. Isolated HGR task

1) Experimental results using the IPN Hand dataset:

On IPN Hand, ResNeXt-101 produced the best results but barely achieved real-time performance; RGB-seg was competitive with RGB-flow and faster. On nvGesture, RGB-seg results remained competitive with state-of-the-art methods.

1) Experimental results using the IPN Hand dataset:

  • ResNeXt-101 achieved the best IPN Hand results but could barely maintain real-time performance; ResNet-50 was most efficient and least accurate.Both models were compared with the robust C3D model, which ResNeXt-101 was faster and more accurate than.Source: 1) Experimental results using the IPN… · p. 6
  • RGB-seg achieved competitive IPN Hand results compared with RGB-flow, while semantic segmentation was more than twice as fast as optical flow.The paper presents inference time and extra-process model size as measures of multimodal efficiency.Source: 1) Experimental results using the IPN… · p. 6
  • The best isolated IPN Hand result, ResNeXt-101 with RGB-flow, confused gestures that were similar, such as clicks and double-clicks.Figure 7 presents the confusion matrix for that result.Source: Figure 7 · p. 6, 1) Experimental results using the IPN… · p. 6

2) Experimental results using the nvGesture dataset:

D. Continuous HGR task

Continuous HGR is evaluated by comparing predicted gesture sequences with ground truth using Levenshtein accuracy. The authors also report binary detection accuracy for isolated gesture-versus-non-gesture samples.

D. Continuous HGR task

  • Levenshtein accuracy measures sequence differences by counting insertions, deletions, and substitutions needed to transform predicted gestures into the ground-truth sequence.For the example in the paper, two edits across nine target classes yield 77.78% accuracy.Source: D. Continuous HGR task · p. 7
  • Continuous HGR performance is the average Levenshtein accuracy across 52 testing videos.Source: D. Continuous HGR task · p. 7
  • Gesture detectors are evaluated on isolated gesture-versus-non-gesture samples using binary classification accuracy and confusion matrices.Source: D. Continuous HGR task · p. 7

1) Experimental results:

The hierarchical detector-classifier evaluation highlights a real-time advantage for RGB-seg: ResNet-50 RGB-seg achieved around 10% higher Levenshtein accuracy than ResNeXt-101 RGB at comparable inference times. The authors conclude that RGB-seg is suitable for real-time continuous HGR.

1) Experimental results:

  • RGB misrecognized 85% of non-gesture frames, while the reported multimodal detection benefits were reflected in detecting those frames.The detector was evaluated with different modalities, and its compact ResLight-10 model's inference time was not affected by modality.Source: 1) Experimental results · p. 7
  • The authors evaluated a hierarchical two-model approach using ResLight-10 for detection and ResNet-50 or ResNeXt-101 for classification.They also compared multimodal alternatives for real-time performance.Source: 1) Experimental results · p. 7
  • Qualitative results showed the most significant network improvements on pixels belonging to large objects.Source: Figure 9 · p. 7
  • Around 10% improvement in Levenshtein accuracy was reported for ResNet-50 RGB-seg over ResNeXt-101 RGB, whose inference times were comparable.The comparison evaluated a hierarchical two-model process, with ResLight-10 as detector and ResNet-50 or ResNeXt-101 as classifier.Source: 1) Experimental results · p. 7

V. CONCLUSION

  • The authors conclude that RGB-seg is a suitable multimodal alternative for real-time continuous hand gesture recognition.They present IPN Hand as a benchmark containing real-world issues including gestures without transition states, natural hand behavior, and large duration variability.Source: §V · p. 7
Loading 2005.02134v2…