Source-linked AI summary
Real-time Hand Gesture Detection and Classification Using Convolutional Neural Networks
Okan Köpüklü, Ahmet Gunduz, Neslihan Kose, Gerhard Rigoll
TL;DR
Real-time hand-gesture recognition must locate gesture boundaries, avoid repeated activations, and respect memory and power limits. The paper combines a lightweight detector with an offline-trained deep classifier in a sliding-window hierarchy, and evaluates single-time outputs with Levenshtein accuracy. Across EgoGesture and nvGesture, the approach achieves early detections with performance close to offline operation.
Problem
Real-time gesture recognition must handle unknown gesture boundaries, single recognition per gesture, and memory and power constraints.
Method
A hierarchical sliding-window architecture uses a lightweight detector to activate an offline-trained deep CNN classifier only when gestures are detected.
Results
The system achieves 91.04% and 77.39% Levenshtein accuracy on EgoGesture and nvGesture, respectively, while providing considerable early detections.
Takeaways & Limitations
The architecture provides resource efficiency, early detections, and single-time activations for real-time gesture recognition.
Abstract
from arXiv · showhide
Real-time recognition of dynamic hand gestures from video streams is a challenging task since (i) there is no indication when a gesture starts and ends in the video, (ii) performed gestures should only be recognized once, and (iii) the entire architecture should be designed considering the memory and power budget. In this work, we address these challenges by proposing a hierarchical structure enabling offline-working convolutional neural network (CNN) architectures to operate online efficiently by using sliding window approach. The proposed architecture consists of two models: (1) A detector which is a lightweight CNN architecture to detect gestures and (2) a classifier which is a deep CNN to classify the detected gestures. In order to evaluate the single-time activations of the detected gestures, we propose to use Levenshtein distance as an evaluation metric since it can measure misclassifications, multiple detections, and missing detections at the same time. We evaluate our architecture on two publicly available datasets - EgoGesture and NVIDIA Dynamic Hand Gesture Datasets - which require temporal detection and classification of the performed hand gestures. ResNeXt-101 model, which is used as a classifier, achieves the state-of-the-art offline classification accuracy of 94.04% and 83.82% for depth modality on EgoGesture and NVIDIA benchmarks, respectively. In real-time detection and classification, we obtain considerable early detections while achieving performances close to offline operation. The codes and pretrained models used in this work are publicly available.
I. INTRODUCTION
Real-time gesture recognition must detect gesture boundaries, recognize each gesture once, react quickly, and operate within memory and power constraints. The proposed hierarchical CNN system addresses these requirements using a lightweight detector, an on-demand classifier, and Levenshtein-based evaluation.
- Real-time systems require classification accuracy, fast reaction, resource efficiency, and single-time activation for each gesture.
- Most prior work emphasizes offline classification accuracy, while some multi-CNN, multimodal systems exceed practical memory and power budgets.
- The proposed architecture uses a lightweight detector to activate an offline-trained deep classifier only when a gesture is detected.
- The system supports early detections and single-time activations, with Levenshtein distance measuring misclassifications, multiple detections, and missing detections together.
- 94.03% and 83.82% depth-modality offline classification accuracies are achieved with ResNeXt-101 on EgoGesture and nvGesture, respectively.
II. RELATED WORK
Prior gesture-recognition work uses wearable devices, keypoint representations, raw visual data, and CNN-based spatio-temporal models. The paper focuses on continuous-stream detection and single-time activation, which earlier approaches do not adequately address.
- 2D CNN approaches perform well but are limited in modeling temporal information and motion patterns, motivating 3D CNNs for spatio-temporal features.
- Real-time systems must detect and classify gestures continuously, while performed gestures must produce only one computer activation.
- The proposed work trains a lightweight 3D CNN detector and addresses single-time activations, which CTC-based detection does not provide.
- The architecture is presented as a way to use state-of-the-art CNN models efficiently in real-time applications.
A. Architecture
The architecture uses sliding windows and a hierarchical detector-classifier workflow for continuous video. A lightweight detector runs continuously, activates the classifier selectively, and supports robust single-time decisions.
- Architecture: 3D CNNs model spatial and temporal video information, but their real-time use lacks a clear dynamic-system design.
- Architecture: Sliding windows feed detector and classifier queues, with detector input length n much smaller than classifier length m.
- Architecture: The workflow post-processes detector outputs and uses a single-time activation service so each performed gesture produces one activation.
- Detector: The detector distinguishes gesture from no gesture and acts as a switch that activates the classifier and supplies it with classifier-queue frames.
- Detector: Detector placement at the classifier-queue beginning enables activation when a gesture starts regardless of gesture duration.
- Detector: ResNet-10 is used as a lightweight detector with less than 1M (≈862K) parameters.
2) Classifier:
The classifier can use a complex offline 3D CNN because it is activated selectively rather than continuously. The experiments use C3D and ResNeXt-101, with pretraining and augmentation supporting training and evaluation.
- 2) Classifier:: The classifier architecture is independent of model type, and the experiments use C3D and ResNeXt-101 for classification.
- 2) Classifier:: ResNeXt-101 follows the specified ResNeXt-block configuration, while C3D modifies the final fully connected layers from 4096 to 2048 nodes.
- 2) Classifier:: Classifier architectures are pretrained on Jester and then fine-tuned on EgoGesture and nvGesture to reduce overfitting and training duration.
- 2) Classifier:: Training uses SGD with Nesterov momentum, weight decay, dropout in C3D, and data augmentation.
- 2) Classifier:: Augmentation includes random spatial crops and scales, elastic displacement, and temporal sampling of consecutive frames.
- 2) Classifier:: Testing scales and center-crops images to 112 × 112 before normalization.
3) Post-processing:
Post-processing filters detector probabilities and weights class scores to support robust, single-time gesture activation. The weighting emphasizes the discriminative nucleus while reducing early ambiguity.
- Median filtering is selected from average, exponentially-weighted average, and median strategies because it achieves slightly better results.
- Single-time activation uses early detection before gesture end or late detection when the detector deactivates the classifier.
- Weighted averaging assigns smaller influence to preparation frames and emphasizes the gesture nucleus to resolve early classification ambiguity.The weighting strategy is based on a sigmoid-like function and is applied to class scores during an active state.
- The confidence measure for early detection is the difference between the weighted average scores of the classes.
- The activation procedure processes incoming frame windows, updates class probabilities, returns a gesture on detection, and can return a late prediction after the gesture ends.
5) Evaluation of the Activations:
Online evaluation must account for classification errors, missed detections, and repeated detections rather than classification accuracy alone. Levenshtein distance measures these sequence-level changes and is converted into Levenshtein accuracy.
- Online evaluation considers classifier misclassification, detector misses, and multiple detections within one gesture.
- Levenshtein distance counts insertions, deletions, and substitutions needed to transform predicted gesture sequences into ground-truth sequences.
- For the example sequence, a distance of 2 over 9 target classes produces a Levenshtein accuracy of 77.78%.
- The detector evaluation includes binary classification accuracy and detection results on the EgoGesture test set.
IV. EXPERIMENTS
The proposed approach is evaluated on the EgoGesture and NVIDIA Dynamic Hand Gestures datasets.
- The approach is tested on the EgoGesture and NVIDIA Dynamic Hand Gestures datasets.
A. Offline Results Using EgoGesture Dataset
Experiments use EgoGesture and compare architectures, input durations, modalities, and an additional NVIDIA benchmark reference. Performance improves with longer inputs, while depth generally outperforms RGB.
- EgoGesture contains 83 static and dynamic gesture classes across six indoor and outdoor scenes, with subject-disjoint train, validation, and test splits.The splits contain 1239, 411, and 431 videos, respectively.
- ResNeXt-101 with 32-frame input achieves the best offline classification performance among the compared architectures.
- Increasing input size improves gesture detection and classification performance across modalities, depending partly on average gesture duration.
- Depth-modality models consistently outperform RGB models, while depth suppresses background motion and focuses on hand motion.
- The experiments also reference state-of-the-art comparison and classifier accuracy tables for the NVIDIA Dynamic Hand Gesture dataset.
B. Offline Results Using nvGesture Dataset
On nvGesture, ResNeXt-101 achieved the best offline classification performance, while larger inputs and depth modality improved classification and detection. An 8-frame ResNet-10 depth detector was selected for online testing.
- Offline classification: ResNeXt-101 architecture achieves the best offline classification performance on nvGesture.The architecture was compared with state-of-the-art models.
- Input size: Increasing input size improves classification and detection performance across all modalities on nvGesture.The reported relationship depends highly on dataset characteristics, especially average gesture duration.
- Modality comparison: Depth modality outperforms RGB modality for all tested input sizes on nvGesture.
- Online detector: An 8-frame ResNet-10 detector using depth modality was selected for online testing.Its detailed online results are reported in Table IX.
C. Real-Time Classification Results
The real-time system achieved strong single-time activation accuracy while balancing detection speed and accuracy through threshold selection. Its two-model design also delivered high execution rates on a single GPU.
- Real-time accuracy: 91.04% and 77.39% Levenshtein accuracies were achieved on EgoGesture and nvGesture, respectively.The evaluation used 431 and 482 test videos, respectively.
- Early detection: Increasing the early-detection threshold reduces average detection time while improving Levenshtein accuracy.Figure 6 compares weighted and uniform averaging for both datasets.
- Early detection: At τearly=0.4 on EgoGesture, activations occurred 9 frames earlier on average with a 1.71% Levenshtein-accuracy reduction.
- Averaging strategy: Weighted averaging performs considerably better than uniform averaging for both EgoGesture and nvGesture.
- Execution performance: The system runs at 460 fps without gestures and at 62 fps with gestures when ResNeXt-101 is the classifier.With C3D as classifier, the reported gesture-present rate is 41 fps; measurements used one NVIDIA Titan Xp GPU and batch size 8.
V. CONCLUSION
The paper concludes that its hierarchical two-model architecture supports resource-efficient real-time hand gesture recognition with early and single-time activations. It evaluates the approach on two datasets and identifies confidence estimation and weighting as areas for further work.
- Conclusion: The two-model hierarchical architecture provides resource efficiency, early detections, and single-time activations for real-time gesture recognition.
- Evaluation: Levenshtein accuracy measures misclassifications, multiple detections, and missing detections simultaneously in real-time evaluation.
- Temporal weighting: Weighted averaging improves overall performance while allowing early gesture detection.
- Confidence measure: Single-time activation confidence is based on the difference between the two highest average class probabilities.
- Future work: The authors plan to investigate statistical hypothesis testing for confidence measurement and different weighting approaches.