Source-linked AI summary
Beyond Temporal Pooling: Recurrence and Temporal Convolutions for Gesture Recognition in Video
Lionel Pigou, Aäron van den Oord, Sander Dieleman, Mieke Van Herreweghe, Joni Dambre
TL;DR
Gesture recognition needs temporal modeling because motion patterns distinguish gestures, whereas temporal pooling loses event order. The paper evaluates end-to-end networks with recurrence and temporal convolutions on Montalbano, achieving a Jaccard index of 0.906 with the combined architecture. The authors do not evaluate optical flow because their models are intended to learn motion features end to end.
Problem
Temporal feature pooling is limited for gesture recognition, where motion patterns are more critical than in general video classification.
Method
The paper evaluates end-to-end CNN architectures using temporal convolutions and bidirectional recurrence for frame-wise gesture recognition on the Montalbano dataset.
Results
0.906 Jaccard index is achieved by combining temporal convolutions with an LSTM-based recurrent network.
Takeaways & Limitations
Bidirectional recurrence and temporal convolutions significantly improve frame-wise gesture recognition, with recurrent models accurately predicting gesture boundaries.
Takeaways & Limitations
The authors did not experiment with optical flow because of its greater computational preprocessing complexity and their preference for end-to-end motion learning.
Abstract
from arXiv · showhide
Recent studies have demonstrated the power of recurrent neural networks for machine translation, image captioning and speech recognition. For the task of capturing temporal structure in video, however, there still remain numerous open research questions. Current research suggests using a simple temporal feature pooling strategy to take into account the temporal aspect of video. We demonstrate that this method is not sufficient for gesture recognition, where temporal information is more discriminative compared to general video classification tasks. We explore deep architectures for gesture recognition in video and propose a new end-to-end trainable neural network architecture incorporating temporal convolutions and bidirectional recurrence. Our main contributions are twofold; first, we show that recurrence is crucial for this task; second, we show that adding temporal convolutions leads to significant improvements. We evaluate the different approaches on the Montalbano gesture recognition dataset, where we achieve state-of-the-art results.
1 Introduction
Gesture recognition requires modeling temporal structure because many gestures are defined by motion patterns, while temporal pooling loses event order. The paper therefore explores temporal convolutions and recurrence in end-to-end trainable video networks.
- Gesture recognition is challenging because background, lighting, performance, clothing, and camera-position variations affect video inputs.
- Frame-wise CNN classification ignores motion, while temporal pooling aggregates features but loses the order of temporal events.
- Many gestures depend on both spatial hand or arm placement and motion patterns, making temporal information especially important.
- The paper explores temporal convolutions for learning motion features and recurrence for modeling temporal dynamics in frame-wise gesture recognition.
2 Related Work
Prior work used frame fusion, recurrent models, optical flow, and multimodal ensembles to capture video information. This paper distinguishes gesture recognition from general action recognition because gesture identity depends more critically on motion.
- On general video benchmarks, frame fusion often yields modest gains, while LSTMs provide no improvement over temporal pooling on UCF-101 and only marginal gains on Sports-1M.
- Dense optical flow represents motion through per-pixel displacement vectors but requires computational preprocessing.
- The authors omit optical flow because their end-to-end models are intended to learn motion features implicitly.
- A prior ChaLearn solution achieved state-of-the-art Montalbano performance using a multimodal ModDrop network, multiple temporal scales, and feature ensembling.
- Three-dimensional convolutions and CNN-RNN combinations had previously been studied for short-clip action recognition, temporal evolution modeling, and speech recognition.
3 Architectures
The evaluated architectures progress from single-frame and pooled CNN baselines to bidirectional recurrent and temporally convolved networks. They trade temporal-order modeling, motion-feature extraction, computational cost, and parameter efficiency in different ways.
- Single-Frame: The single-frame CNN estimates how much static image information contributes but is poorly suited to frame-wise gesture recognition.
- Temporal Feature Pooling: Temporal feature pooling collects spatial features across frames but loses the order of temporal events.
- Recurrent Networks: Bidirectional recurrence processes sequences in both temporal directions so predictions can use information from past and future frames.
- Temporal Convolutions: Temporal convolutions learn hierarchies of motion features from early layers, while factorization avoids the parameter increase of full three-dimensional convolutions.
- Temporal Convolutions: One temporal-convolution model uses three-dimensional max-pooling, whereas the other retains temporal dimensionality throughout the network.
4 Experiments
Experiments evaluate end-to-end architectures on Montalbano using the challenge’s mean Jaccard score. Temporal pooling, temporal convolutions, recurrence, and their combination progressively improve gesture-recognition performance, while predictions become better aligned with gesture boundaries.
- Dataset: The Montalbano dataset contains multimodal RGB-D videos with annotated gesture boundaries and labels, recorded under varied clothes, positions, backgrounds, and lighting.It includes 1,720,800 labeled frames across 13,858 fragments, with 11,116 training gestures and 2,742 test gestures.
- Evaluation: Performance is measured by the mean Jaccard index across 20 gesture categories and test sequences, comparing binary predictions with binary ground truth.The Jaccard index is the overlap rate between predicted and ground-truth binary vectors.
- Architecture comparison: 0.775 vs. 0.465: temporal feature pooling substantially improves the single-frame baseline, while mean-pooling outperforms max-pooling.The single-frame baseline achieves a Jaccard index below 0.5 because it extracts no motion features.
- Architecture comparison: 0.906: combining temporal convolutions with an LSTM improves performance beyond recurrent networks acting only on high-level spatial features.The recurrent models surpass the temporal-convolution CNN alone, with a Jaccard index of 0.888 vs. 0.842; the two cell types perform similarly for short-range dependencies.
- Comparison with previous work: The best models outperform previous work under RGB-D input and remain competitive without depth or pose information.Reported comparisons include 0.906 vs. 0.836 with RGB-D input, 0.842 without depth, and 0.876 vs. 0.870 without depth or pose.
- Architecture comparison: 0.842: temporal convolutions with three-dimensional max-pooling further improve the Jaccard index.Feature-map activations concentrate at movement locations, suggesting that temporal convolutions learn motion features.
5 Conclusion and Future Work
Bidirectional recurrence and temporal convolutions significantly improve frame-wise gesture recognition in video. Recurrence improves temporal boundary predictions, while temporal convolutions learn motion-feature hierarchies; the paper identifies sign language recognition as future work.
- Bidirectional recurrence and temporal convolutions significantly improve frame-wise gesture recognition in video.
- RNNs operating on high-level spatial features outperform single-frame and temporal-pooling architectures without lower-layer temporal modeling.
- Temporal convolutions learn hierarchies of motion features that recurrent networks do not.
- RNNs predict gesture beginning and ending frames accurately, whereas non-recurrent models remain uncertain at these boundaries.
- Future work targets sign language recognition, which involves larger vocabulary, subtler movements, context dependence, and simultaneous facial and manual communication.