Source-linked AI summary
Motion Fused Frames: Data Level Fusion Strategy for Hand Gesture Recognition
Okan Köpüklü, Neslihan Köse, Gerhard Rigoll
TL;DR
Video recognition with CNNs faces the challenge of capturing temporal information, while multimodal fusion can require separate networks and lose pixel-wise correspondence. The paper proposes Motion Fused Frames, which append optical flow to static color images, and reports competitive or state-of-the-art results across three hand-gesture datasets.
Problem
Image-oriented CNNs focus on spatial analysis, while video recognition must process temporal information; feature- and decision-level fusion can require separate networks and fail to preserve pixel-wise correspondences.
Source: §1 · pp. 1–2Method
Motion Fused Frames append optical-flow frames to static color images to represent spatio-temporal states for hand-gesture recognition.
Source: §1 · p. 2Results
96.28% accuracy on Jester (2nd place), 57.4% on ChaLearn, and 84.7% on nvGesture (state of the art) are reported.
Source: §1 · p. 2Takeaways & Limitations
The authors report that adding more motion information improved performance in all tested cases, with the first appended optical-flow frame giving an especially significant increase.
Source: §6 · p. 8The authors leave evaluation on different modalities and more challenging video-understanding tasks, and better ways to exploit data-level fusion on CNNs, for future work.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Acquiring spatio-temporal states of an action is the most crucial step for action classification. In this paper, we propose a data level fusion strategy, Motion Fused Frames (MFFs), designed to fuse motion information into static images as better representatives of spatio-temporal states of an action. MFFs can be used as input to any deep learning architecture with very little modification on the network. We evaluate MFFs on hand gesture recognition tasks using three video datasets - Jester, ChaLearn LAP IsoGD and NVIDIA Dynamic Hand Gesture Datasets - which require capturing long-term temporal relations of hand movements. Our approach obtains very competitive performance on Jester and ChaLearn benchmarks with the classification accuracies of 96.28% and 57.4%, respectively, while achieving state-of-the-art performance with 84.7% accuracy on NVIDIA benchmark.
1. Introduction
The paper addresses the difficulty of capturing temporal information with image-oriented CNNs and the trade-offs of common fusion strategies. It introduces Motion Fused Frames (MFFs) and evaluates them on three hand-gesture benchmarks.
1. Introduction
- Image-oriented CNNs analyze spatial content, while video recognition must handle temporal data and its greater processing load and model complexity.The paper notes that these factors make recognition from video more challenging.Source: §1 · p. 1
- Feature- and decision-level fusion can require separate networks, lose pixel-wise correspondence, and need complex network modifications; data-level fusion offers a single-network alternative with automatic correspondence and little modification.The paper also notes that data-level fusion requires frame registration, which can be difficult when modalities come from different hardware.Source: §1 · p. 2
- MFFs append optical-flow frames to static color images, adding motion information to their spatial content.For example, flow reveals a left-to-right hand movement that is not apparent from the static image alone.Source: §1 · p. 2
- 96.28% accuracy on Jester (2nd place), 57.4% on ChaLearn, and 84.7% on nvGesture (state of the art) are the reported results.The three datasets are Jester, ChaLearn LAP IsoGD, and NVIDIA Dynamic Hand Gesture (nvGesture).Source: §1 · p. 2
2. Related Work
Related work covers CNN strategies for extracting temporal information from videos and fusion strategies for combining modalities. The paper applies data-level fusion through MFFs and uses a TSN-based approach as its building block.
2. Related Work
- Video CNN approaches include multi-frame 2D CNN inputs, 3D CNNs, TSN, multi-scale temporal relations, and CNN features with recurrent temporal modeling.The cited examples include LSTM and recurrent 3D CNN approaches for hand gestures.Source: §2 · p. 2
- Deep-learning fusion strategies combine modalities at the data, feature, or decision level, with decision-level examples including score averaging, concatenation, and consensus voting.Feature-level approaches can fuse CNN features at different layers or use canonical correlation analysis.Source: §2 · pp. 2–3
- The paper applies MFF data-level fusion to avoid separate modality networks and establish pixel-wise correspondence, using segmented TSN-style clips to represent and fuse action states.The approach may sample fewer than all frames in a segment, which the authors describe as critical for real-time applications.Source: §2 · p. 3
3. Approach
The approach fuses optical-flow motion with RGB appearance in Motion Fused Frames, then classifies video by extracting and combining segment features. It uses a TSN-based CNN pipeline with training choices tailored to the evaluated datasets.
3.1. Motion Fused Frames
- MFFs append optical-flow frames from consecutive previous frames to RGB images, preserving spatial content while adding motion cues with automatic pixel-wise correspondence.The paper notes that flow is computed from RGB, so the approach uses RGB image modality rather than requiring additional depth or infrared sensors.Source: §3.1 · p. 3
3.2. Network Architecture
- The network divides a video into N segments, forms an MFF for each, extracts segment features with a CNN, concatenates them in order, and predicts classes through fully connected layers and softmax.The illustrated 3-flow, 1-color configuration uses nine channels per MFF.Source: Figure 2 · p. 3, §3.2 · p. 4
3.3. Training Details
- The feature extractor is ImageNet-pretrained BN-Inception; Jester adapts its first-layer weights for flow channels, while ChaLearn and nvGesture start from Jester-pretrained models.The training setup also uses partial batch-normalization and dropout.Source: §3.3 · p. 4
- Training uses SGD with categorical cross-entropy, weight decay, and dataset-specific learning-rate schedules; regularization and augmentation are also applied.ChaLearn and nvGesture use an additional dropout layer, and augmentation includes spatial transformations, with horizontal flipping for ChaLearn and temporal scaling and jittering for nvGesture.Source: §3.3 · pp. 4–5
4. Experiments
The experiments test MFF variants on Jester, ChaLearn, and nvGesture, spanning frame and segment ablations and comparisons with existing systems.
4.1. Results Using Jester Dataset
- 8-MFFs-3f1c with 5-crop augmentation was the best tested Jester model, and its official test-set submission ranked second at submission time.In single-segment tests, accuracy rose from 63.60% to 82.93% as appended flow frames increased from zero to 11; the first flow frame added around 9% accuracy.Source: Table 2 · p. 6, §4.1 · p. 6
4.2. Results Using ChaLearn Dataset
- 15.6% validation and 13.9% test accuracy gains accompanied increasing optical-flow frame counts on ChaLearn; its 10-fps videos give MFFs a larger time span.Using flow frames from earlier timestamps was left for future work.Source: Table 3 · p. 6, Table 4 · p. 6, §4.2 · p. 7
- Three modalities produced the best reported ChaLearn results, while 8-MFFs-3f1c outperformed most listed methods without depth.Table 5 compares methods by validation accuracy.Source: Table 5 · p. 6, §4.2 · p. 7
4. Experiments
- Evaluation used center crops with equidistant frames; for weakly segmented nvGesture test videos, the first and last 10 frames were cropped and the center 60 evaluated.Source: §4 · p. 5, §4.3 · p. 7
4.3. Results Using nvGesture Dataset
- 14.4% and 19.1% accuracy gains over C3D and two-stream CNNs, respectively, accompany state-of-the-art nvGesture performance using color and optical flow.Table 6 compares methods on nvGesture.Source: Table 6 · p. 7, §4.3 · p. 7
- Human accuracy on nvGesture test videos was 88.4% for the color modality.Source: §4.3 · p. 7
5. Discussion
MFFs fuse color and optical flow, with the approach also applicable to depth and infrared modalities. The discussion identifies temporal frame selection and classifier architecture as directions for further investigation.
5. Discussion
- MFFs fuse color and optical flow, and the approach can also be applied to depth and infrared modalities.Depth and infrared require additional hardware but not the extra computation associated with optical flow.Source: §5 · pp. 7–8
- Using optical-flow frames from temporally different positions relative to the selected RGB image could further increase performance.The current construction appends flow frames calculated from consecutive previous frames.Source: §5 · p. 8
- MFFs could be combined with a multiple-time-scale approach to capture temporal relations and potentially improve performance.The cited approach is described as capturing temporal relations at multiple time scales.Source: §5 · p. 8
- The experiments use MLPs for conditional-class scores after feature extraction, while RNNs are proposed as an alternative for future analysis.The authors say different architectures like RNNs can perform better.Source: §5 · p. 8
6. Conclusion
The paper presents Motion Fused Frames, which fuse optical-flow motion information into RGB images for hand gesture recognition. Across the evaluated datasets, adding more motion information improves performance, and the authors propose evaluating the approach on other modalities and more challenging tasks.
6. Conclusion
- Motion Fused Frames fuse optical-flow frames into RGB images to represent action states for hand gesture recognition.Source: §6 · p. 8
- Fusion of more motion information improves performance in all evaluated cases, with the increase at the first appended optical-flow frame especially significant.The evaluation used only optical-flow and color modalities and obtained competitive results on several recent datasets.Source: §6 · p. 8
- Future work will examine other modalities and more challenging tasks requiring human understanding in videos.The authors also intend to find better ways to exploit data-level fusion on CNNs for video analysis.Source: §6 · p. 8