Source-linked AI summary
Chained Multi-stream Networks Exploiting Pose, Motion, and Appearance for Action Classification and Detection
Mohammadreza Zolfaghari, Gabriel L. Oliveira, Nima Sedaghat, Thomas Brox
TL;DR
Action recognition must combine pose, motion, appearance, and temporal context. The paper introduces a three-stream network with Markov-chain fusion that sequentially refines labels, achieving state-of-the-art classification and spatio-temporal localization across several benchmarks.
Problem
Action recognition involves varied actions and multiple visual cues, while motion cues are not directly inferred reliably from raw images alone.
Method
A three-stream spatio-temporal architecture integrates RGB, optical flow, and pose through a Markov chain that sequentially refines action predictions.
Results
The approach achieves state-of-the-art action classification on UCF101, HMDB51, J-HMDB, and NTU RGB+D, and state-of-the-art spatial and spatio-temporal detection results.
Takeaways & Limitations
Sequential fusion improves over alternative fusion strategies while the pose network provides person localization for action detection without additional computational cost.
Takeaways & Limitations
The fusion model assumes that class predictions are conditionally independent across input modalities.
Abstract
from arXiv · showhide
General human action recognition requires understanding of various visual cues. In this paper, we propose a network architecture that computes and integrates the most important visual cues for action recognition: pose, motion, and the raw images. For the integration, we introduce a Markov chain model which adds cues successively. The resulting approach is efficient and applicable to action classification as well as to spatial and temporal action localization. The two contributions clearly improve the performance over respective baselines. The overall approach achieves state-of-the-art action classification performance on HMDB51, J-HMDB and NTU RGB+D datasets. Moreover, it yields state-of-the-art spatio-temporal action localization results on UCF101 and J-HMDB.
1. Introduction
The paper addresses action recognition by combining pose, motion, and appearance cues in a chained multi-stream network. A Markov-chain fusion strategy sequentially refines action labels and supports classification and spatial-temporal localization.
- Action recognition requires modeling multiple visual cues, including pose, motion, objects, and temporal context.
- Multi-stream architectures became popular because separate raw-image and optical-flow networks capture complementary information that a single network may not directly infer from raw images.
- The proposed architecture adds a pose stream to raw-image and optical-flow streams using a fast convolutional network and spatio-temporal convolutions.The pose stream captures body-part positions and their temporal dynamics, while also providing person localization.
- Markov-chain fusion sequentially refines action labels by integrating modalities successively rather than combining independently or by simple score fusion.The chain is reported to improve over independent training and alternative fusion strategies while providing implicit regularization against over-fitting.
- The network analyzes videos at multiple temporal scales while retaining frame-level actionness scores for temporal localization.This design supports temporal and spatio-temporal action detection in addition to classification.
2. Related work
Related work progresses from hand-crafted global video descriptors to learned spatio-temporal CNNs and multi-modal fusion. Prior studies also establish pose and temporal body-part dynamics as useful action-recognition signals.
- Feature based approaches: Traditional approaches encode hand-crafted discriminative features with bag-of-words or Fisher vectors, but often miss context and high-level information.
- CNN based approaches: Deep networks replace hand-crafted features with end-to-end learned representations that capture localized features, context, and spatio-temporal information.
- Fusion of multiple modalities: Multi-stream methods combine complementary appearance and motion cues using separate streams, with later work exploring fusion position, gating, and adaptive regularization.
- Pose feature based methods: Pose-based methods use body-part temporal dynamics or pose-guided features for action recognition, including high-level features extracted from appearance and optical flow.
3. Inputs to the Network
The network uses RGB images, optical flow, and human body-part segmentation as spatio-temporal inputs. Optical flow is computed with a fast variational method, while Fast-Net supplies pose information through body-part segmentation.
- The model receives raw RGB images, optical flow, and human pose represented as body-part segmentation across multiple frames.
- 3.1. Optical Flow: Optical flow is computed with a reliable variational method and converted into a three-channel RGB representation using its components and magnitude.Values are scaled by 16 and quantized to the [0,255] interval.
- 3.2. Body Part Segmentation: Figure 2 distinguishes convolutions, pooling, dropout, up-convolutional layers, and softmax by color.
- 3.2. Body Part Segmentation: Fast-Net is an encoder-decoder network that provides body-pose information through human body-part segmentation.Its encoder is initialized with VGG, and skip connections preserve output details at the original input resolution.
- 3.2. Body Part Segmentation: Body-part masks are constructed across J-HMDB and MPII using joint locations, including torso polygons and ellipse approximations for other parts.
- 3.2. Body Part Segmentation: The pose estimator runs at 33 fps with 150×150 resolution and handles images containing multiple people.The authors report good accuracy on body-part segmentation for J-HMDB and MPII examples.
4. Action Recognition Network
The action-recognition network processes pose, optical flow, and RGB inputs in a sequential Markov-chain fusion architecture, refining predictions across streams. It supports classification and temporal localization while using multi-granular temporal inputs for longer actions.
- Multi-stream fusion: Each input cue is processed by a separate convolutional stream, beginning with pose and refining action evidence through optical flow and RGB.The proposed fusion is sequential rather than an independent score or feature combination.
- Multi-stream fusion: The Markov-chain model conditions each stream’s prediction on the new modality and all previous predictions, producing sequentially refined action labels.The architecture jointly trains streams so later streams learn complementary features, with separate losses providing additional regularization.
- Multi-stream fusion: Pose-to-RGB ordering performed best among the tested stream orderings.The paper reports that the sequence starts with pose and ends with the RGB image.
- Network configuration: Each stream uses a C3D-based 3DCNN, takes 16 frames, and connects to the next stream through layer FC6.The base architecture has 17.5M parameters, eight 3D convolution layers, five 3D pooling layers, and two fully connected layers.
- Temporal processing: For classification, scores are averaged across temporal windows and ten crops per clip, with an optional multi-granular ensemble spanning three temporal resolutions.The multi-granular setup uses consecutive frames, subsampled frames from a 32-frame window, and randomly sampled frames from the video.
- Temporal processing: Temporal action localization thresholds per-frame scores, while the pose network prevents detecting an action when no human is detected.The multi-granular approach is not applicable to temporal detection.
5. Experiments
Experiments show that sequentially combining pose, optical flow, and RGB improves action classification across datasets, while temporal multi-granularity further increases performance. The architecture also achieves strong classification and action-detection results while retaining efficient temporal and spatial localization.
- Robustness analysis: Estimated optical flow and body-part inputs nearly reach the accuracy obtained with their ground-truth counterparts, indicating limited sensitivity to imperfect estimates.On J-HMDB, the difference between recognition with estimated and ground-truth inputs is reported as rather small, independently of fusion method.
- Action classification: The sequential Markov-chain fusion consistently outperforms jointly trained feature-concatenation fusion across UCF101, HMDB51, and J-HMDB.The baseline concatenates modalities and feeds them through fully connected layers.
- Action classification: Adding pose substantially improves the two-stream version, confirming pose as a complementary modality for action recognition.The Markov-chain fusion is additionally reported to provide a large margin.
- Temporal analysis: Temporal multi-granularity fusion further improves results, with a particularly large benefit on HMDB51.Larger temporal windows also consistently improve classification accuracy across datasets.
- State-of-the-art comparison: The proposed system outperforms state-of-the-art methods on J-HMDB, NTU, and HMDB51, and is on par with state of the art on UCF101 without hand-crafted features.A two-stream RGB+OF variant using TSN reaches 94.05% accuracy on UCF101, but TSN does not support action detection.
- Action detection: The method achieves state-of-the-art spatial and spatio-temporal detection on J-HMDB and UCF101, with spatial detection at 31 fps and spatio-temporal detection at 10 fps.Its pose network provides one detection box per person, and chained three-modality fusion yields accurate per-frame scores.
6. Conclusion
The proposed architecture sequentially integrates multiple visual cues through a Markov chain, achieving state-of-the-art results in action classification and spatial and spatio-temporal detection.
- The architecture integrates multiple cues sequentially via a Markov chain, outperforming alternative fusion methods by modeling cue dependencies while reducing over-fitting.
- The approach achieves state-of-the-art performance on UCF101, HMDB51, J-HMDB, and NTU RGB+D for action classification without additional hand-crafted features.
- The method also generalizes to spatial and spatio-temporal action detection, where it obtains state-of-the-art results.