Source-linked AI summary
Scene Flow to Action Map: A New Representation for RGB-D based Action Recognition with Convolutional Neural Networks
Pichao Wang, Wanqing Li, Zhimin Gao, Yuyao Zhang, Chang Tang, Philip Ogunbona
TL;DR
Scene flow had not been studied for action recognition with ConvNets, and earlier RGB-D methods extracted features from the two modalities separately. The paper uses scene flow to build SFAM representations and learns channel transforms for ConvNets; it reports state-of-the-art results on two relatively large datasets.
Problem
Scene flow's use for action recognition, particularly with ConvNets, had not been studied, while prior RGB-D methods extracted features from RGB and depth separately.
Source: §1 · p. 1Method
The method derives SFAM video maps from RGB-D scene flow, learns Channel Transform Kernels, and proposes self-calibration for unregistered RGB-D data.
Source: §1 · p. 2Results
Takeaways & Limitations
SFAM combines RGB-D scene-flow information in a representation that can be used with available pretrained ConvNets.
Source: §1 · p. 2Scene-flow computation requires spatially aligned RGB and depth, which may not be registered in some Kinect recordings.
Source: §3.2 · p. 4
Abstract
from arXiv · showhide
Scene flow describes the motion of 3D objects in real world and potentially could be the basis of a good feature for 3D action recognition. However, its use for action recognition, especially in the context of convolutional neural networks (ConvNets), has not been previously studied. In this paper, we propose the extraction and use of scene flow for action recognition from RGB-D data. Previous works have considered the depth and RGB modalities as separate channels and extract features for later fusion. We take a different approach and consider the modalities as one entity, thus allowing feature extraction for action recognition at the beginning. Two key questions about the use of scene flow for action recognition are addressed: how to organize the scene flow vectors and how to represent the long term dynamics of videos based on scene flow. In order to calculate the scene flow correctly on the available datasets, we propose an effective self-calibration method to align the RGB and depth data spatially without knowledge of the camera parameters. Based on the scene flow vectors, we propose a new representation, namely, Scene Flow to Action Map (SFAM), that describes several long term spatio-temporal dynamics for action recognition. We adopt a channel transform kernel to transform the scene flow vectors to an optimal color space analogous to RGB. This transformation takes better advantage of the trained ConvNets models over ImageNet. Experimental results indicate that this new representation can surpass the performance of state-of-the-art methods on two large public datasets.
1. Introduction
The introduction proposes using scene flow to jointly extract features from RGB-D data for 3D action recognition. It presents SFAM and learned channel transforms as ways to encode video dynamics for ConvNets.
1. Introduction
- Prior RGB-D action methods extracted features from RGB and depth separately and fused them later, while the paper proposes joint feature extraction through scene flow.Scene flow estimates actual 3D motion rather than motion projected onto the image plane.Source: §1 · pp. 1–2
- Scene-flow action recognition must organize its vectors and represent long-term dynamics; Lagrangian motion also requires reliable long-term point tracking and is prone to error.The paper adopts an Eulerian final representation to avoid long-term tracking, while using two-frame scene flow as a short Lagrangian motion representation.Source: §1 · p. 2
- SFAM encodes RGB-D video sequences from scene flow into motion maps, with variants designed to capture spatio-temporal information from different perspectives.The three scene-flow components can be treated as channels of a Scene Flow Map sequence before encoding.Source: §1 · p. 2
- Channel Transform Kernels learn relations among the three scene-flow channels and convert them into RGB-like maps for use with pretrained ConvNets.The motivation is that hand-crafted channel organization may sacrifice relations among components, while pretrained ConvNet inputs should resemble RGB images.Source: §1 · p. 2
- The paper reports state-of-the-art results on two relatively large datasets using joint RGB-D scene-flow features, SFAM variants, and Channel Transform Kernels.It also proposes self-calibration to estimate scene flow from unregistered RGB-D data.Source: §1 · p. 2
2. Related Work
Related work contrasts separate RGB-D feature extraction with joint extraction, reviews scene-flow computation, and situates the approach among video representations used by deep networks.
2.1. Feature Extraction from RGB-D Data
- Earlier RGB-D methods extracted features separately from depth and RGB before fusion; this work instead extracts features jointly from RGB-D data.The passage identifies depth-, RGB-, skeleton-based, and late-fusion approaches as prior alternatives.Source: §2.1 · pp. 2–3
2.2. Scene Flow
- Scene flow is a dense or semidense non-rigid 3D motion field, and the paper uses a primal-dual method designed for real-time dense RGB-D scene flow.The passage notes that many earlier methods took seconds to hours per frame, limiting real-application usefulness.Source: §2.2 · p. 3
2.3. Deep Leaning based Action Recognition
- Deep action-recognition approaches represent video as still frames, 3D volumes, RNN sequences, or compact images; the proposed method belongs to the compact-image category.The passage reports that this last category had achieved state-of-the-art results on multiple RGB and depth/skeleton datasets.Source: §2.3 · p. 3
3. Scene Flow to Action Map
The section outlines a pipeline that computes scene flow, constructs SFAM representations, and learns Channel Transform Kernels for an end-to-end SFAM method.
3. Scene Flow to Action Map
- The pipeline encodes RGB-D sequence dynamics using scene-flow vectors, assuming depth and RGB are prealigned or realigning them with the described self-calibration method.Scene flow is computed with the primal-dual framework introduced as PD-flow.Source: §3 · p. 3
- The section presents several hand-crafted SFAM constructions to represent scene-flow dynamics.Source: §3 · p. 3
- It also proposes learning SFAM end-to-end through Channel Transform Kernels.Source: §3 · p. 3
3.1. PD-flow
PD-flow estimates dense 3D motion from RGB and depth images by minimizing data and regularization terms over optical and range flow.
3.1. PD-flow
- PD-flow estimates the 3D motion field between consecutive instants from RGB and depth images, expressing it in the camera reference frame in meters per second.The field is represented through optical-flow components and range flow; estimating those flows simplifies implementation.Source: §3.1 · pp. 3–4
- The motion estimate minimizes a data term and a regularization term, with the data term enforcing intensity and depth matching and regularization smoothing the flow field.Source: §3.1 · p. 4
- The data term combines photometric and geometric consistency, weighting the geometric residual against brightness constancy; the regularizer is based on total variation.Source: §3.1 · p. 4
- A linearized data term and convex total-variation regularizer allow the energy to be solved with an iterative primal-dual formulation implemented in parallel on GPUs.Source: §3.1 · p. 4
3.2. Self-Calibration
The self-calibration method aligns spatially misregistered RGB and depth data without camera parameters, enabling scene-flow computation on those RGB-D videos.
3.2. Self-Calibration
- A 2D homography maps depth points into RGB coordinates to spatially align misregistered RGB-D data.The method estimates the mapping from corresponding points in an RGB frame and depth map.Source: §3.2 · p. 4
- Four pairs of corresponding points determine the homography through direct linear transformation.The homography is a 3 × 3 projective transformation.Source: §3.2 · p. 4
- RANSAC excludes unreliable matching-point outliers, while minimizing distances over more matches improves the homography estimate.The objective uses estimated matching points and a distance function; Levenberg–Marquardt is given as an example optimizer.Source: §3.2 · p. 4
3.3. Construction of Hand-crafted SFAM
SFAM represents RGB-D video as sequences of three-channel scene-flow maps, using Eulerian construction to avoid long-term point-tracking errors while retaining the benefit of matching points across two frames. Its hand-crafted variants encode motion through accumulated differences or sums, rank pooling, and transformations of the scene-flow components.
3.3.1 SFAM-D
- SFAM-D accumulates absolute differences between consecutive scene-flow maps to characterize accumulated motion-difference energy.The maps are indexed over the video sequence, whose total map count is T.Source: §3.3.1 · p. 5
3.3.2 SFAM-S
- SFAM-S accumulates sums between consecutive scene-flow maps and mainly captures large action motions after normalization.Source: §3.3.2 · p. 5
3.3.3 SFAM-RP
- Rank pooling encodes an SFM sequence as a descriptor that aggregates frame information, with forward and backward maps assigning more weight to recent frames.The ranking objective penalizes frame pairs whose scores are not separated by at least a unit margin.Source: §3.3.3 · p. 5
3.3.4 SFAM-AMRP
- SFAM-AMRP applies rank pooling to scene-flow vector amplitudes, using motion magnitude to weight frames.The amplitude map represents the relation among the three scene-flow components.Source: §3.3.4 · p. 5
3.3.5 SFAM-LABRP
- SFAM-LABRP nonlinearly transforms the three scene-flow components into another space before bidirectional rank pooling.Examples of SFAM variants for “Bounce Basketball” in the M2I Dataset show large visual differences among their action maps.Source: §3.3.5 · pp. 5–6
3.4. Constructing SFAM with Channel Transform Kernels (SFAM-CTKRP)
Channel Transform Kernels learn relationships among the three scene-flow channels, replacing hand-crafted channel relationships in SFAM variants. The learned maps are approximated with convolutional and nonlinear layers before rank pooling.
3.4. Constructing SFAM with Channel Transform Kernels (SFAM-CTKRP)
- Channel Transform Kernels learn relationships among the three scene-flow channels from data with ConvNets.The approach addresses hand-crafted relationships among the channel triple, which can vary in effectiveness across datasets or scenarios.Source: §3.4 · p. 6
- The learned maps are constructed from the original three-channel scene-flow triple.The section formulates the relationship among the triple as new learned maps.Source: §3.4 · p. 6
- Three successive convolution layers, each using nine 1×1 kernels followed by ReLU, approximate the channel transforms.The learned maps have the same size as the input map; the resulting representation is combined with RankPool temporal encoding.Source: §3.4 · p. 6
3.5. Multiply-Score Fusion for Classification
Multiply-score fusion combines the outputs of separate ConvNet channels for SFAM variants. The element-wise product determines the class prediction by its maximum score.
3.5. Multiply-Score Fusion for Classification
- Multiply-score fusion combines the score vectors from two ConvNet channels by element-wise multiplication.For SFAM-RP, the channels receive two SFAM-RP streams, one SFAM-RPf, and one SFAM-RPb.Source: §3.5 · p. 6
- The class with the maximum value in the multiplied score vector is assigned to the test sequence.The index of the maximum score corresponds to the recognized class label.Source: §3.5 · p. 6
- The multiply-score procedure can be extended to multiple channels.Source: §3.5 · p. 6
4. Experiments
Experiments evaluate SFAM on ChaLearn LAP IsoGD and M2I. SFAM outperformed the compared methods on ChaLearn despite estimated depth values, and achieved leading results in M2I single-task and cross-view settings.
4.1. Implementation Details
- Experiments use public scene-flow code, bidirectional rank pooling, and ImageNet-initialized AlexNet models.For ChaLearn, pixels beyond the final depth-histogram peak minus a fixed 0.1 tolerance are treated as background and removed from scene-flow computation.Source: §4.1 · pp. 6–7
4.2. ChaLearn LAP IsoGD Dataset
- ChaLearn LAP IsoGD contains 47,933 RGB-D gesture sequences across 249 gestures; evaluation reports validation results because the test set is unavailable for public use.The dataset does not provide true depth values, so the authors estimated them using average minimum and maximum values supplied for the CGD dataset.Source: §4.2 · p. 7
- SFAM significantly outperformed the compared methods on ChaLearn LAP IsoGD, even though its depth values were estimated rather than exact.Among hand-designed variants, SFAM-S performed best; SFAM-CTKRP performed best among variants, and multiply-score fusion combined complementary variants.Source: Table 1 · p. 7, §4.2 · p. 7
4.3. M2I Dataset
- M2I contains 1,760 samples across 22 actions, with group-based training, validation, and test splits.Its 22 actions involve person-person and person-object interactions, with front and side views.Source: §4.3 · pp. 7–8
- On M2I, SFAM-AMRP achieved the best results in both cross-view scenarios; SFAM-AMRP led the side view and SFAM-LABRP the front view in single-task evaluation.The cross-view comparison reports significant outperformance of the best baseline, especially from side-view training to front-view testing.Source: §4.3 · p. 8
5. Conclusion and Future Work
The paper presents SFAM variants for action recognition from scene flow and learns Channel Transform Kernels end-to-end with ConvNets. Experiments on two benchmark datasets demonstrate the method’s effectiveness; improving temporal encoding is left for future work.
5. Conclusion and Future Work
- The proposed scene-flow method uses SFAM variants that capture spatio-temporal information from different perspectives.Source: §5 · p. 8
- Channel Transform Kernels learn relationships among scene-flow map channels end-to-end with ConvNets.Source: §5 · p. 8
- Experiments on two benchmark datasets demonstrated the effectiveness of the proposed method, while improved temporal encoding remains future work.Source: §5 · p. 8