PaperlayerView exact revision on arXiv

Source-linked AI summary

Slow Feature Analysis for Human Action Recognition

Zhang Zhang, Dacheng Tao

arXiv:1907.06670v1cs.CV

TL;DR

Human action recognition needs features that capture slowly changing action concepts despite rapidly varying image signals. The paper introduces discriminative SFA variants and the ASD representation for local cuboids, then evaluates them across multiple action databases. The experiments report strong performance, including 97.33 percent for D-SFA + ASD versus 73 percent for a bag-of-words model on complex multiperson activities.

  • Problem

    Human action recognition requires features that relate rapidly varying image sequences to slowly varying high-level action concepts.

  • Method

    The paper learns four SFA variants from sampled motion-boundary cuboids and represents transformed action sequences with accumulated squared temporal derivatives.

  • Results

    97.33 percent average performance for D-SFA + ASD compared with 73 percent for the bag-of-words model on complex multiperson activities.

  • Takeaways & Limitations

    SFA-based features, including discriminative learning and ASD encoding, are effective for recognizing actions across the evaluated databases.

  • Takeaways & Limitations

    The number of slow feature functions is selected empirically, and simple squared-derivative accumulation does not recover complex spatiotemporal relationships among cuboids.

Abstract

from arXiv · show

Slow Feature Analysis (SFA) extracts slowly varying features from a quickly varying input signal. It has been successfully applied to modeling the visual receptive fields of the cortical neurons. Sufficient experimental results in neuroscience suggest that the temporal slowness principle is a general learning principle in visual perception. In this paper, we introduce the SFA framework to the problem of human action recognition by incorporating the discriminative information with SFA learning and considering the spatial relationship of body parts. In particular, we consider four kinds of SFA learning strategies, including the original unsupervised SFA (U-SFA), the supervised SFA (S-SFA), the discriminative SFA (D-SFA), and the spatial discriminative SFA (SD-SFA), to extract slow feature functions from a large amount of training cuboids which are obtained by random sampling in motion boundaries. Afterward, to represent action sequences, the squared first order temporal derivatives are accumulated over all transformed cuboids into one feature vector, which is termed the Accumulated Squared Derivative (ASD) feature. The ASD feature encodes the statistical distribution of slow features in an action sequence. Finally, a linear support vector machine (SVM) is trained to classify actions represented by ASD features. We conduct extensive experiments, including two sets of control experiments, two sets of large scale experiments on the KTH and Weizmann databases, and two sets of experiments on the CASIA and UT-interaction databases, to demonstrate the effectiveness of SFA for human action recognition.

1 INTRODUCTION

The paper introduces SFA-based features for human action recognition by exploiting the contrast between rapidly changing image signals and slowly changing action concepts. It addresses global modeling difficulty with local cuboids, learned slow feature functions, and statistical feature encoding.

  • Motivation and Overview: Temporal slowness motivates features that map quickly varying image sequences to slowly varying high-level action concepts.The paper illustrates this contrast using pixel values and action-category labels over time.
  • Motivation and Overview: A single global function cannot model entire action sequences because high-dimensional image space contains complex variations.
  • Motivation and Overview: The proposed system randomly samples local cuboids in motion boundaries, learns slow feature functions, transforms test cuboids, and computes a statistical feature vector.The final classification stage uses a support vector machine.
  • Purpose and Contributions: The paper extends unsupervised SFA with supervised, discriminative, and spatial discriminative learning strategies to encode action-selective information.
  • Purpose and Contributions: The Accumulated Squared Derivative feature represents an action sequence through the statistical distribution of its slow features.

2 RELATED WORK

Related work spans holistic features, local descriptors, and biologically inspired representations for action recognition. The paper situates SFA within biologically motivated approaches while using a local cuboid-based pipeline.

  • Feature-based Methods: Human action-recognition features are organized into holistic, local-descriptor, and biologically inspired approaches.
  • Holistic Features: Holistic methods use properties such as body shape, silhouettes, joint trajectories, and motion templates, but require accurate segmentation and tracking.They are sensitive to background motion noise and tracking errors.
  • Local Descriptors: Local-descriptor methods reduce effects of background noise and tracking errors through interest-point detection, descriptor construction, codebook quantization, and bag-of-words representation.
  • Local Descriptors: Local methods also model spatiotemporal relationships using correlograms or relationship matching for multiperson activities.
  • Biologically Inspired Methods: Biologically inspired systems use motion-direction-sensitive filters and position-invariant spatiotemporal feature detectors motivated by visual-cortex models.

3 SLOW FEATURE ANALYSIS

SFA learns output functions whose temporal derivatives vary as slowly as possible under normalization and decorrelation constraints. Linear and nonlinear implementations reduce this objective to generalized eigenanalysis in an expanded feature space.

  • Motivation: Unlike sparseness and nonnegativity principles, temporal slowness explicitly models temporal information in image sequences.
  • SFA Definition: SFA transforms an input signal into output components whose mean squared first-order temporal derivatives are minimized.The output is constrained to be zero-mean and unit-variance.
  • SFA Definition: The decorrelation constraint makes different output components carry different information and orders them from slowest to faster features.
  • Linear SFA: For linear transformations, SFA becomes a generalized eigenvalue problem based on temporal-derivative and input covariance matrices.The slowest feature corresponds to the smallest generalized eigenvalue.
  • Nonlinear SFA: Nonlinear SFA applies a nonlinear expansion and then performs linear SFA in the expanded space.A quadratic expansion includes individual input terms and pairwise products.
  • Nonlinear SFA: The nonlinear procedure centers the expanded signal, solves the generalized eigenproblem, and selects eigenvectors associated with the smallest eigenvalues as slow feature functions.

4 SFA-BASED ACTION RECOGNITION

The method learns slow feature functions from locally sampled motion cuboids, extending SFA with supervised, discriminative, and spatial information. It represents action snippets by accumulating squared temporal derivatives and classifies the resulting features.

  • Collection of Training Cuboids: Training cuboids are randomly sampled from informative motion boundaries after action-sequence normalization.Cuboids are collected from foreground regions and motion boundaries before SFA learning.
  • Slow Feature Function Learning: The pipeline reformats cuboids using successive patches, applies nonlinear expansion after PCA reduction, and learns slow feature functions.The reformed sequence has time length d − τ_t + 1, while PCA reduces the input dimensionality to 50 before nonlinear expansion.
  • Slow Feature Function Learning: Four strategies are evaluated: U-SFA, S-SFA, D-SFA, and SD-SFA, with the latter methods incorporating action labels or spatial information.U-SFA shares functions across actions; S-SFA learns separately by category, while D-SFA targets class-specific slowdown and SD-SFA uses body-part-related spatial regions.
  • Slow Feature Function Learning: D-SFA learns class-specific function sets that slow within-class signals while encouraging signals from other classes to vary quickly.The learning objective uses a tradeoff parameter and is formulated as a generalized eigenvalue problem.
  • Action Feature Representation: The ASD feature accumulates squared first-order derivatives of transformed cuboids across all slow functions and normalizes the resulting vector with L1 normalization.For SD-SFA, regional feature vectors are concatenated before normalization, retaining information about cuboid counts across regions.

5 DATA SETS

The experiments use four action databases spanning single-person actions and multiperson interactions. KTH and Weizmann provide simple individual actions, while CASIA and UT-Interaction test interaction recognition under varied conditions.

  • Overview: Four databases are used: KTH, Weizmann, CASIA, and UT-Interaction.They cover both single-person action recognition and multiperson interaction recognition.
  • KTH: KTH contains 598 sequences of six single-person actions performed by 25 people across four scenarios.The scenarios include outdoor, scale variation, clothing variation, and lighting variation.
  • Weizmann: Weizmann contains 90 sequences covering 10 single-person action types performed by nine subjects.The actions include bending, jumping, running, walking, and waving variants.
  • CASIA: CASIA includes single-person actions and seven two-person interaction types; experiments select meet, fight, and rob.The selected segments contain two people approaching closely, requiring subtle visual features for discrimination.
  • UT-Interaction: UT-Interaction contains six interaction classes and 120 video segments divided between a quieter parking lot and a noisier windy lawn.The two sets each contain 60 segments, with 10 segments per class.

6 EXPERIMENTAL RESULTS

Experiments across multiple datasets and representations show that SFA-based features support effective human-action recognition. Discriminative and spatially informed variants improve selectivity and benchmark performance, while ASD features and local representations provide useful action descriptors.

  • Local representations generally outperform holistic representations, although SFA is more effective than PCA within the holistic representation.
  • The ASD feature outperforms cuboid classification because individual cuboids provide weak supervision and different actions can share similar cuboids.
  • SFA slows transformed input signals, and learned features encode action-dependent motion patterns, including similarities among jogging, running, and walking.
  • D-SFA yields stronger action selectivity than S-SFA by incorporating interclass information into slow feature learning.S-SFA and D-SFA favor intraclass cuboids, but D-SFA also makes other-action cuboids vary quickly.
  • SD-SFA achieves the highest recognition accuracy among the SFA methods under the RSD testing strategy.

7 CONCLUSION

The paper presents four SFA-based methods and an ASD feature for human action recognition, with experiments indicating effective motion-pattern extraction and competitiveness across several datasets. It also identifies empirical feature-count selection and limited modeling of cuboid relationships as open issues.

  • Four SFA-based methods extend unsupervised SFA with learning strategies that encode discriminative information for action recognition.
  • The Accumulated Squared Derivative feature characterizes the statistical distribution of slow features in an action sequence.
  • Control experiments suggest that SFA extracts effective motion patterns and benefits action-recognition performance.
  • Experiments on KTH and Weizmann demonstrate competitiveness with state-of-the-art methods, while multiperson datasets show potential for recognizing complex activities.
  • The slowest functions and their number are selected empirically, and future work aims to determine the number automatically.
  • Simple accumulation of squared derivatives does not recover complex spatiotemporal relationships among cuboids, motivating future strategies linking slow features with hidden patterns.
Loading 1907.06670v1…