PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchical Long Short-Term Concurrent Memory for Human Interaction Recognition

Xiangbo Shu, Jinhui Tang, Guo-Jun Qi, Wei Liu, Jian Yang

arXiv:1811.00270v1cs.CV

TL;DR

The paper addresses human interaction recognition by modeling long-term inter-related dynamics among multiple persons, which existing LSTM-based approaches do not explicitly capture. It proposes H-LSTCM, a hierarchical architecture combining Single-Person LSTMs with a Concurrent LSTM, and reports better recognition accuracy than baseline and state-of-the-art methods across evaluated datasets.

  • Problem

    Existing LSTM-based approaches model individual dynamics independently or combine persons without explicitly modeling their concurrent inter-related dynamics over time.

  • Method

    H-LSTCM first learns each person’s dynamics with a Single-Person LSTM, then uses a Concurrent LSTM to selectively integrate and store inter-related motion information.

  • Results

    H-LSTCM achieves higher recognition accuracy than baseline and state-of-the-art methods, including 88.4% versus 83.6% for Shu et al. on the Volleyball dataset.

  • Takeaways & Limitations

    Modeling inter-related motion among multiple persons supports recognition of human interactions and complex collective activities.

Abstract

from arXiv · show

In this paper, we aim to address the problem of human interaction recognition in videos by exploring the long-term inter-related dynamics among multiple persons. Recently, Long Short-Term Memory (LSTM) has become a popular choice to model individual dynamic for single-person action recognition due to its ability of capturing the temporal motion information in a range. However, existing RNN models focus only on capturing the dynamics of human interaction by simply combining all dynamics of individuals or modeling them as a whole. Such models neglect the inter-related dynamics of how human interactions change over time. To this end, we propose a novel Hierarchical Long Short-Term Concurrent Memory (H-LSTCM) to model the long-term inter-related dynamics among a group of persons for recognizing the human interactions. Specifically, we first feed each person's static features into a Single-Person LSTM to learn the single-person dynamic. Subsequently, the outputs of all Single-Person LSTM units are fed into a novel Concurrent LSTM (Co-LSTM) unit, which mainly consists of multiple sub-memory units, a new cell gate and a new co-memory cell. In a Co-LSTM unit, each sub-memory unit stores individual motion information, while this Co-LSTM unit selectively integrates and stores inter-related motion information between multiple interacting persons from multiple sub-memory units via the cell gate and co-memory cell, respectively. Extensive experiments on four public datasets validate the effectiveness of the proposed H-LSTCM by comparing against baseline and state-of-the-art methods.

I. INTRODUCTION

Human interaction recognition requires modeling long-term, inter-related dynamics among multiple persons, which existing individual-action or whole-group approaches do not fully capture. H-LSTCM addresses this gap hierarchically by learning single-person dynamics and selectively integrating their interactions with Co-LSTM.

  • Motivation: Concurrent inter-related motions among multiple persons provide discriminative information for recognizing human interactions.
  • Motivation: Existing approaches either combine individual motion descriptors or use interacting regions, respectively ignoring inter-related information or requiring difficult region localization.
  • Motivation: Existing LSTMs model human dynamics independently, while merging persons or fusing individual LSTM outputs neglects their concurrent inter-relation over time.
  • Proposed approach: H-LSTCM first learns each person’s long-term motion with a Single-Person LSTM, then feeds all outputs into a Co-LSTM for inter-related modeling.
  • Proposed approach: The Co-LSTM uses sub-memory units, cell gates, and a co-memory cell to selectively integrate inter-related motion information among multiple persons.
  • Scope: H-LSTCM is designed to recognize group activities involving at least three persons and multiple sub-group activities, extending beyond two-person interactions.

III. PRELIMINARIES: LSTM-BASED ACTION RECOGNITION

LSTM-based action recognition extends recurrent modeling by retaining and selectively updating temporal information from video features. Its hidden states represent frame dynamics, while outputs are transformed into activity-class probabilities.

  • Motivation: RNNs model video dynamics through hidden-state sequences but do not retain long-term context well because contextual information decays exponentially.
  • LSTM architecture: LSTM incorporates memory units that learn when to forget previous hidden states and when to update them using new information.
  • LSTM architecture: A traditional LSTM contains input, forget, and output gates together with a memory cell.
  • Representations: The hidden state h_t denotes the dynamic representation of the t-th video frame, and all hidden states describe the clip’s dynamics.
  • Classification: The LSTM output is transformed by softmax into probabilities for the activity classes, forming the predicted class-label vector.

A. The Architecture

H-LSTCM hierarchically models individual motion before integrating inter-related dynamics among all interacting persons. Its Co-LSTM uses person-specific sub-memory units, cell gates, and a shared co-memory cell to produce interaction representations.

  • A. The Architecture: H-LSTCM extends the preliminary two-person Co-LSTSM to group activities involving at least three persons and multiple sub-groups.The architecture is designed to model inter-related dynamics among all persons in a group rather than only selected individuals.
  • A. The Architecture: The Co-LSTM contains one sub-memory unit per interacting person, along with multiple cell gates, a common output gate, and a co-memory cell.Each sub-memory unit includes its own input gate, forget gate, and memory cell.
  • A. The Architecture: H-LSTCM first models each person's temporal motion with a Single-Person LSTM before modeling group-level inter-related dynamics.Static features from each person's tracklet are processed to obtain single-person hidden states.
  • A. The Architecture: Cell gates selectively pass inter-related motion memory from person-specific sub-memory units into the shared co-memory cell.The co-memory cell contributes to a common hidden state representing the activity scene.
  • A. The Architecture: The common hidden state is used to compute the probability vector for one human interaction.The stacked Co-LSTM units operate recurrently over time to capture concurrently inter-related dynamics.

B. Learning Algorithm

H-LSTCM is trained by comparing predicted interaction probabilities with ground-truth labels and optimizing the model parameters through backpropagation through time. The loss is applied at the final video time step for video-level labels.

  • B. Learning Algorithm: The training loss measures deviation between the ground-truth class-label vector and the predicted probability vector associated with the hidden state.The parameter set includes all H-LSTCM parameters.
  • B. Learning Algorithm: When the target class is l_t, its ground-truth vector entry is one and all other entries are zero.This reduces the loss to the negative logarithm of the predicted probability for the target class.
  • B. Learning Algorithm: Model parameters are updated using Backpropagation Through Time, with training organized into forward passes through Single-Person LSTMs and Co-LSTM followed by parameter updates.Algorithm 1 describes the training loop over epochs and video clips.
  • B. Learning Algorithm: For a video-level label, H-LSTCM minimizes the negative log probability of the target class at the final time step T.For frame-level labels, the alternative is cumulative loss over the sequence.

V. EXPERIMENTS

The experiments evaluate H-LSTCM against state-of-the-art methods and baseline models on four public datasets.

  • V. EXPERIMENTS: H-LSTCM is evaluated against baseline and state-of-the-art methods on four public datasets.The evaluation concerns human interaction recognition performance.

A. Datasets

The evaluation uses two-person interaction datasets and two multiple-person activity datasets with dataset-specific class labels and train-test protocols. The BIT dataset provides eight interaction classes under cluttered backgrounds.

  • A. Datasets: The BIT dataset contains eight interaction classes and 50 videos per class, with 34 videos per class used for training.The remaining videos are used for testing, and the videos have cluttered backgrounds.
  • A. Datasets: The UT dataset contains 60 clips across six interaction classes and uses leave-one-out cross-validation.Its ten source videos each contain the six classes.
  • A. Datasets: The Collective Activity Dataset contains 44 videos covering five multiple-person activities, using one-third of clips for testing.The remaining clips are used for training with one-versus-all recognition.
  • A. Datasets: The Volleyball Dataset contains 55 videos and 4,830 annotated frames with group-level labels for eight directional volleyball activities.Two-thirds of annotated frames are used for training and the remainder for testing.

B. Implementation Details

The experiments use tracked person bounding boxes and pretrained AlexNet fc6 features, with dataset-specific temporal settings and several CNN/LSTM baselines.

  • Person tracklets are detected and tracked across frames, and pretrained AlexNet fc6 features are extracted from each person’s bounding box.
  • The sequence length T is set to 30, 40, 10, and 10 for BIT, UT, CAD, and VD, respectively.Each Single-Person LSTM has 2048 memory-cell nodes and 1024 output nodes, while each Co-LSTM has 512 sub-memory-cell nodes.
  • B1 concatenates person-box CNN features over people and time before softmax classification, illustrating the role of deep features.
  • B2 merges interacting persons into one bounding box and processes the resulting AlexNet features with an LSTM.
  • B3 uses separate LSTMs for each person and averages their softmax scores, while B4 pools multiple Single-Person LSTM states into another LSTM.

C. Results on the BIT dataset

On BIT, H-LSTCM outperforms the evaluated baselines and state-of-the-art methods, with a reported approximately 9% gain over the strongest cited LSTM-based alternative.

  • Comparison with baselines: H-LSTCM improves recognition accuracy over all baseline methods on BIT.Adding temporal information with LSTM improves B1, while Co-LSTSM exceeds B2, B3, and B4.
  • Recognition examples: Figure 3 presents recognition-result examples from BIT, UT, and CAD using enlarged person bounding boxes to include more context.

D. Results on the UT dataset

On UT and CAD, H-LSTCM consistently outperforms the compared methods, while the CAD extension shows that static single-person features alone do not capture its more complex group dynamics.

  • Comparison with baselines: H-LSTCM performs consistently better than all baselines on UT, with H-LSTCM and Co-LSTSM achieving especially strong accuracy by modeling inter-related dynamics.
  • Results on CAD: On CAD, H-LSTCM achieves the best performance among the compared methods.
  • Results on CAD: Co-LSTSM+ does not provide significant improvements over B4 because static single-person CNN features cannot capture CAD’s complex inter-related dynamics.
  • Volleyball framework: For volleyball, two Co-LSTMs separately model the two teams before concatenation and an additional LSTM layer combine their subgroup outputs.

F. Results on the Volleyball dataset

On the Volleyball dataset, H-LSTCM models the two team subgroups separately before combining their representations, and achieves higher recognition accuracy than the compared methods.

  • Modeling team subgroups: H-LSTCM models each team subgroup with a Co-LSTM, concatenates the two outputs, and processes the resulting sequence with an LSTM layer.This design reflects stronger interactions among players on the same team than across teams.
  • State-of-the-art comparison: H-LSTCM achieves higher recognition accuracy than the state-of-the-art methods evaluated on the Volleyball dataset.The comparison includes Ibrahim et al., Shu et al., Li et al., and Biswas et al.
  • State-of-the-art comparison: 88.4% accuracy for H-LSTCM is approximately 5% higher than Shu et al.'s 83.6%.The authors connect this improvement with modeling complex collective activity among a subgroup of persons.
Loading 1811.00270v1…