Source-linked AI summary
Structure Inference Machines: Recurrent Neural Networks for Analyzing Relations in Group Activity Recognition
Zhiwei Deng, Arash Vahdat, Hexiang Hu, Greg Mori
TL;DR
Higher-level visual recognition requires reasoning about relations among image entities, but bridging low-level classifications to compositional scenes remains challenging. The paper combines deep networks with recurrent structure inference and gated edge learning, improving group activity recognition accuracy through iterative inference and structure learning.
Problem
Relations among image entities support higher-level visual understanding, yet bridging low-level classifier outputs to compositional scene interpretation remains challenging.
Method
The model uses a recurrent neural network to pass contextual messages between person and scene classifications while trainable gates learn which connections to use.
Results
The approach improves group activity recognition accuracy over rounds of inference, with structure gates adaptively capturing useful connections.
Takeaways & Limitations
Joint deep classification, recurrent inference, and learned structure enable effective classification of high-level concepts in group activity recognition.
Takeaways & Limitations
Evaluation uses a new 2241-frame training and 1106-frame testing split instead of leave-one-out testing because leave-one-out is computationally intensive.
Abstract
from arXiv · showhide
Rich semantic relations are important in a variety of visual recognition problems. As a concrete example, group activity recognition involves the interactions and relative spatial relations of a set of people in a scene. State of the art recognition methods center on deep learning approaches for training highly effective, complex classifiers for interpreting images. However, bridging the relatively low-level concepts output by these methods to interpret higher-level compositional scenes remains a challenge. Graphical models are a standard tool for this task. In this paper, we propose a method to integrate graphical models and deep neural networks into a joint framework. Instead of using a traditional inference method, we use a sequential inference modeled by a recurrent neural network. Beyond this, the appropriate structure for inference can be learned by imposing gates on edges between nodes. Empirical results on group activity recognition demonstrate the potential of this model to handle highly structured learning tasks.
1. Introduction
The paper targets higher-level visual understanding by combining deep image analysis with relational reasoning. It proposes an end-to-end model that bridges low-level classifications and compositional group activities through message passing and learned structure.
- Relations such as spatial distance, relative motion, and concurrent actions support recognition of higher-level visual activities.
- The method extends deep networks with recurrent inference over lower-level outputs to learn structures effective for higher-level tasks.
- Group activity analysis requires reasoning over individual people, their varied actions, and their interactions within a scene.
- Deep networks are effective for low-level classification but typically produce flat outputs rather than compositional scene interpretations.
- The proposed end-to-end network classifies low-level inputs, refines them through messages, learns connections with gates, and classifies high-level concepts.
2. Previous Work
Previous work combines graphical models, deep networks, and message passing for structured visual recognition. Group activity recognition commonly models individual actions together with interactions and scene-level structure.
- Structured deep-learning research has integrated neural networks with CRFs, HEX graphs, message passing, and feedforward approximations to graphical inference.
- Graphical-model approaches to group activity recognition include hierarchical models, subgroup discovery, social-role interactions, gradient boosting, and joint recognition-detection.
- The paper frames group activity recognition as structured prediction involving both individual actions and interactions with other people.
- The RNN inference pipeline initializes messages from unary scores and iteratively computes messages using related units, inputs, and prior predictions.
3. Structure Inference Machine
The Structure Inference Machine uses recurrent message passing to refine person and scene classifications, while trainable gates learn which people should be connected. This addresses ambiguity in actions and uncertainty in social relations.
- Group activity recognition is difficult because image features are ambiguous and relations between people are uncertain.
- The paper seeks an end-to-end neural model that retains deep learning’s low-level action recognition while performing higher-level relational reasoning.
- An RNN repeatedly passes messages among person representations to refine estimates of individual actions using contextual information.
- Trainable gating functions turn connections between people on or off to learn which individuals provide relevant context for others and for group activity recognition.
- The recurrent structure passes information among individual person classifications and the scene classification for contextual refinement.
4. Group Activity Recognition with an RNN
The model refines person-action and group-activity classifications by passing contextual information through an RNN structured according to a graphical model. It uses learned connections and prediction layers to produce refined node classifications.
- CNN classifiers provide action distributions for detected persons and a whole-image distribution for the scene-level group activity.Each person classification is based on a cropped detection, while the scene classification uses the entire image.
- 4.1. Recurrent Networks for Refining Action/Activity Classification: The RNN passes information among person and scene nodes to iteratively refine classification decisions using local observations and contextual messages.Local visual inputs remain fixed while contextual information is updated across refinement iterations.
- 4.1. Recurrent Networks for Refining Action/Activity Classification: At each timestep, static image-based scores, previous scores, and hidden contextual information are combined to update the recurrent messages.The recurrent computation uses learned parameters and an element-wise nonlinear activation function.
- 4.2. Belief Propagation in an RNN: The graphical-model structure motivates connections among people, from people to the scene, and from the scene to people.These three connection types correspond to action-to-scene, scene-to-action, and person-to-person relations.
- 4.2.1 Messages in an RNN: Message units aggregate neighboring messages, static unary inputs, and previous outputs, with shared message-computation weights across timesteps.Each message roughly classifies one entity using related entities and its own local observation.
- 4.2.2 Output Prediction Layer: Prediction layers collect messages around each node to infer the scene label and person actions, while softmax losses at each timestep train the model end to end.The losses are imposed on message-unit predictions during standard mini-batch backpropagation.
5. Structure Learning for Group Activity Inference
The model learns sparse, input-adaptive graph structures for group activity inference by gating messages between person and scene nodes. It extends recurrent graphical-model inference by allowing untied message-passing weights and selectively passing information.
- Motivation: Fully connecting all people can introduce noise, so the model focuses on relevant connections that better represent group activity.Irrelevant people may create ambiguity, whereas relevant connections can improve the model.
- Instance-Level Gating: Instance-level scalar gates modify graphical-model edges between person-person and person-scene nodes instead of selecting information element-wise.The gates adapt connectivity to each input situation and enforce structural sparsity.
- Gate Computation: Gates compare each edge message with other messages, previous classification results, and the receiving node’s unary distribution to estimate whether the edge is useful.The gain for a person-person edge averages the gains of its two directed messages.
- Gated Message Passing: After gates are imposed, message units are recalculated by scaling messages between person or scene nodes with the learned gate values.The scalar-vector product selectively passes or suppresses information on each edge.
- Sparse Structure Learning: L1 regularization on gate values encourages sparse, discriminative graph structures while balancing sparsity against prediction loss.The regularization applies to scene-person, person-person, or general graph edges and is evaluated at each message-passing timestep.
- Deep Inference Machine: Untied message-computation weights extend recurrent graphical-model inference into a deep inference machine capable of learning nonlinear structures.Tied weights correspond directly to graphical-model inference, while relaxed weight sharing yields structure learning.
6. Experiments
Experiments evaluate the model on three group activity recognition datasets, including ablations, state-of-the-art comparisons, and analyses of learned structures. Structure inference improves person-level action classification across the reported datasets, with gains of approximately 6%, 10%, and 4% under different settings.
- Datasets: The framework is evaluated on the Collective Activity, Collective Activity Extended, and Nursing Home datasets.The first two are standard group activity recognition benchmarks; the Nursing Home dataset contains cluttered scenes and large intra-class variation.
- Collective Activity Dataset: The Collective Activity experiments compare four model variants with standard baselines in an ablation study.The study examines the effects of different model components, including structure gates.
- Collective Activity Dataset: ≈6% improvement in person-level action classification follows three steps of gated message passing on the Collective Activity Dataset.The reported improvement concerns node accuracy over the whole graph.
- Collective Activity Dataset: The model outperforms other deep learning and structure learning models on the Collective Activity Dataset, although [17] achieves better results using a cardinality kernel.That kernel directly mimics the dataset’s majority-action scene-label definition.
- Collective Activity Extended Dataset: ≈10% improvement in person-level action classification is obtained through structure inference on the Collective Activity Extended Dataset.The experiments use a train-test split of 2241 training frames and 1106 testing frames after removing walking and adding Jogging and Dancing.
- Nursing Home Dataset: ≈4% improvement in person-level action classification occurs after the second iteration on the Nursing Home Dataset.Accuracy is reported as superior to baselines including Deng et al.; the smaller gating gain is attributed to irrelevant actions being easier to identify in a fully connected model.
7. Conclusion
The paper presents recurrent inference with gating functions to learn structures within deep learning for group activity recognition, improving accuracy across inference rounds.
- A recurrent neural network implements inference for refining individual node estimates and determining connections between nodes.
- Gating functions learn connections between people and their relation to the overarching scene-level activity label.
- The approach improves accuracy over rounds of inference and structure learning via gating functions.