Source-linked AI summary
CERN: Confidence-Energy Recurrent Network for Group Activity Recognition
Tianmin Shu, Sinisa Todorovic, Song-Chun Zhu
TL;DR
CERN addresses video recognition across individual actions, interactions, and group activities, where uncertainty can make cascaded LSTM predictions brittle. It adds an energy layer and p-value-based confidence regularization to a two-level LSTM hierarchy, enabling joint energy minimization and confidence maximization. On the Collective Activity and Volleyball datasets, CERN reports advantages over softmax and energy-based formulations and superior performance relative to state-of-the-art methods.
Problem
Video activity recognition must identify individual actions, interactions, and group events, but cascaded LSTM predictions can become brittle when training data are limited and inputs are uncertain.
Method
CERN extends a two-level LSTM hierarchy with an energy layer that regularizes energy minimization using p-values and is trained end-to-end.
Results
CERN outperforms previous feed-forward deep recurrent approaches and improves stability under uncertainty, with CERN-1 surpassing a strong softmax LSTM baseline on collective activities.
Takeaways & Limitations
Jointly selecting low-energy, high-confidence LSTM predictions provides a more reliable inference formulation for recognizing group activities, interactions, and actions.
Takeaways & Limitations
The benchmark datasets are relatively small for robustly training all LSTMs in the cascade.
Abstract
from arXiv · showhide
This work is about recognizing human activities occurring in videos at distinct semantic levels, including individual actions, interactions, and group activities. The recognition is realized using a two-level hierarchy of Long Short-Term Memory (LSTM) networks, forming a feed-forward deep architecture, which can be trained end-to-end. In comparison with existing architectures of LSTMs, we make two key contributions giving the name to our approach as Confidence-Energy Recurrent Network -- CERN. First, instead of using the common softmax layer for prediction, we specify a novel energy layer (EL) for estimating the energy of our predictions. Second, rather than finding the common minimum-energy class assignment, which may be numerically unstable under uncertainty, we specify that the EL additionally computes the p-values of the solutions, and in this way estimates the most confident energy minimum. The evaluation on the Collective Activity and Volleyball datasets demonstrates: (i) advantages of our two contributions relative to the common softmax and energy-minimization formulations and (ii) a superior performance relative to the state-of-the-art approaches.
1. Introduction
CERN targets video recognition of individual actions, interactions, and group activities where cascaded LSTM predictions can become brittle under uncertainty. It extends a two-level LSTM hierarchy with energy minimization and confidence-based inference, trained end-to-end and evaluated against softmax, energy-based, and state-of-the-art approaches.
- CERN recognizes individual actions, human interactions, and group activities arising from coordinated activity in videos.
- Existing two-level LSTM cascades forward lower-level predictions to higher levels using softmax outputs, but errors can propagate directly under unseen conditions.
- CERN jointly minimizes prediction energy and maximizes confidence, using p-values to prefer reliable solutions over potentially unstable minimum-energy assignments.
- CERN adds an energy layer that captures dependencies among LSTM predictions and replaces the common softmax output for energy-based, end-to-end training.
- Evaluation on the Collective Activity and Volleyball datasets reports advantages over common softmax and energy-based formulations and superior performance relative to state-of-the-art methods.
2. Related Work
Prior work models group activity structure with graphical or grammar-based methods and increasingly uses recurrent deep architectures. CERN extends energy-based learning by incorporating confidence estimates through p-values to obtain more reliable, numerically stable predictions.
- Group activity recognition often represents spatiotemporal structure through individual actions and pairwise interactions, using graphical or AND-OR grammar models.
- Energy-based objectives have shown potential for structured prediction, especially when training data are limited, but are seldom used in deep group activity architectures.
- Recent recurrent architectures use multilevel LSTM cascades grounded on human trajectories, with softmax predictions forwarded from individual actions to events.
- CERN supplements energy minimization with p-values from conformal prediction to select more reliable and numerically stable predictions.
3. Components of the CERN Architecture
CERN assigns recurrent predictors to human trajectories and their pairwise relations, then uses an energy layer to produce a high-confidence structured prediction. Its two variants differ in whether an event-level LSTM supplies an additional event-class distribution.
- CERN represents a video as a graph whose nodes are human trajectories, edges are trajectory pairs, and labels include individual actions, interactions, and an event class.
- Node and edge LSTMs use CNN-derived trajectory features to output distributions for individual actions and human interactions, which are forwarded to the energy layer.
- CERN variants: CERN-1 predicts individual actions and interactions with LSTMs while the energy layer predicts the event class.
- CERN variants: CERN-2 adds an event LSTM that max-pools node and edge outputs, and its energy layer uses all three class-distribution types for assignment.
4. Formulation of Energy
CERN defines energy functions over predictions at multiple semantic levels, using LSTM softmax outputs and visual cues to represent individual actions, interactions, and events.
- CERN-1 defines the energy of a graph over individual actions and interactions.Node and edge LSTM outputs provide the corresponding potentials and visual representations.
- The model uses CNN-derived visual cues from human trajectories together with node and edge LSTM softmax distributions.
- CERN-2 augments this energy with the softmax output of the event LSTM.The event representation is obtained by max-pooling outputs from node and edge LSTMs.
5. Formulation of Confidence
CERN estimates prediction confidence with conformal-prediction p-values derived from nonconformity measures, then combines node, edge, and event confidences for structured hypotheses.
- 5. Formulation of Confidence: Conformal prediction estimates confidence by comparing a new prediction’s nonconformity with predictions made on training data.The nonconformity measure represents dissimilarity from the system’s training predictions.
- 5.1. Nonconformity Measure and P-values: For an action prediction, the nonconformity measure is one minus its normalized node-LSTM softmax output.The resulting p-value is computed using training trajectories from the same event class.
- 5.1. Nonconformity Measure and P-values: An action prediction has a high p-value when many same-class training examples have larger nonconformity measures.
- 5.1. Nonconformity Measure and P-values: The paper illustrates that a larger softmax output need not identify the more confident action class.In the example, class 2 has the larger softmax output, while class 1 has the higher p-value and confidence.
- 5.1. Nonconformity Measure and P-values: Interaction predictions use an analogous edge-LSTM nonconformity measure and p-value under the event-class context.
- 5.1. Nonconformity Measure and P-values: CERN-2 also defines a nonconformity measure and p-value for event predictions using the event LSTM’s softmax output.The event p-value is computed relative to the training videos.
- 5.2. Confidence of the Structured Prediction G: Fisher’s combined hypothesis test aggregates the p-values of node, edge, and event predictions into a compound structured-prediction confidence.The combined statistic follows a chi-squared distribution with 2N degrees of freedom.
- 5.2. Confidence of the Structured Prediction G: Minimizing the Fisher statistic yields the maximum p-value for the combined hypothesis.This result motivates the energy-layer formulation that follows.
6. The Energy Layer of CERN
CERN’s energy layer combines energy minimization with confidence maximization, selecting structured predictions that are both low-energy and statistically reliable.
- The energy layer jointly minimizes the CERN energy and maximizes a Fisher combined-test p-value.
- Lowering the p-value constraint parameters enforces higher p-values for the selected solution.The constraints apply separately to node, edge, and event p-values.
- The regularized energy is obtained as the Lagrangian of the constrained energy-and-confidence optimization.
- The energy layer takes softmax outputs and estimated p-values from all LSTMs as inputs.
- The energy-layer computation is implemented after deriving the regularized-energy expression.
- After stacking energy-layer units, CERN selects the solution with minimum regularized energy.
7. Learning Regularized By Confidence
CERN trains its energy layer with an energy-based loss that penalizes incorrect event assignments relative to the most violated alternative.
- The energy-based loss learns energy-layer parameters from training instances, ground-truth labels, and regularized energies.
- The most violated alternative is the non-ground-truth assignment minimizing the regularized-energy margin expression.
- Because the normalization term is not modeled, the alternative assignment need not be a global minimum.Its latent action and interaction labels can be set to the node- and edge-LSTM outputs.
- The representation excluding the energy layer can first be trained with common losses such as cross-entropy.P-values are then computed with each training instance removed from the relevant training sets before training the energy-layer weights.
8. Results
CERN is evaluated against recurrent-network baselines on Collective Activity and Volleyball, including corrupted-input stability and qualitative recognition. Across both datasets, energy minimization with p-value regularization improves recognition and robustness relative to softmax or unregularized alternatives.
- The evaluation compares CERN with state-of-the-art methods and baselines using multi-class classification accuracy (MCA) and mean per-class accuracy (MPCA).
- Collective Activity: The energy layer significantly boosts Collective Activity group-activity accuracy, while CERN-1 outperforms the two-layer LSTM baseline even with only the bottom LSTM layer.
- Collective Activity: 72.7% individual-action accuracy and 59.9% interaction accuracy are obtained on Collective Activity using node and edge LSTMs, respectively.
- Robustness: Under corrupted trajectories, CERN-2 consistently degrades less than baselines without p-values, with the benefit increasing as corruption rises.The corruption procedure randomly shifts selected bounding boxes by 20%–80% of their original dimensions.
- Qualitative and robustness results: CERN-2 correctly recognizes Collective Activity examples despite incorrect individual-action labels and improves Volleyball inference reliability under input distortion.
- Volleyball: On Volleyball, regularized energy minimization improves accuracy over conventional energy minimization, and CERN-2 outperforms the state of the art with either pooling type.CERN-2 adds the event LSTM, which addresses complex group activities not sufficiently represented by individual actions alone.
9. Conclusion
CERN combines hierarchical LSTMs with an energy layer and confidence-based regularization for activity recognition. Evaluations on Collective Activity and Volleyball report superior performance and more numerically stable inference under uncertainty.
- CERN extends a two-level LSTM hierarchy with confidence and energy modeling for group activities, interactions, and individual actions.
- The differentiable energy layer computes energy regularized by Fisher’s combined-test p-value, and the resulting energy-based loss supports end-to-end learning.
- CERN delivers superior performance and more numerically stable solutions under uncertainty on the Collective Activity and Volleyball datasets.
- CERN-1 is more accurate than a strong two-level softmax-LSTM baseline for collective activities, while CERN-2 improves accuracy on complex group activities.
Appendix
The appendix presents the compact regularized-energy formulation and identifies the energy-layer parameters and inputs used by CERN.
- The regularized energy is reformulated in a compact form using grouped energy-layer weights and parameter vectors.
- The energy layer receives LSTM softmax outputs and p-values as its inputs.