Source-linked AI summary
Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videos
Serena Yeung, Olga Russakovsky, Ning Jin, Mykhaylo Andriluka, Greg Mori, Li Fei-Fei
TL;DR
Dense action understanding requires labeling multiple actions across every video frame, a capability not provided by existing datasets described here. The paper introduces MultiTHUMOS and a MultiLSTM model with temporal input-output context, and reports improved dense action labeling while enabling structured retrieval and action prediction.
Problem
Existing datasets and action-recognition settings do not provide untrimmed, densely labeled videos with multiple labels per frame for detailed multilabel action understanding.
Method
The paper extends THUMOS into MultiTHUMOS and introduces MultiLSTM, an LSTM-based model with temporally extended input and output connections.
Results
MultiLSTM improves action-labeling performance over a plain LSTM baseline on the dense, multilabel benchmark.
Takeaways & Limitations
The dataset and model support dense multilabel action detection and further tasks including structured retrieval and action prediction.
Takeaways & Limitations
MultiTHUMOS contains very short action instances, making some actions difficult to detect, and soft output attention did not show noticeable improvements.
Abstract
from arXiv · showhide
Every moment counts in action recognition. A comprehensive understanding of human activity in video requires labeling every frame according to the actions occurring, placing multiple labels densely over a video sequence. To study this problem we extend the existing THUMOS dataset and introduce MultiTHUMOS, a new dataset of dense labels over unconstrained internet videos. Modeling multiple, dense labels benefits from temporal relations within and across classes. We define a novel variant of long short-term memory (LSTM) deep networks for modeling these temporal relations via multiple input and output connections. We show that this model improves action labeling accuracy and further enables deeper understanding tasks ranging from structured retrieval to action prediction.
1 Introduction
The paper targets dense, multilabel understanding of continual human activity in unconstrained video, addressing the limits of single-action descriptions. It introduces the MultiTHUMOS dataset and MultiLSTM model to capture temporal relations among simultaneous and consecutive actions.
- Motivation: Single-action recognition is insufficient because people perform multiple simultaneous and connected actions throughout videos.The paper motivates moving beyond one-label descriptions to represent activity more completely.
- Dataset: The dataset expands coverage from 20 to 65 action classes and from 0.3 to 1.5 labels per frame.It also increases the average number of distinct action categories per video from 1.1 to 10.5.
- Dataset: MultiTHUMOS includes hierarchical and fine-grained action relationships, such as Throw versus BasketballShot and Guard versus Block.These labels support analysis of intricate relationships among action categories.
- Model: MultiLSTM extends LSTM models with temporally extended input and output connections for dense, multilabel action detection.The model learns dependencies between nearby frames and between actions occurring in the same frame.
- Results: MultiLSTM improves action-labeling performance over a plain LSTM baseline on the dense, multilabel benchmark.The paper also reports that the model supports deeper human-action understanding tasks.
2 Related Work
Related work spans datasets for individual, localized, unconstrained, and domain-specific action recognition, alongside models for temporal structure and visual attention. The paper positions MultiTHUMOS and its LSTM model as emphasizing general, dense multilabel labeling in unconstrained videos.
- Datasets: Earlier datasets catalyzed recognition of individual actions, while later benchmarks addressed surveillance, unconstrained internet video, retrieval, and restricted action domains.The cited examples include KTH, Weizmann, TRECVID, Hollywood, HMDB, UCF101, ActivityNet, THUMOS, MPII Cooking, and Breakfast.
- Datasets: Existing datasets differ in whether labels are nonlocalized, temporally localized, sequential per frame, or restricted to closed-world environments.The paper uses these distinctions to motivate a dataset combining generality, multiple labels per frame, and broad action categories.
- Video representations: Video representations have shifted from hand-crafted features toward learned convolutional features, including two-stream architectures using image and optical-flow inputs.Related work also studies spatial and temporal feature aggregation for video analysis.
- Temporal modeling: Temporal action modeling includes HMMs, pose-state diagrams, temporal pooling, latent SVMs, action grammars, and LSTM-based video models.These approaches address temporal evolution, aggregation, localization, classification, and sentence generation in different settings.
- Positioning: Unlike prior LSTM video work focused on classification or sentence generation, the paper develops an LSTM for spatial input aggregation and dense labeling output.This distinction connects the model design directly to the multilabel action-detection setting.
- Attention: Prior attention methods use eye gaze, visual attention, or soft temporal attention for action detection, localization, and caption generation.The paper builds on these directions with attention-based temporal context for dense action labeling.
3 The MultiTHUMOS Dataset
MultiTHUMOS extends THUMOS with dense, frame-level multilabel annotations across unconstrained videos, enabling analysis of simultaneous actions, temporal interactions, and action relationships. Its broad coverage also introduces substantial variation in class frequency, action duration, and visual appearance.
- Dataset construction: MultiTHUMOS provides dense, multilabel, frame-level annotations for 30 hours across 400 THUMOS videos, with annotations collected for 65 action classes.The dataset includes all videos in THUMOS’s Validation Data and Test Data sets.
- Dataset scale and density: The dataset contains 38,690 annotations over 65 classes, compared with 6,365 annotations over 20 classes in THUMOS.The added classes span varied lengths, hierarchical and fine-grained categories, and sport-specific and general actions.
- Action relationships and variation: Dense annotations allow action relationships to be learned from per-frame co-occurrence, revealing frequent pairs such as squat and body contract and rare pairs such as run and billiards.The dataset contains both sport-specific and general actions, with high intra-class variation across viewpoints and group sizes.
- Dataset challenges: MultiTHUMOS exhibits a long-tailed class distribution, ranging from 27 seconds to 5 hours of annotated video per action class.This creates uneven amounts of training evidence across action categories.
- Dataset challenges: Action instances are shorter on average in MultiTHUMOS than THUMOS, with seven classes averaging less than one second and some instances lasting only two frames.Short actions provide little visual signal, making contextual modeling and multi-action reasoning important for localization.
4 Technical Approach
The approach combines dense multilabel video modeling with a MultiLSTM that accesses temporal context through expanded input and output connections.
- MultiLSTM extends recurrent models for dense action labeling by modeling temporal relations among action labels across frames.
- The model expands an LSTM’s temporal receptive field at both its input and output connections.These connections provide direct access to recent inputs and allow predictions to be refined after additional frames are observed.
- Multilabel Loss: For multilabel prediction, frame features are mapped to independent action-class scores rather than a single softmax distribution.The inputs are 4096-dimensional VGG features, and each class is trained with an independent logistic regression loss.
- Multiple Inputs with Temporal Attention: A fixed-length window of previous frames replaces the single-frame input, with soft attention weighting each frame’s contribution.The attention weights are normalized and quantify agreement between observed features and features sought by the network.
- Multiple Outputs: Multiple output predictions are consolidated across a temporal window using a weighted average, with uniform averaging used in the experiments.Soft attention over output predictions did not yield noticeable improvements in these experiments.
- Offset Predictions: Offset predictions train the network to predict labels at shifted future times, supporting action prediction from unconstrained video.The paper illustrates predicting Catch or Hit from a frame showing a person cocking an arm to throw.
5 Experiments
The experiments evaluate MultiLSTM for dense per-frame action labeling, retrieval, and temporal action prediction. MultiLSTM improves labeling over CNN and LSTM baselines and supports temporally informed retrieval and prediction.
- Experimental Setup: The experiments evaluate dense per-frame action labeling on MultiTHUMOS using Average Precision and mean Average Precision across action categories.The study also evaluates action detection and prediction, using the THUMOS train/test splits while treating clipped training videos as weak supervision.
- Dense Action Labeling: 29.7% mAP is achieved by MultiLSTM, compared with 28.1% for the base LSTM and 25.4% for the single-frame CNN.The model’s input connections, input attention, and output connections each contribute to accurate action labeling.
- Dense Action Labeling: MultiLSTM outperforms the CNN on 56 of 65 action classes and the LSTM on 50 of 65 action classes.The comparisons indicate stronger temporal modeling on many classes, including actions associated with preceding or surrounding context.
- Dense Action Labeling: Increasing attention units improves per-frame mAP up to 75 units, after which performance decreases as the parameter count becomes too large.The experiments use 50 attention units in practice.
- Dense Action Labeling: MultiLSTM produces more consistent detections than the CNN, which often produces short, disjoint detections, by using temporal and co-occurrence context.The outputs shown for visualization are binarized from the model’s per-frame probabilistic action labels.
- Structured Retrieval: The model retrieves segments containing ordered or simultaneous actions, such as Pass then Shot and Sit and Talk, by learning temporal and co-occurrence relationships.This extends dense multilabel action labeling to structured retrieval of detailed action descriptions.
- Action Prediction: Prediction mAP is approximately 30% for actions 0.5 seconds in the past and approximately 20–25% up to 12 seconds into the future.The model predicts past actions more strongly than future actions, while still using learned temporal relationships to infer both directions.
- Action Prediction: MultiLSTM correctly predicts that Fall may follow Jump and BasketballShot may follow Dribble one second into the future.These examples illustrate temporal action prediction from current-frame context.
6 Conclusion
The paper advances dense, multilabel human action understanding through the MultiTHUMOS dataset and a recurrent model with soft-attention temporal context. Experiments show improved labeling accuracy and support more detailed action understanding.
- Contributions: MultiTHUMOS provides substantial labeled data for research on dense, multiple-label action recognition.The dataset broadens human action understanding beyond single labels per frame.
- Contributions: MultiLSTM incorporates soft-attention temporal context through multiple input and output connections for dense action labeling.The model is presented as a novel LSTM-based approach to modeling temporal relationships.
- Conclusions: Using MultiLSTM on MultiTHUMOS improves action-labeling accuracy and permits detailed understanding of human action.The conclusion frames the contribution as progress in both the task definition and the modeling approach.