Source-linked AI summary
When will you do what? - Anticipating Temporal Occurrences of Activities
Yazan Abu Farha, Alexander Richard, Juergen Gall
TL;DR
The paper targets the gap between short-horizon activity anticipation and forecasting long sequences of future actions with their durations. It proposes RNN and CNN predictors operating on inferred activity labels, and reports accurate long-horizon predictions that outperform several baselines. The evaluation also shows that errors in observed labels can propagate into future predictions.
Problem
Existing approaches mainly recognize observed activities or anticipate only the very recent future, whereas this paper targets activity sequences over horizons of up to 5 minutes.
Method
The paper first infers observed activities with an RNN-HMM, then uses either a recursive RNN or a direct CNN to predict future activity labels and durations.
Results
The two approaches outperform grammar-based and nearest-neighbor baselines by a large margin, perform similarly beyond 40 seconds, and are competitive with direct visual-representation prediction.
Takeaways & Limitations
The methods provide long-horizon predictions across long, varied videos, including sequences with many possible activities and varying observed-data quality.
Takeaways & Limitations
With noisy observed labels, errors in the observed part are propagated to future predictions, so the clean evaluation assumes perfect observed segmentation.
Abstract
from arXiv · showhide
Analyzing human actions in videos has gained increased attention recently. While most works focus on classifying and labeling observed video frames or anticipating the very recent future, making long-term predictions over more than just a few seconds is a task with many practical applications that has not yet been addressed. In this paper, we propose two methods to predict a considerably large amount of future actions and their durations. Both, a CNN and an RNN are trained to learn future video labels based on previously seen content. We show that our methods generate accurate predictions of the future even for long videos with a huge amount of different actions and can even deal with noisy or erroneous input information.
1. Introduction
The paper addresses long-term anticipation of activity sequences, including their order, classes, and temporal boundaries, rather than only recognizing observed or immediately upcoming actions. It proposes RNN- and CNN-based predictors and reports strong performance across challenging long videos.
- Motivation: Existing video systems mainly classify observed content or anticipate activities only over a few seconds.This short horizon is insufficient for applications such as robots interacting closely with humans.
- Problem: The task is to anticipate all activities within a horizon of up to 5 minutes, including each activity’s class, order, start, and end.
- Approach: The proposed methods first infer observed activities with an RNN-HMM, then predict future activities using either an RNN or CNN.The RNN recursively predicts durations and the next class, while the CNN provides the second proposed architecture.
- Results: Both approaches outperform grammar-based and nearest-neighbor baselines by a large margin on two challenging datasets.The datasets contain long sequences and substantial variation.
- Results: The RNN and CNN perform similarly beyond 40 seconds, while the RNN performs better below 20 seconds.The comparison concerns different future-prediction time horizons.
2. Related Work
Prior work largely addresses short-horizon prediction, early recognition, or future visual representations. This paper instead predicts longer sequences of activities together with their temporal boundaries, up to 5 minutes.
- Future prediction: Future-frame prediction at pixel level is limited to a few frames, while visual-representation prediction is limited to a 5-second horizon.
- Early activity detection: Early activity detection recognizes an ongoing action from partial observations, rather than forecasting a sequence of future activities.
- Next-action prediction: Other approaches predict the next action or its starting time, generally over only a short horizon.
- Sports prediction: Sports-video studies predict immediate future events such as shot locations, player moves, or ball locations.
- Contribution: This work anticipates an activity sequence, including start and end points, within time horizons of up to 5 minutes.
3. Anticipating Activities
The paper separates recognition of observed activities from future anticipation, then develops recursive RNN and direct CNN predictors for future labels and durations. Their training procedures encode temporal activity structure in sequence or matrix form.
- Task formulation: Observed video frames are first converted into inferred activity labels, which are then used to predict future activities.Separating the two stages allows the effects of recognition and anticipation to be studied independently.
- RNN-based Anticipation: The RNN recursively predicts the remaining duration of the current action and the class and duration of the next action.Each prediction is appended to the input and forwarded again until the desired future horizon is reached.
- RNN-based Anticipation: RNN training uses random split points in ground-truth segmentations to create input sequences and target triplets for remaining duration, next-action duration, and next-action class.This procedure generates many training examples from each labeled sequence.
- CNN-based Anticipation: The CNN encodes observed and future activity labels as matrices indexed by action class and temporal segment, then predicts the future matrix in one step.Unlike the RNN, it does not rely on recursive prediction.
- CNN-based Anticipation: CNN training uses observations of 10%, 20%, 30%, or 50% of each video and predicts the following 50% as ground truth.
- CNN-based Anticipation: The CNN output uses squared error with row-wise ℓ2 normalization, followed by one-dimensional Gaussian temporal smoothing.The authors report this combination as more robust than softmax with cross-entropy and say smoothing reduces spurious short-segment predictions.
4. Experiments
Experiments on Breakfast and 50Salads evaluate long-horizon future action prediction under ground-truth and noisy observations. The proposed RNN and CNN outperform baselines, with performance depending on prediction horizon, video length, observation noise, and model design choices.
- Datasets and evaluation: The experiments use Breakfast and 50Salads, long-sequence datasets with different numbers of videos, action classes, video lengths, and action instances.Accuracy is reported as mean over classes; the longest videos in both datasets are 10 minutes.
- Experimental settings: The evaluation compares grammar and nearest-neighbor baselines with ground-truth observations and labels decoded from observed video features.The noisy-observation setting tests robustness when observed labels contain errors that may propagate into future predictions.
- Ground-truth observations: RNN and CNN predictions perform well against baselines, with RNN generally stronger for short horizons and CNN similar or sometimes better for longer horizons.Recursive RNN predictions can propagate errors, whereas CNN predicts future actions directly but tends to miss short action segments.
- Noisy observations: Prediction remains comparatively stable under noisy observations, although errors in decoded observed labels can propagate into future segment predictions.On 50Salads, the accuracy loss relative to perfect observations is described as surprisingly small, partly because of decoder performance and strong inter-class dependencies.
- Prediction horizon and video length: Performance is better for shorter videos and decreases as the amount of future or the number of predicted actions increases.The authors attribute the video-length effect mainly to shorter videos having less future to predict.
- CNN analysis: The CNN architecture improves over VGG-16 by up to 9% with ground-truth observations, and its squared loss with ℓ2 normalization reduces over-segmentation relative to softmax cross-entropy.Direct prediction from features is up to +5% worse than the two-step approach, which decouples observed-action recognition from future prediction.
5. Conclusion
The paper introduces efficient RNN and CNN methods for predicting future video actions over several minutes. The methods produce accurate predictions across varied datasets, video lengths, observed-data quality, and future-action diversity.
- The paper introduces RNN and CNN methods for predicting future actions over several minutes.These methods address a task described as previously unaddressed.
- The methods produce accurate predictions across datasets and videos with varying lengths and observed-data quality.
- The methods handle huge variations in the possible future actions.