Source-linked AI summary
An end-to-end generative framework for video segmentation and recognition
Hilde Kuehne, Juergen Gall, Thomas Serre
TL;DR
The paper addresses the difficulty of recognizing and parsing complex, long-duration activities with models suited to structured temporal behavior and generative observations. It combines reduced Fisher Vectors with HMMs and grammar-based recognition, finding significant gains over the state of the art when sufficient training data is available.
Problem
Existing action-recognition methods focus mainly on short single-action clips, while generative models require low-dimensional, normally distributed inputs that standard visual representations often lack.
Method
The system combines compact reduced Fisher Vectors with Gaussian-mixture HMMs and a grammar modeling sequences of action units.
Results
When sufficient training data was available, the structured generative system outperformed the state of the art for activity recognition and parsing.
Takeaways & Limitations
Structured generative temporal models can be competitive for complex activity segmentation and recognition on adequately sized datasets.
Takeaways & Limitations
The evaluation used only video data and did not combine camera inputs from different views.
Abstract
from arXiv · showhide
We describe an end-to-end generative approach for the segmentation and recognition of human activities. In this approach, a visual representation based on reduced Fisher Vectors is combined with a structured temporal model for recognition. We show that the statistical properties of Fisher Vectors make them an especially suitable front-end for generative models such as Gaussian mixtures. The system is evaluated for both the recognition of complex activities as well as their parsing into action units. Using a variety of video datasets ranging from human cooking activities to animal behaviors, our experiments demonstrate that the resulting architecture outperforms state-of-the-art approaches for larger datasets, i.e. when sufficient amount of data is available for training structured generative models.
1. Introduction
Real-world activity recognition requires parsing long, complex behavior sequences, but prevailing unstructured and sliding-window methods are poorly suited to this setting. The paper proposes reduced Fisher Vectors for structured generative models and reports substantial gains, especially with sufficient training data.
- Real-world monitoring requires recognizing and parsing long recordings composed of goals and sub-goals.
- Unstructured clip-level classifiers perform well on single actions but are poorly suited to complex daily activity sequences.
- Sliding-window methods have mainly been tested on small datasets and are not competitive on more challenging activity datasets.
- Gaussian-mixture generative models favor normally distributed, low-dimensional inputs, whereas standard Bag-of-Words and Fisher Vectors are sparse and high-dimensional.
- The proposed reduced Fisher Vectors are designed for structured temporal models and improve recognition across activity segmentation and recognition tasks.
- Structured generative models show significant improvements on datasets containing sufficient training data.
2. Related work
Prior Fisher Vector work largely targeted discriminative recognition of short, pre-segmented actions, while structured temporal models addressed activity structure through generative and grammar-based approaches. This paper focuses on the statistical suitability of Fisher Vectors for generative action recognition.
- Fisher vectors: Fisher Vectors have achieved strong accuracy and compact representations in image and action recognition.
- Fisher vectors: Earlier Fisher Vector action-recognition studies used discriminative classifiers on short, single-action, pre-segmented clips.
- Research gap: The authors identify a lack of prior work studying Fisher Vectors' statistical properties for generative action-recognition models.
- Structured temporal models: Previous structured approaches used generative mixtures, Bayesian networks, HMM/SVM combinations, dynamical systems, and grammar-based models for temporal activity structure.
- Structured temporal models: The paper builds on HMMs combined with a simple grammar to model complex activities as sequences of action units.
3. System description
The system reduces Fisher Vector representations to make them more compatible with Gaussian-mixture HMMs, then combines action-unit HMMs with a grammar for activity parsing. PCA improves the normality fit, while training and recognition estimate probable action-unit sequences over time.
- Fisher vectors: Fisher Vectors concatenate GMM gradients into a 2 × D × K representation, then apply L2 and power normalization followed by PCA reduction and whitening.
- Normality test: Normality is evaluated with Lilliefors and Jarque-Bera tests across sampled feature dimensions before and after PCA.
- Normality test: PCA produces feature vectors that better fit the HMM's normality assumption and yields gains in activity-recognition accuracy.
- Recognition pipeline: Each frame is represented by a 64-dimensional Fisher Vector after sampling features, fitting GMMs, sliding-window computation, PCA, and normalization.
- Recognition pipeline: Action-unit HMMs use left-to-right states initialized from temporal subdivisions, with state boundaries and GMMs updated during training.
- Recognition pipeline: During recognition, HMMs and a learned context-free grammar find the most probable action-unit sequence, with Viterbi providing boundaries and observation probabilities.
4. Evaluation
The system was evaluated across diverse complex-activity datasets for activity classification, action-unit detection, and segmentation. It generally surpassed prior methods on larger datasets with sufficient training data, while lagging on smaller datasets.
- Datasets: Eight complex-activity datasets were evaluated, including cooking, daily-activity, assembly, sports, and animal-behavior videos.The evaluation used ADL, Olympic, ToyAssembly, CMU-MMAC, MPIICooking, 50Salads, Breakfast, and CRIM13.
- Tasks: The evaluation covered activity classification, action-unit detection, and segmentation, except Olympic Sport lacked action-unit labels.For Olympic Sport, ten clips per class were manually labeled to initialize the system before labeling remaining training clips with system outputs.
- System evaluation: Replacing HOGHOF with DTFs improved overall Breakfast-system accuracy by ∼10 −14% in the HTK comparison.This comparison used FV-based encoding on the Breakfast dataset.
- System evaluation: HTK-based classification exceeded SVM classification by ∼20 −30% with identical features, while reduced FVs improved baseline recognition by ∼20% for HOGHOF and ∼30% for DTF.SVM performed better with full rather than reduced FVs, but remained below the HTK system.
- Segmentation: For datasets with at least 8 hours of video, segmentation significantly outperformed the state of the art, including BF +27.5%.The system underperformed the best ToyAssembly segmentation result, a small dataset with about one hour of video.
- Activity classification: Activity classification was below the state of the art on smaller ADL and Olympic Sports datasets but exceeded it on Breakfast.The classification evaluation considered ADL, Olympic, and Breakfast because they provide multiple activity classes.
5. Conclusion
The paper combines compact Fisher Vector representations with Hidden Markov Models for recognizing goal-oriented activities and parsing action units. With sufficient training data, the structured generative system outperformed the state of the art.
- Conclusion: Compact Fisher Vectors combined with Hidden Markov Models yielded significant accuracy gains for activity recognition and action-unit parsing.The study systematically compared the resulting structured generative system with state-of-the-art methods for segmentation and classification.
- Conclusion: With sufficient training data, structured generative temporal models outperformed the state of the art.This conclusion applied to both recognition of goal-oriented activities and parsing into task-oriented action units.