Source-linked AI summary
Recognizing Fine-Grained and Composite Activities using Hand-Centric Features and Script Data
Marcus Rohrbach, Anna Rohrbach, Michaela Regneri, Sikandar Amin, Mykhaylo Andriluka, Manfred Pinkal, Bernt Schiele
TL;DR
The paper addresses underexplored fine-grained and composite activity recognition, where subtle motions, activity variability, and limited composite-specific training data pose challenges. It introduces MPII Cooking 2 and combines hand- and pose-centric features with attribute-based representations and script data. The results show benefits for fine-grained recognition and script-enabled transfer to composites, including unseen ones.
Problem
Fine-grained and composite activity recognition remain underexplored, especially for visually similar motions and variable composites with limited training data.
Method
The paper introduces MPII Cooking 2 and evaluates pose- and hand-centric features while representing composite activities through transferable attributes derived from video and scripts.
Results
Hand-centric features benefit fine-grained activity and object recognition, while attribute-based recognition with scripts improves composite recognition and enables zero-shot transfer.
Takeaways & Limitations
Hand regions are useful for fine-grained kitchen activities, and scripts provide information that supports recognizing composites without training videos for those composites.
Takeaways & Limitations
Future work remains to validate the hand-centric approach in other domains and model video temporal structure using scripts.
Abstract
from arXiv · showhide
Activity recognition has shown impressive progress in recent years. However, the challenges of detecting fine-grained activities and understanding how they are combined into composite activities have been largely overlooked. In this work we approach both tasks and present a dataset which provides detailed annotations to address them. The first challenge is to detect fine-grained activities, which are defined by low inter-class variability and are typically characterized by fine-grained body motions. We explore how human pose and hands can help to approach this challenge by comparing two pose-based and two hand-centric features with state-of-the-art holistic features. To attack the second challenge, recognizing composite activities, we leverage the fact that these activities are compositional and that the essential components of the activities can be obtained from textual descriptions or scripts. We show the benefits of our hand-centric approach for fine-grained activity classification and detection. For composite activity recognition we find that decomposition into attributes allows sharing information across composites and is essential to attack this hard task. Using script data we can recognize novel composites without having training data for them.
1 Introduction
The paper targets fine-grained activities with subtle, hand-centric differences and composite activities built from multiple steps. It introduces MPII Cooking 2 and combines hand- and pose-based recognition with attribute sharing and script-based transfer.
- Motivation: Fine-grained activities have low inter-class variability and often require distinguishing subtle body motions and participating objects.Examples include mix versus stir, slice versus cut dice, and open egg versus open tin.
- Motivation: Composite activities are temporally decomposable, highly variable, and difficult to recognize from limited training data.The paper represents their components and participants as attributes that can be shared across composites.
- Dataset: MPII Cooking 2 is a large annotated kitchen dataset designed for fine-grained activity classification and detection, composite recognition, and related pose challenges.The dataset contains realistic cooking activities and detailed annotations for activities and participants.
- Approach: The authors focus on body pose and hand regions because many kitchen activities are hand-centric and relevant objects occur near the hands.Hand detections support extracting color SIFT and Dense Trajectories for activities and participating objects.
- Approach: The proposed composite-activity approach uses attribute-based representations to share knowledge and textual scripts to transfer attributes to unseen composites.Scripts provide information about relevant activities, tools, and ingredients despite limited video examples.
2 Related work
Prior activity-recognition datasets and methods largely emphasize full-body, web, movie, or holistic settings rather than subtle fine-grained motions. The paper positions MPII Cooking 2 around this gap and extends recognition with pose, hand-centric, and script-based approaches.
- Activity datasets: Existing datasets span full-body pose, movie and web, surveillance, and assisted daily living categories, with different classification and detection capabilities.Table 1 organizes datasets by task support, activity classes, clips, subjects, frames, and resolution.
- Activity datasets: MPII Cooking 2 combines fine-grained activity detection with recognition of highly variable composite activities.This addresses a gap not jointly covered by the reviewed datasets.
- Activity datasets: Prior datasets commonly emphasize full-body actions, large inter-class variability, or challenging web and movie conditions rather than fine-grained motions.The paper identifies limited benchmarks for visually similar, hand-centric activities.
- Recognition approaches: The paper evaluates holistic Dense Trajectories alongside two pose-based and two hand-centric approaches on fine-grained activities.Its pose features encode joint trajectories, while hand-centric features focus on regions where relevant objects are manipulated.
- Text for recognition: Cooking instructions are mined for relevant activities, tools, and ingredients, with tf∗idf reported to discriminate dishes better than co-occurrence statistics in most cases.The script representation supports zero-shot recognition and composite-activity transfer.
- Relation to this work: The extended work updates MPII Cooking 2, adds hand-centric methods and propagated semantic transfer, expands experiments, and releases dataset-related resources.The release includes the updated dataset, intermediate features, and script data.
3 Dataset “MPII Cooking 2”
MPII Cooking 2 is a large-scale kitchen dataset designed for fine-grained and composite activity recognition, with varied cooking behavior, temporal activity annotations, participants, pose and hand data, and independently collected scripts.
- Dataset scope: Videos depict composite cooking activities such as preparing dishes, while annotated intervals identify fine-grained activities and participating tools, ingredients, and locations.
- Dataset scope: MPII Cooking 2 contains 273 videos from 30 subjects, totaling more than 27 hours and 2,881,616 frames.
- Recording protocol: Subjects were instructed only which dish to prepare, producing variation in tools, activity choices, and temporal order.
- Annotations: Annotations include composite labels per video, fine-grained start and end frames, participant categories, and selected frames with human pose and hand annotations.
- Script data: The dataset provides independently collected tutorial-like scripts for 53 kitchen tasks, with sequential instructions limited to 15 steps per sequence.
- Script data: The script corpus adds variation beyond video examples but includes different wording, granularity, preparation choices, and coordinated events.
4 Hand detection and pose estimation
The paper combines specialized hand detection with flexible pose estimation to improve localization of hands and upper-body parts for hand-centric activity recognition.
- Hand detection: Hand localization uses hand appearance, person context, and articulated pose constraints to handle occlusion, object obstruction, and posture variability.The person detector constrains hand search locations and scale, while hand hypotheses are integrated into pose estimation.
- Hand detection: The hand detector distinguishes left and right hands because they often play different roles in an activity.Separate DPM components are jointly trained for the two hands, with multiple components capturing posture variation.
- Pose estimation: The flexible pictorial-structures model represents 10 upper-body parts connected by pairwise relationships and supports efficient exact inference through a tree structure.Part appearance uses boosted detectors and shape-context features; pairwise terms model relative position and orientation.
- Evaluation: 75.9 PCP versus 66.0 for PS demonstrates the overall pose-estimation improvement, including gains for lower arms important to hand-centric activities.Lower-arm PCP improves from 48.9 to 74.4 and from 49.6 to 70.3 in the reported comparisons.
- Evaluation: The hand detector alone outperforms the proposed FPS model for hand localization, while integrating hand hypotheses into pose estimation provides a further moderate improvement.The authors attribute the moderate gain to pose estimation balancing hand localization against localization of other body parts.
5 Approaches for fine-grained activity recognition and detection
For fine-grained activity recognition, the paper compares pose-based, holistic, and hand-centric representations, emphasizing localized hand and object information for activities in assisted daily living.
- Feature families: The evaluation benchmarks body-pose features, Dense Trajectories, and hand-centric visual features for recognizing fine-grained activities and participating objects.The hand-centric features target activities and objects typically located near the hands.
- Pose-based features: Body model features summarize joint velocity, acceleration, and inter-joint distances, while FFT features encode frequency information from joint trajectories.The FFT representation has 256 dimensions, including exponential bands, cepstral coefficients, spectral entropy, and energy.
- Holistic features: Dense Trajectories provide holistic HOG, HOF, MBH, and trajectory-speed features from densely sampled points tracked through the full frame.The implementation follows the authors’ state-of-the-art code and parameters.
- Hand-centric features: Hand-Trajectories represent hand motion, hand shape, temporal shape variation, and manipulated-object appearance by tracking dense points around estimated hands.Hand-cSift complements this with color Sift features extracted around both hands on RGB and grayscale channels.
- Classification and detection: Activity classification trains one-vs-all SVMs for activity attributes, while detection uses sliding windows, integral histograms, and non-maximum suppression across window lengths.Attributes include both fine-grained activities and participating objects, such as cut, knife, or cucumber.
6 Modeling composite activities
The paper models composite activities through shared activity and object attributes, exploiting temporal context, co-occurrence, and script-derived semantic associations to address diverse composites and limited training data.
- Composite representation: Composite activities are represented as flexible combinations of attributes describing fine-grained activities and their participating objects.This decomposition enables information sharing across composite classes.
- Context and co-occurrence: Context features max-pool attribute scores across other intervals, while co-occurrence features use the other attribute scores within the same interval.These features encode relationships such as peeling preceding or co-occurring with other activities and objects.
- Composite classification: Sequence-level composite recognition max-pools each attribute over all intervals and classifies the resulting representation with nearest neighbors or one-versus-all SVMs.The representation captures attribute presence or absence but leaves temporal ordering for future work.
- Script-based knowledge: Script data supplies sparse class-attribute associations that produce weighted composite scores and can constrain nearest-neighbor distances to relevant attributes.Binarizing nonzero associations reduces distance computation to relevant attributes and normalizes by their count.
- Script-based knowledge: Propagated semantic transfer extends script-based scoring with label propagation over distances in unlabeled data under a transductive setting.The method assumes all test data are available when predicting an individual test label.
7 Evaluation
Evaluation shows that hand-centric and holistic features complement each other across fine-grained recognition tasks, while attribute-based representations and script data support composite recognition and transfer to unseen composites.
- Fine-grained activity classification: 43.7% recognition AP for activities and objects together is reached by combining Hand-Trajectories with Hand-cSift features.Hand-centric features improve recognition because relevant information is concentrated around the hands.
- Fine-grained activity classification: Hand-Trajectories outperform Dense Trajectories on 35 activity classes versus 25, and on 65 object classes versus 43.The advantage is strongest for activities and objects involving hand-centered actions, while coarser motions can favor holistic features.
- Fine-grained activity detection: 28.6% overall AP is achieved for fine-grained activity detection by combining holistic and hand-centric features.Hand-centric features alone perform worse than Dense Trajectories for detection, because holistic information helps capture activity intervals and detection boundaries.
- Attribute recognition: 39.3% AP is obtained for attribute recognition when context and co-occurrence are combined with the base classifier, improving its result by 3.2%.Temporal context alone drops performance from 36.1% AP to 11.1%, whereas co-occurrence exploits relationships between activities and their participants.
- Composite activity classification: Attribute-based composite recognition outperforms low-level features, and the authors attribute gains from their segmentation to coarser, less noisy attributes.The segmentation removes noisy and background segments, producing more robust attributes for composite recognition.
- Composite activity classification: 57.5% AP is achieved by PST with the combined feature, while script data also enables transfer and zero-shot recognition of composites absent from training.In the Disjoint Composites setting, PST+Script data is outperformed by an SVM; zero-shot performance remains below supervised recognition.
8 Conclusion
The paper addresses fine-grained and composite activity recognition with the MPII Cooking 2 dataset, hand-centric features, attribute representations, and textual scripts. Hand-centric features benefit fine-grained recognition, while script data improves composite recognition and enables zero-shot transfer.
- Dataset: MPII Cooking 2 contains 273 videos totaling more than 27 hours from 30 subjects performing realistic cooking activities.The dataset includes annotations for activities, objects, human pose, and text descriptions.
- Composite recognition: The qualitative composite results show top-six activity and object attributes, with composite predictions evaluated against ground truth and similar attributes sometimes replacing exact matches.Activities are marked with (A), and correct results are shown in bold.
- Fine-grained recognition: Hand-centric features benefit fine-grained activity and participating-object recognition because relevant motions and objects occur near the hands.The evaluation compares pose-based, hand-centric, and holistic approaches on classification and detection.
- Composite recognition: Attribute-based recognition addresses composite-activity variability by sharing information across activities and using textual scripts when training data is scarce.The approach supports recognition of novel composites without training data for those composites.
- Future work: Future work will validate the hand-centric approach in other domains and model the temporal structure of videos when exploiting scripts.These extensions are identified as planned rather than completed contributions.