Source-linked AI summary
Going Deeper into First-Person Activity Recognition
Minghuang Ma, Haoqi Fan, Kris M. Kitani
TL;DR
Egocentric activity recognition requires combining appearance and motion cues centered on hand-object interactions. The paper introduces a twin-stream CNN with hand-guided object localization and joint action, object, and activity learning, reporting improvements over state-of-the-art methods and visual evidence of the intended features.
Problem
The problem is to recognize first-person actions using appearance and motion cues such as hand appearance, object attributes, local hand motion, and camera ego-motion.
Method
The method is a twin-stream CNN combining an appearance stream with hand segmentation and object localization and a motion stream using optical-flow inputs.
Results
The method reports over 6.6% higher egocentric activity-recognition accuracy, with 30% action and 14% object recognition improvements through joint learning.
Takeaways & Limitations
The learned representations capture hand-object configurations, object attributes, and hand-motion signatures isolated from global motion.
Takeaways & Limitations
The object-localization approach assumes the object of interest is frequently near the hands, rather than always centered in the camera image.
Abstract
from arXiv · showhide
We bring together ideas from recent work on feature design for egocentric action recognition under one framework by exploring the use of deep convolutional neural networks (CNN). Recent work has shown that features such as hand appearance, object attributes, local hand motion and camera ego-motion are important for characterizing first-person actions. To integrate these ideas under one framework, we propose a twin stream network architecture, where one stream analyzes appearance information and the other stream analyzes motion information. Our appearance stream encodes prior knowledge of the egocentric paradigm by explicitly training the network to segment hands and localize objects. By visualizing certain neuron activation of our network, we show that our proposed architecture naturally learns features that capture object attributes and hand-object configurations. Our extensive experiments on benchmark egocentric action datasets show that our deep architecture enables recognition rates that significantly outperform state-of-the-art techniques -- an average $6.6\%$ increase in accuracy over all datasets. Furthermore, by learning to recognize objects, actions and activities jointly, the performance of individual recognition tasks also increase by $30\%$ (actions) and $14\%$ (objects). We also include the results of extensive ablative analysis to highlight the importance of network design decisions..
1. Introduction
The paper proposes a twin-stream CNN for egocentric activity recognition that jointly integrates appearance and motion, using hand-object cues, object attributes, and motion signatures. Joint multi-task learning improves activity, action, and object recognition while visualizations reveal the learned features.
- A twin-stream CNN integrates appearance and motion information for recognizing egocentric objects, actions, and activities.The appearance stream analyzes hand configurations and object attributes, while the motion stream analyzes object and head movement.
- The appearance stream explicitly trains hand segmentation and object localization to encode hand-object configurations and object attributes.Object images are cropped using hand-based localization before object recognition.
- The motion stream uses optical-flow fields and learns which camera, object, or hand motion patterns discriminate action types.It avoids camera ego-motion compensation as a preprocessing step.
- Over 6.6% higher accuracy on egocentric activity recognition and 30% and 14% improvements for action and object recognition, respectively, are reported through joint learning.The activity comparison is against state-of-the-art techniques without gaze information.
- Neuron visualizations show learned hand-object configurations, object attributes, and hand-motion signatures isolated from global motion.These visualizations are used to analyze the features learned by the network.
- The paper also provides ablative analysis of design choices and visualizes intermediate-layer representations.These analyses are presented as contributions for understanding component effects and learned features.
2. Egocentric Activity Deep Network
The proposed egocentric activity network decomposes video understanding into appearance and motion streams, localizes manipulated objects near hands, and combines learned representations for joint prediction. Its pipeline trains hand segmentation, object localization, object recognition, motion recognition, and activity prediction as connected stages.
- ObjectNet: Recognizing Objects from Appearance: The localization network is fine-tuned from hand segmentation using images and Gaussian heatmaps synthesized from annotated object locations.The synthesized heatmaps place a Gaussian bump at each object location.
- ObjectNet: Recognizing Objects from Appearance: ObjectNet localizes and recognizes the manipulated object because detecting every scene object provides limited information about the object of interest.The object of interest is frequently near the hands, though head motion means it need not be centered in the image.
- ObjectNet: Recognizing Objects from Appearance: Hand segmentation uses images and binary hand masks to produce a hand probability map for object localization.Training may use manually annotated masks or outputs from pixel-level hand detectors.
- ObjectNet: Recognizing Objects from Appearance: Localization outputs are thresholded, and the centroid of the largest blob determines the predicted object center for cropping.The resulting cropped object sequences are used for object recognition.
- Motion stream: ObjectNet recognizes objects from cropped object images, while the motion stream uses optical-flow fields to predict action labels.The motion stream represents temporal information by stacking fixed-length optical-flow frames.
- Activity recognition: The activity network combines the two streams because separate feature extraction ignores correlations among actions, objects, and activities.The activity label is predicted from representations learned by the object and action networks.
3. Experiments
Experiments on three egocentric datasets evaluate object, action, and activity recognition using hand-guided localization, appearance and motion streams, and joint training. The visualizations and comparisons show improved recognition and learned representations of hands, objects, motion, and temporal patterns.
- Datasets and setup: Experiments use GTEA, Gaze, and Gaze+ datasets, with leave-one-subject-out cross-validation and fixed-split comparisons.The datasets contain head-mounted-camera recordings dominated by hand-object interactions.
- ObjectNet Performance: The ObjectNet stream trains hand segmentation before object localization, then uses localized object images for object recognition.Hand-based pre-training biases localization toward the object of interest and supports different one-hand or two-hand configurations.
- ObjectNet Performance: 14% better object recognition accuracy is reported for the proposed method than the baseline.The paper attributes the gain to improved localization through hand-segmentation-based pre-training, while joint training improves accuracy across datasets.
- ObjectNet Performance: Object-network activations respond to hand regions and object attributes such as color, texture, shape, bottles, containers, cups, and round forms.These visualizations indicate that hand appearance and object attributes contribute to object recognition.
- ActionNet performance: 30% better action recognition accuracy is reported for the proposed method than the baseline.The action experiments use stacked optical flow fields, with L = 10 optical-flow frames selected empirically for good performance.
- ActionNet performance: Motion-network visualizations show foreground-motion selection and temporal-pattern encoding, including strong responses to put actions and weaker responses when optical-flow order is reversed.The learned motion representation distinguishes object and hand motion from camera motion and preserves temporal ordering relevant to put versus take.
- Activity recognition: Joint training of action, object, and activity networks boosts performance consistently by 27% across datasets.The authors report that joint optimization avoids over-fitting newly added top layers and produces a joint activity representation.
4. Conclusion
The twin-stream CNN integrates features characterizing egocentric activities and jointly recognizes actions, objects, and activities. Evaluation on three public datasets showed significant outperformance of state-of-the-art methods, while visualizations revealed hand, object, and motion cues.
- The twin-stream CNN integrates features that characterize egocentric activities.
- The network jointly learns to recognize actions, objects, and activities.
- The model significantly outperformed state-of-the-art methods on three public datasets.
- Visualizations showed learned cues including hand appearance, object attributes, local hand motion, and global ego-motion.