Source-linked AI summary
Pooled Motion Features for First-Person Videos
M. S. Ryoo, Brandon Rothrock, Larry Matthies
TL;DR
First-person activity recognition needs representations that capture both egomotion and salient local motion, but motion features for this setting have been limited. The paper introduces PoT, which pools temporal changes in per-frame descriptors, and reports that it outperforms prior representations and state-of-the-art video features across first-person datasets.
Problem
Motion features for first-person activity recognition have been limited, although capturing egomotion is important for recognizing actions and interactions.
Method
PoT treats each element of per-frame appearance or motion descriptors as a time series and summarizes its short-term and long-term changes using temporal filters and pooling operators.
Results
PoT clearly outperforms BoW and IFV as well as other state-of-the-art video features on first-person activity recognition.
Takeaways & Limitations
A general representation using temporal changes in high-dimensional descriptors can capture entire-scene dynamics and local motion for first-person videos.
Takeaways & Limitations
On temporally misaligned UEC Park segments, temporal pooling structures harm recognition, so results are shown without temporal structure and PoT is disadvantaged.
Abstract
from arXiv · showhide
In this paper, we present a new feature representation for first-person videos. In first-person video understanding (e.g., activity recognition), it is very important to capture both entire scene dynamics (i.e., egomotion) and salient local motion observed in videos. We describe a representation framework based on time series pooling, which is designed to abstract short-term/long-term changes in feature descriptor elements. The idea is to keep track of how descriptor values are changing over time and summarize them to represent motion in the activity video. The framework is general, handling any types of per-frame feature descriptors including conventional motion descriptors like histogram of optical flows (HOF) as well as appearance descriptors from more recent convolutional neural networks (CNN). We experimentally confirm that our approach clearly outperforms previous feature representations including bag-of-visual-words and improved Fisher vector (IFV) when using identical underlying feature descriptors. We also confirm that our feature representation has superior performance to existing state-of-the-art features like local spatio-temporal features and Improved Trajectory Features (originally developed for 3rd-person videos) when handling first-person videos. Multiple first-person activity datasets were tested under various settings to confirm these findings.
1. Introduction
First-person videos differ from conventional third-person footage because the camera wearer participates in the recorded events, producing strong egomotion and distinctive motion dynamics. The paper introduces pooled time series (PoT) to capture these dynamics and reports superior recognition performance over prior representations and third-person motion features.
- Motivation: First-person videos are recorded from an actor’s viewpoint, with the camera wearer actively involved in the events and strong egomotion.These properties distinguish them from conventional third-person videos.
- Motivation: Motion features for first-person activity recognition have been relatively limited despite their importance for visual grounding of actions and events.Prior work included first-person-specific semantic features such as hand locations and gaze, but motion representations remained lacking.
- Proposed representation: PoT represents short-term and long-term changes in per-frame feature descriptors to capture detailed entire-scene dynamics in first-person videos.The framework can use descriptors such as HOF or CNN features and combines pooling operators with temporal filters.
- Proposed representation: PoT captures both global and local motion by using different descriptor types and tracking detailed descriptor changes while suppressing noise.This design targets recognition of ego-actions and interactions.
- Results: PoT clearly outperforms bag-of-visual-words, improved Fisher vector, and the best-known motion feature designed for third-person videos on first-person activity recognition.The comparison uses identical underlying descriptors where applicable and also evaluates local spatio-temporal and trajectory-based features.
2. Pooled times series representation
PoT represents a video by treating each descriptor dimension as a time series, pooling its temporal behavior across multiple intervals and operators, and concatenating the results into one vector. This design preserves short- and long-term dynamics, temporal structure, and multiple aspects of motion while supporting diverse descriptors.
- Framework: PoT converts per-frame descriptor vectors into time series and applies temporal pooling to produce one tractable video representation for classifiers.Each descriptor element becomes a function of time; pooling results are concatenated into the final vector.
- Temporal structure: Multiple temporal filters let PoT capture subtle short-term motion and longer-term motion within the same representation.A temporal pyramid supplies local time intervals, although the framework can use overlapping filters more generally.
- Design abilities: The framework preserves descriptor-element dynamics, imposes temporal structure through subintervals, and combines multiple pooling operators to capture different data aspects.These properties distinguish PoT from bag-of-visual-words, which discretizes descriptor values from individual frames or short subsequences.
- Descriptor generality: PoT supports per-frame motion and appearance descriptors, including HOG, HOF, and high-dimensional CNN features whose values track changing local or global appearance.The framework was designed to exploit high-dimensional descriptors while modeling how their elements change as the camera and scene evolve.
- Pooling operators: Four pooling operators combine conventional max and sum pooling with histogram-of-time-series-gradients pooling that captures positive and negative temporal changes.The gradient operators can count positive or negative gradients, or sum their magnitudes; each generates a pair of values.
3. Experiments
Experiments on DogCentric and UEC Park evaluate PoT with motion and CNN appearance descriptors against conventional representations and state-of-the-art features. PoT generally performs best, especially when temporal structure and complementary descriptors are used.
- Experimental setup: Experiments used DogCentric and UEC Park first-person datasets, with HOF, MBH, Overfeat CNN, and Caffe CNN per-frame descriptors and nonlinear χ2-kernel SVMs.DogCentric evaluation averaged 100 repeated random half-training/half-testing splits.
- Feature representation evaluation: 0.730 mean accuracy with PoT exceeded 0.666 for combined IFV with temporal pyramid and 0.605 for the previous state of the art on DogCentric.PoT outperformed BoW and IFV in all descriptor settings except MBH, where representations performed similarly.
- Feature representation evaluation: 4096-dimensional CNN descriptors showed particularly strong gains with PoT over BoW and IFV, consistent with PoT preserving subtle descriptor changes that per-frame abstraction can ignore.The paper attributes this advantage to time-series pooling of descriptor-element changes rather than discretization into visual words or soft assignments.
- Comparison to state-of-the-art features: PoT outperformed ITF on DogCentric, including 0.730 versus 0.676 for ITF with temporal pyramid and 0.692 for combined ITF plus mean per-frame CNN.ITF performed slightly better for ‘turn head’ classes, whose videos involved less camera motion and were more similar to third-person videos.
- Cross-dataset and descriptor analysis: PoT achieved the best performance on UEC Park, and combining PoT with ITF increased performance slightly further; CNN descriptors also significantly outperformed HOG under identical PoT settings.CNN appearance descriptors captured information different from motion descriptors, with effectiveness depending on the dataset and activities.
4. Conclusion
The paper introduces pooled time series (PoT), a feature representation for first-person videos that captures both entire-scene dynamics and local motion through long- and short-term changes in high-dimensional descriptors. Across two public datasets, PoT outperforms BoW, IFV, and other state-of-the-art video features.
- PoT represents long-term and short-term changes in high-dimensional feature descriptors to capture entire-scene dynamics and local motion.
- PoT was combined with four types of per-frame descriptors, including CNN features.
- PoT was evaluated on two public first-person video datasets.
- PoT clearly outperforms BoW, IFV, and other state-of-the-art video features.