Source-linked AI summary
Actor and Observer: Joint Modeling of First and Third-Person Videos
Gunnar A. Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, Karteek Alahari
TL;DR
Egocentric action recognition lacks the aligned, diverse first- and third-person data needed to connect observer and actor perspectives. The paper introduces Charades-Ego and a weakly supervised joint representation, then demonstrates transfer from third-person to first-person recognition, while relying on simplifying selector assumptions.
Problem
Egocentric action understanding lacks sufficiently diverse, semantically aligned first- and third-person video pairs for learning a connection between the two domains.
Method
The paper collects Charades-Ego and learns a coordinated joint representation with weak supervision, using a selector distribution to identify useful frames and moments.
Results
ActorObserverNet improves over the state-of-the-art two-stream network by 3.2% in zero-shot first-person action recognition and transfers knowledge from third-person videos.
Takeaways & Limitations
Paired first- and third-person data can support joint modeling and practical transfer of third-person knowledge to the first-person domain.
Takeaways & Limitations
The selector assumes factorization into independent frame-selection distributions and normalizes selection probabilities within each video.
Abstract
from arXiv · showhide
Several theories in cognitive neuroscience suggest that when people interact with the world, or simulate interactions, they do so from a first-person egocentric perspective, and seamlessly transfer knowledge between third-person (observer) and first-person (actor). Despite this, learning such models for human action recognition has not been achievable due to the lack of data. This paper takes a step in this direction, with the introduction of Charades-Ego, a large-scale dataset of paired first-person and third-person videos, involving 112 people, with 4000 paired videos. This enables learning the link between the two, actor and observer perspectives. Thereby, we address one of the biggest bottlenecks facing egocentric vision research, providing a link from first-person to the abundant third-person data on the web. We use this data to learn a joint representation of first and third-person videos, with only weak supervision, and show its effectiveness for transferring knowledge from the third-person to the first-person domain.
1. Introduction
The paper argues that humanlike action understanding may require connecting first-person and third-person perspectives, but progress is limited by scarce aligned, diverse data. It introduces Charades-Ego and ActorObserverNet to learn this connection and transfer third-person knowledge to first-person recognition.
- Motivation: Human action understanding may involve simulating goals and intentions from an egocentric perspective while connecting observer and actor views.The motivation draws on Theory of Mind and neuroscience findings involving mirror neurons and motor regions during passive observation.
- Motivation: First-person and third-person videos represent related views of action, yet existing datasets leave them poorly connected.This connection could leverage abundant third-person videos for egocentric understanding.
- Data gap: Aligned, diverse first- and third-person videos are scarce because egocentric datasets are difficult to collect and often contain few subjects.The paper identifies aligned data and actor/action diversity as requirements for generalization.
- Contribution: Charades-Ego contains semantically aligned first- and third-person videos from 112 actors performing 157 action types.The same actors perform the same action sequences from both perspectives, enabling joint modeling.
- Contribution: ActorObserverNet learns a weakly supervised joint embedding across the two domains and transfers third-person knowledge to zero-shot first-person action recognition.The contribution targets a major bottleneck in egocentric vision research.
2. Charades-Ego
Charades-Ego is designed to scale collection of semantically aligned first- and third-person videos. It uses scripted recordings from the same actors and provides 4,000 paired videos across diverse settings, with measured but imperfect temporal alignment.
- Dataset motivation: Charades-Ego addresses the need for scalable, semantically aligned videos recorded from both first- and third-person views.Controlled lab collection is difficult to scale, while diverse egocentric collection also faces privacy concerns.
- Collection: Workers record the same Charades script twice: once from third person and once with a forehead-mounted camera.The collection uses publicly available Charades scripts and covers all 157 activity classes sufficiently often.
- Dataset scale: 4,000 paired videos comprise 8,000 total videos, averaging 31.2 seconds and depicting the same activities, actors, environments, and styles.The paired videos are semantically similar rather than identical, encouraging models to use scene semantics.
- Dataset diversity: The dataset includes 112 actors across many rooms and is presented as a large-scale source of paired first- and third-person views.The paper states that the data is publicly available.
- Alignment: Human-identified shared moments have 1.3 seconds median alignment error and 2.1 seconds average error.This provides a compromise between synchronized laboratory recording and scalable collection.
3. Jointly Modeling First and Third-Person
The paper learns a shared first- and third-person representation by aligning corresponding frames and contrasting them with noncorresponding frames. A learned selector weights informative triplets, enabling the joint model to retrieve or hallucinate the other viewpoint.
- 3.1. Formulation: Corresponding first- and third-person frames are mapped close together in a shared representation, while noncorresponding frames are pushed apart.Ground-truth alignment uses temporal proximity, with nonalignment defined beyond a separate time margin.
- 3.1. Formulation: The objective uses triplets of one third-person frame, one corresponding first-person frame, and one noncorresponding first-person frame.The loss is defined over triplets (x,z,z′) from the two modalities and is applied on top of ConvNet outputs.
- 3.1. Formulation: A learned selector Pθ samples more informative triplets by weighting their contribution to the loss, while constraints prevent concentrating on a single sample.Importance sampling rewrites the objective using a fixed proposal distribution Q, and the selector factorizes across the three frames.
- 3.1. Formulation: The joint model retrieves the closest first-person frames for a third-person query and can generate the other viewpoint from either input perspective.Figure 3 illustrates nearest-neighbor retrieval and observer-view prediction from a first-person frame.
- 3.3. Architecture of ActorObserverNet: ActorObserverNet uses one third-person stream and two partly shared first-person streams, combining them with an L2 distance that separates corresponding and noncorresponding samples.The architecture also computes selector values through fully connected layers and VideoSoftmax normalization across frames from the same video.
4. Experiments
The experiments evaluate ActorObserverNet for cross-view correspondence, temporal alignment, and zero-shot transfer from third-person to first-person videos. Learned frame weighting improves correspondence and qualitative selection emphasizes informative hands, objects, and action frames.
- Implementation details: ActorObserverNet shares parameters between first- and third-person ResNet-152 streams pretrained for Charades-based frame classification.Training samples positive temporally corresponding pairs and unrelated first-person negatives using a uniform distribution.
- Mapping third-person to first-person: The correspondence experiment evaluates finding a matching first-person frame for each third-person frame using all test data or top-weighted subsets.Top 5%, 10%, and 50% highest-weight triplets are evaluated in the selective setting.
- Mapping third-person to first-person: ActorObserverNet with learned frame weighting significantly outperforms competing methods, whereas unfiltered evaluation remains only slightly above random chance.The model assigns each image frame an independent weight to identify good mapping candidates, with similar behavior for different-person videos.
- Mapping third-person to first-person: 68% of high-ranked frames contain hands, compared with 15% of low-ranked frames.Activations and image-layer gradients show attention to hands, objects, and field of view; high selector values correspond to useful moments such as hands manipulating objects.
- Alignment and localization: The temporal-alignment experiment selects one-second moments from shuffled first- and third-person videos and evaluates their median alignment error in seconds.ActorObserverNet outperforms ImageNet ResNet-152 and Charades-trained Two-Stream baselines.
- Zero-shot first-person action recognition: In zero-shot first-person action recognition, ActorObserverNet improves over the Charades-trained two-stream state of the art by 3.2%.On third-person Charades videos, it reaches 23.5%, only 0.7% above the original model, suggesting the gain is mainly associated with third-to-first-person transfer.
5. Summary
The paper links first- and third-person activity recognition through paired videos and a joint representation, with benefits demonstrated across several cross-perspective tasks. It acknowledges the contributions and support underlying this research.
- The framework links first- and third-person worlds through the Charades-Ego dataset of paired videos.The model jointly represents the two domains using a robust triplet loss.
- Semantic equivalence across perspectives allows the model to relate videos from different people.
- Benefits are demonstrated in third-to-first-person mapping, cross-domain video alignment, and zero-shot first-person action recognition.
- The work received support from Intel, Sloan Fellowship, Inria, the ERC, Amazon, and CEFIPRA, with additional assistance from colleagues and Mechanical Turk workers.
6. Supplementary Material
The supplementary material documents implementation details for the model, its VideoSoftmax and loss updates, training setup, and derivative calculations. It explains how normalization and sampling are handled online across batches and frames.
- The supplement describes implementation details for new layers, ActorObserverNet, and the derivation of losses used to update the selector.
- VideoSoftmax is implemented with explicit normalization and an online denominator update for video-level terms.The normalization constant is approximated using sampled terms and a fixed k=0.1 scaling.
- The loss-layer derivation uses importance sampling over triplets and produces an online update compatible with SGD.The derivation separates the current sample from other samples while estimating the normalization term from samples drawn from Q.
- Training details include toggled classification losses, batchsize 15, bounded selector values, and parameter sharing between first-person streams.
- Triplets pair each third-person frame with an estimated corresponding frame and a randomly sampled noncorresponding frame at least 10 seconds away.
- The derivative accounts for each sample’s occurrence in both the numerator and the VideoSoftmax denominator, including contributions from other frames.The final expressions are factorized and implemented online as the paper’s update equation.