Source-linked AI summary
Future Person Localization in First-Person Videos
Takuma Yagi, Karttikeya Mangalam, Ryo Yonetani, Yoichi Sato
TL;DR
The paper introduces future person localization: predicting where people observed in first-person videos will appear in future frames, a task relevant to navigation around nearby people. It combines ego-motion, scale, pose, and location histories in a multi-stream convolution-deconvolution network, which outperforms baseline methods on a new dataset and performs promisingly on a public dataset.
Problem
The paper addresses the task of predicting future locations of people observed in first-person videos to support navigation in crowded places.
Method
A multi-stream convolution-deconvolution network uses histories of person locations, scales, poses, and camera-wearer ego-motion to predict subsequent locations.
Results
The method outperformed state-of-the-art static-camera trajectory methods on the FPL Dataset and showed promising performance on a public first-person video dataset.
Takeaways & Limitations
Ego-motion, person scales, and poses were all necessary ingredients for accurately predicting target-person locations in future first-person frames.
Takeaways & Limitations
The paper identifies future localization of camera wearers as a possible extension that could improve prediction of observed people.
Abstract
from arXiv · showhide
We present a new task that predicts future locations of people observed in first-person videos. Consider a first-person video stream continuously recorded by a wearable camera. Given a short clip of a person that is extracted from the complete stream, we aim to predict that person's location in future frames. To facilitate this future person localization ability, we make the following three key observations: a) First-person videos typically involve significant ego-motion which greatly affects the location of the target person in future frames; b) Scales of the target person act as a salient cue to estimate a perspective effect in first-person videos; c) First-person videos often capture people up-close, making it easier to leverage target poses (e.g., where they look) for predicting their future locations. We incorporate these three observations into a prediction framework with a multi-stream convolution-deconvolution architecture. Experimental results reveal our method to be effective on our new dataset as well as on a public social interaction dataset.
1. Introduction
This work introduces future person localization for first-person videos, motivated by safe navigation among nearby people. It uses ego-motion, person scale, pose, and temporal observations to predict future locations, and validates the approach on a new dataset and a public dataset.
- Task and motivation: Future person localization predicts where observed people will appear in subsequent frames from first-person video observations.The task is proposed as a step toward guiding users around people and avoiding collisions in crowded places.
- Key observations: Camera-wearer ego-motion changes target locations and can also influence target walking directions through collision-avoidance interactions.Forward camera motion can make apparent person locations move downward, while approaching people may alter their paths.
- Key observations: Person scale provides a cue for perspective effects and physical distance, especially when people move toward or away from the camera wearer.The wearable camera’s optical axis tends to be parallel to the ground plane, so image position alone does not determine physical distance.
- Key observations: Person pose indicates likely near-future movement, and first-person videos often capture people close enough to expose pose information.The paper highlights pose cues such as where a person looks.
- Approach: A deep convolution-deconvolution network learns histories of ego-motion, poses, scales, and locations to predict subsequent target locations.The method encodes temporal evolution from present and past frames using a multi-cue architecture.
- Evaluation: The FPL Dataset contains about 5,000 people, and the method outperformed static-camera trajectory methods while also performing promisingly on a public first-person dataset.The reported evaluation includes diverse places and a public social interaction dataset.
2. Related Work
The paper positions future person localization as a distinct first-person prediction task related to, but different from, prior egocentric localization and human trajectory forecasting. Its single-camera setting targets assistive scenarios where synchronized multi-camera capture is impractical.
- Task distinction: The work claims to be the first to predict future locations of people in first-person videos.It distinguishes this task from predicting the future locations of camera wearers.
- Task distinction: Egocentric future localization predicts where camera wearers will be located, whereas this task predicts locations of other observed people.The two tasks differ in their prediction targets.
- Task distinction: Prior basketball behavior prediction required multiple synchronized first-person cameras to reconstruct accurate 3D configurations.That requirement contrasts with this work’s use of a single camera and is described as unsuitable for assistive scenarios where only the user wears a camera.
- Trajectory forecasting: Human trajectory prediction has used inverse reinforcement learning, Bayesian methods, recurrent networks, fully convolutional networks, and social or contextual features.These approaches generally address future trajectories from images or videos and form the broader related-work context.
3. Proposed Method
The proposed method predicts relative future person locations from recent first-person observations using location, scale, ego-motion, and pose cues. A multi-stream convolution-deconvolution network encodes these temporal signals and decodes their joint representation into future locations.
- Problem formulation: Given observations from Tprev previous frames, the task predicts a target person’s relative locations in the next Tfuture frames.The output is measured relative to the person’s location in the current frame.
- Overview: The method uses target locations, scales, camera-wearer ego-motion, and target poses as prediction cues.These cues are identified as salient observations for estimating future relative locations.
- Architecture: A multi-stream convolution-deconvolution architecture learns temporal cue sequences, concatenates their features, and deconvolves them into future locations.The network is trained end-to-end through back-propagation.
- Location and scale cue: Jointly modeling locations and scales addresses perspective effects because scale provides a rough estimate of physical movement under a similar-person-height assumption.The locationscale stream extends each 2D location with the person’s scale and learns its temporal evolution.
- Ego-motion cue: Ego-motion is needed because camera movement changes the coordinate system and creates diverse relationships between observed past and future locations.Interactive changes in walking speed or direction can further complicate prediction when people approach one another.
- Ego-motion cue: The ego-motion stream represents accumulated camera rotation and translation histories, with rotation vectors and translation features forming per-frame inputs.The method estimates motion between successive frames and accumulates it over the input interval.
- Pose cue: The pose stream tracks temporal changes in body-part locations, using parts such as eyes, shoulders, and hips as indicators of movement direction.First-person views often capture people up-close, making these pose observations accessible.
4. Experiments
Experiments evaluate the method on the new FPL Dataset and a public first-person social interaction dataset, using tracked person cues and multi-stream temporal modeling. The method outperforms baselines, with errors analyzed quantitatively, qualitatively, and through cue ablations.
- Dataset: The FPL Dataset contains about 4.5 hours of chest-mounted-camera video from diverse environments, with more than 5,000 observed people.Samples were generated from tracked people and extracted locations, scales, poses, and camera ego-motion.
- Evaluation Protocol: Training and testing use ten observed frames and ten future frames at 10 fps, with five-fold splits drawn from different videos.Each sample includes location-scale, ego-motion, pose, and future relative location-scale information.
- Architecture and Training: The network uses separate temporal convolution streams for location-scale, ego-motion, and pose cues, followed by concatenation and temporal deconvolution.Inputs are normalized, horizontally augmented, and optimized with MSE using Adam.
- Quantitative Results: The method significantly outperforms baselines on FPL, while all methods achieve FDE below about 15% of frame width and Social LSTM performs poorly.The authors attribute temporal-model instability to failing to account explicitly for substantial camera ego-motion.
- Error Analysis: With the method, 73% of samples have error below 100 pixels and only 1.4% exceed 300 pixels; average scale-normalized error corresponds to about 60 cm physically.Qualitative examples show failures under sudden unexpected camera motion and successful predictions when posture changes provide useful information.
- Ablation and Generalization: Ablations show that locations, scales, ego-motion, and poses each improve prediction, with especially large gains from scales and poses on the Toward subset.The method also outperforms baselines, including Social LSTM, on the public First-Person Social Interaction Dataset.
- Limitations and Extensions: Sudden unexpected ego-motion remains a failure case, motivating future prediction of camera-wearer movements.Flow-based ego-motion features produced a 1.89 FDE drop on FPL relative to the original result, while potentially capturing yaw rotations more accurately.
5. Conclusion
The paper introduces future person localization for first-person videos and identifies camera-wearer ego-motion, target scales, and poses as necessary ingredients for predicting future appearances. It proposes incorporating future camera-wearer localization as an extension to improve predictions further.
- Future person localization predicts where observed people will appear in future frames of first-person videos.
- Experimental results identify camera-wearer ego-motion, target scales, and target poses as necessary ingredients for accurate future localization.
- A proposed extension is to incorporate future localization of camera wearers to improve predictions of observed people’s future locations.
A. Data Statistics
The experiments use tracklets from the First-Person Locomotion and Social Interaction datasets, with clips categorized by normalized body scale and directional labels. Tables report final displacement error for feature combinations and input/output-length configurations.
- Data Statistics: Tracklet selection required sufficient length for meaningful temporal dynamics and frequent observation for stable network learning, producing 50,000 samples.
- Data Statistics: Clips were categorized as Across when mean scale-normalized hip distance was below 0.25; remaining clips used framewise Toward and Away labels.
- Data Statistics: The reported tables evaluate final displacement error across input features and input/output-length combinations, including two-second futures.
B.1. Other Choices of Input/Output Lengths
Longer observation histories improved performance, whereas predicting more distant futures was more difficult. The method also remained effective when predicting two-second futures after changing the deconvolution kernel sizes.
- Longer input histories led to better performance, while longer prediction horizons made future localization more difficult.The fixed setting was Tprev = Tfuture = 10; comparisons included Tprev = 6 versus 10 and Tfuture = 10 versus 6.
- The model was evaluated on two-second futures with Tfuture = 20 and still worked well under this challenging condition.The deconvolution kernel sizes were changed from 3, 3, 3, 3 to 3, 5, 7, 7.
B.2. Other Visual Examples
Figure 9 provides additional visual examples comparing the proposed method with several baselines on future person localization.
- Figure 9 shows additional visual examples of predictions from the proposed method and several baselines.
B.3. Ablation Study on Social Interaction Dataset
On the Social Interaction Dataset, combining ego-motion and pose cues improved future-person-localization performance.
- Combining ego-motion and pose cues improved performance on the Social Interaction Dataset.