Source-linked AI summary
Predicting Gaze in Egocentric Video by Learning Task-dependent Attention Transition
Yifei Huang, Minjie Cai, Zhenqiang Li, Yoichi Sato
TL;DR
Gaze prediction in natural egocentric video is difficult because task context shapes attention beyond visual saliency alone. The paper learns task-dependent attention transitions and combines them with CNN-based saliency, achieving state-of-the-art performance on two public datasets. Its broader task-domain generalizability remains future work.
Problem
Gaze prediction in natural dynamic egocentric scenes is difficult because task knowledge strongly influences attention beyond saliency-based predictions.
Method
A hybrid deep model combines CNN-based bottom-up saliency with recurrent task-dependent attention transitions learned from previous gaze fixations and head motion.
Results
The model achieves state-of-the-art gaze prediction performance on the GTEA Gaze and GTEA Gaze Plus public egocentric datasets.
Takeaways & Limitations
Bottom-up visual saliency and high-level task-dependent attention are complementary cues for modeling human attention in egocentric videos.
Takeaways & Limitations
The paper leaves generalizability across broader task domains, including office and manufacturing tasks, for future work.
Abstract
from arXiv · showhide
We present a new computational model for gaze prediction in egocentric videos by exploring patterns in temporal shift of gaze fixations (attention transition) that are dependent on egocentric manipulation tasks. Our assumption is that the high-level context of how a task is completed in a certain way has a strong influence on attention transition and should be modeled for gaze prediction in natural dynamic scenes. Specifically, we propose a hybrid model based on deep neural networks which integrates task-dependent attention transition with bottom-up saliency prediction. In particular, the task-dependent attention transition is learned with a recurrent neural network to exploit the temporal context of gaze fixations, e.g. looking at a cup after moving gaze away from a grasped bottle. Experiments on public egocentric activity datasets show that our model significantly outperforms state-of-the-art gaze prediction methods and is able to learn meaningful transition of human attention.
1 Introduction
Gaze prediction in natural egocentric scenes requires modeling task-dependent attention transitions alongside visual saliency. The proposed hybrid model learns these transitions and reports state-of-the-art performance on public egocentric activity datasets.
- Saliency-based gaze prediction becomes difficult in dynamic scenes because task knowledge influences human attention.
- Object-manipulation tasks produce successive gaze transitions tied to the task, such as shifting from a grasped bottle to a cup while pouring.
- The model combines bottom-up visual saliency with task-dependent attention transition learned from temporally successive fixations.
- The approach uses a two-stream CNN for saliency prediction and a recurrent attention-transition module conditioned on previous fixations and head motion.
- The proposed approach achieves state-of-the-art gaze prediction performance on public egocentric activity datasets.
2 Related Works
Prior work models gaze using visual saliency and egocentric cues, but does not capture attention-transition patterns inherent in complex tasks. This paper introduces a hybrid model that combines bottom-up saliency with task-dependent transitions.
- Visual Saliency Prediction: Visual saliency estimates image regions likely to attract attention and gaze fixation, using features such as color, intensity, and contrast.
- Visual Saliency Prediction: Deep CNN saliency models have incorporated increasingly higher-level context, extending earlier bottom-up approaches.
- Egocentric Gaze Prediction: Earlier methods did not explore attention-transition patterns inherent in complex tasks, motivating task-dependent learning of gaze shifts between objects or regions.
- Egocentric Gaze Prediction: Previous egocentric gaze methods use head motion, camera motion, hand motion, and hand positions to model gaze during activities.
- Egocentric Gaze Prediction: The proposed hybrid model combines bottom-up visual saliency with task-dependent attention transition for egocentric gaze prediction.
3 Gaze Prediction Model
The model combines bottom-up saliency with task-dependent attention transition to predict gaze in egocentric video. It uses spatial and temporal features, fixation history, recurrent channel-weight prediction, late fusion, and gaze-map decoding.
- Architecture: The hybrid model predicts a saliency map, an attention map from temporal gaze context, and a fused final gaze map.The saliency module processes video-frame features, while the attention transition module exploits previous gaze fixations.
- Feature Encoding: Spatial CNN and temporal CNN streams extract latent features from RGB frames and stacked optical flow images.The optical-flow interval is fixed at 10 frames.
- Saliency Prediction: The saliency decoder fuses the two feature streams and outputs a pixelwise visual saliency map with values in [0, 1].A 3D convolution and pooling operation fuse the input representations before decoding.
- Attention Transition Module: The attention transition module represents the previous gaze region as a 512-dimensional channel-weight vector extracted around the predicted gaze point.It crops and averages each channel of the 14 × 14 feature map around the projected gaze location.
- Attention Transition Module: An LSTM learns transitions between channel-weight vectors extracted at gaze-fixation boundaries and predicts the next attention representation.A fixation-state predictor estimates whether the previous frame contains a fixation, and the transition uses that probability to combine prior and anticipated attention.
- Fusion and Training: Late fusion combines the saliency map and attention-transition map into the predicted gaze map, whose maximum-value coordinate gives the 2D gaze position.The model trains gaze-map components with a distance-weighted loss and trains fixation prediction with binary cross-entropy and channel-weight prediction with mean squared error.
4 Experiments
The experiments evaluate gaze prediction on two public egocentric activity datasets and separately test learned attention transitions on a held-out subset containing future-attended regions. The datasets vary in subjects, activities, recording conditions, and evaluation design.
- Evaluation Setup: The study evaluates gaze prediction on the GTEA Gaze and GTEA Gaze Plus public egocentric activity datasets.It compares state-of-the-art methods, performs ablations and visualizations, and analyzes module outputs.
- Datasets: GTEA Gaze contains 17 kitchen-task videos from 14 subjects, with 15 fps recording and 480 × 640 image resolution.The paper uses videos 1, 4, and 6–22 for training and the remaining videos for testing.
- Datasets: GTEA Gaze Plus contains 37 videos in which 5 subjects perform 7 meal-preparation activities at 24 fps and 960 × 1280 resolution.Each clip lasts 10–15 minutes on average, and evaluation uses 5-fold cross-validation across subjects.
- Attention Transition Analysis: GTEA-sub contains 227 frames under gaze fixation that also show the object or region targeted at the next fixation.Bounding boxes are manually drawn from future frames to test whether a model trained on GTEA Gaze Plus learns task-dependent transitions.
4.2 Evaluation Metrics
The evaluation uses AUC and AAE as standard metrics for gaze prediction in egocentric video. AUC measures threshold-based discrimination on gaze maps, while AAE measures angular distance between predicted and ground-truth gaze positions.
- Metrics: AUC and AAE are the two standard metrics used to evaluate gaze prediction in egocentric videos.AUC is computed from true- and false-positive rates across predicted-map thresholds, whereas AAE measures angular distance between gaze positions.
- Metrics: AUC measures the area under the true-positive-rate versus false-positive-rate curve across thresholds on the predicted gaze map.It is commonly used in saliency prediction.
- Metrics: AAE measures the average angular distance between predicted and ground-truth gaze positions.
Baselines.
The comparison includes traditional and deep saliency models, a center-bias predictor, and prior gaze-prediction methods. The proposed method significantly outperforms all baselines on both datasets, especially on AAE.
- Baselines: The saliency baselines include Itti’s model, GBVS, and the deep neural network model SALICON.
- Baselines: The center-bias baseline predicts the image center, reflecting the strong center bias observed in egocentric gaze data.
- Baselines: The gaze-prediction baselines are Yin et al.’s egocentric cue-based method and DFG, a GAN-based method.DFG also reports current-frame gaze prediction although its primary goal is future-frame gaze anticipation.
- Results: The proposed method significantly outperforms all baselines on both datasets, particularly on AAE.The AUC improvement is small because DFG already achieves a high AUC, leaving limited room for improvement.
- Results: The center-bias predictor outperforms all saliency-based methods and is only slightly worse than Yin et al. on GTEA Gaze Plus.This result demonstrates a strong spatial bias in egocentric gaze data.
Performance Comparison.
The paper compares gaze-prediction methods on two public datasets and evaluates separate model components through ablation. The full model combines complementary saliency and task-dependent attention cues.
- Table 1 compares different gaze-prediction methods on two public datasets, with higher AUC or lower AAE indicating better performance.
- The ablation study evaluates single-stream and two-stream saliency variants, loss functions, the saliency prediction module, and the attention transition module.
- The modified binary cross entropy loss is more suitable for egocentric gaze prediction, while two-stream saliency benefits from both spatial and temporal information.
- The attention transition module performs competitively or better than saliency prediction, supporting task-dependent attention modeling.
- The full model outperforms its separate components by a large margin, indicating that bottom-up saliency and task-dependent attention provide complementary cues.
Visualization.
The cross-task validation examines whether task-dependent attention transition generalizes across seven meal-preparation tasks in the same kitchen scene.
- The evaluation uses cross validation across 7 different meal preparation tasks from the GTEA Gaze Plus dataset.
- The validation is conducted under a shared kitchen scene while testing generalization across tasks.
- The experiment is designed to examine how task-dependent attention transition generalizes to different tasks.
Cross Task Validation.
Cross-task validation compares saliency prediction, attention transition, and their late fusion across five settings. Performance trends show that task-specific transition modeling is stronger than cross-task transition alone, while fusion can recover gains.
- Five settings compare generic saliency prediction, task-specific or cross-task attention transition, and late fusion of saliency with each transition variant.
- Both AUC and AAE show the same performance trend across the different settings.
- AT d performs worse than SP, whereas AT s outperforms SP, likely reflecting differences in gaze behavior across tasks.
- SP+AT d improves performance over SP and AT s despite learning attention context from different tasks.
4.4 Examination of the attention transition module
The attention transition module is tested for meaningful prediction of the next fixation using latent convolutional representations. Its best representation substantially exceeds random-position performance.
- The experiment tests attention transition on the GTEA-sub dataset while focusing on the module’s ability to predict transitions between adjacent gaze fixations.
- Three latent representations from S-CNN convolutional layers conv5_1, conv5_2, and conv5_3 are compared.
- 86.8% accuracy from conv5_3 exceeds 83.0% from conv5_2, 71.7% from conv5_1, and 10.7% from random position.
- A random channel-weight baseline achieves 9.4% accuracy, while the learned module predicts the image region of the next fixation in qualitative results.
5 Conclusion and Future Work
The hybrid gaze prediction model combines task-dependent attention transitions with CNN-based saliency and achieves state-of-the-art performance on two public egocentric datasets. Future work will examine broader task settings and generalizability across domains.
- The model learns task-dependent attention transitions from previous fixations and integrates them with CNN-based bottom-up saliency cues.
- The proposed approach achieves state-of-the-art performance on two public egocentric datasets.
- Future work will explore tasks in offices and manufacturing factories and study generalizability across different task domains.