Source-linked AI summary
Connecting Gaze, Scene, and Attention: Generalized Attention Estimation via Joint Modeling of Gaze and Scene Saliency
Eunji Chong, Nataniel Ruiz, Yongxin Wang, Yun Zhang, Agata Rozga, James Rehg
TL;DR
The paper addresses visual-attention estimation beyond constrained settings by modeling gaze direction, scene saliency, fixation strength, and out-of-frame targets jointly. Using multi-task learning across three datasets, it achieves strong single-task benchmark results and promising generalized-attention performance in naturalistic social interactions.
Problem
Existing gaze and attention methods largely address constrained contexts, motivating generalized visual attention prediction across naturalistic scenarios, including out-of-frame gaze.
Method
A multi-task model uses whole images, face crops, and face locations to jointly estimate yaw and pitch, subject-dependent saliency, and fixation likelihood.
Results
The model achieves state-of-the-art GazeFollow performance, competes on EYEDIAP gaze estimation, and reports superior performance over baselines on MMDB generalized attention prediction.
Takeaways & Limitations
Jointly modeling gaze direction and scene saliency supports attention estimation across multiple naturalistic social scenarios, including attention outside the image frame.
Takeaways & Limitations
The method can mistake occluded in-frame targets and salient background objects because it lacks scene-depth understanding.
Abstract
from arXiv · showhide
This paper addresses the challenging problem of estimating the general visual attention of people in images. Our proposed method is designed to work across multiple naturalistic social scenarios and provides a full picture of the subject's attention and gaze. In contrast, earlier works on gaze and attention estimation have focused on constrained problems in more specific contexts. In particular, our model explicitly represents the gaze direction and handles out-of-frame gaze targets. We leverage three different datasets using a multi-task learning approach. We evaluate our method on widely used benchmarks for single-tasks such as gaze angle estimation and attention-within-an-image, as well as on the new challenging task of generalized visual attention prediction. In addition, we have created extended annotations for the MMDB and GazeFollow datasets which are used in our experiments, which we will publicly release.
1 Introduction
The paper formulates generalized visual attention prediction across naturalistic scenarios, including visible targets, out-of-frame gaze, and looking toward the camera. Its model jointly estimates gaze direction, subject-dependent saliency, fixation likelihood, and outside-frame targets using three datasets and multi-task learning.
- 1 Introduction: Generalized visual attention prediction covers visible targets, out-of-frame gaze, and subjects looking at or around the camera.Prior annotation setups often force fixation points inside the image, excluding some of these cases.
- 1 Introduction: The model jointly learns a subject-dependent saliency map and a 3D gaze vector represented by yaw and pitch to estimate fixation likelihood.The fixation map is designed to approach zero for out-of-frame gaze while predicting attended locations for visible targets.
- 1 Introduction: Three datasets provide complementary supervision: EYEDIAP for gaze angles, modified GazeFollow for gaze-relevant saliency, and SynHead for pose variation and outside-frame attention.The approach uses multi-task learning across datasets collected for different tasks.
- 1 Introduction: The method achieves state-of-the-art GazeFollow performance, competes on EYEDIAP gaze estimation, and reports superior baseline performance on generalized attention prediction in MMDB.MMDB provides a new naturalistic social-interaction evaluation with frame-level visual-target annotations.
2 Related work
The related work connects generalized attention prediction to gaze estimation, visual saliency, and gaze following while distinguishing the paper’s broader setting from constrained prior tasks.
- 2 Related work: Gaze estimation methods predict three-dimensional gaze direction or fixation locations, often in screen-based or tracking settings.The paper relates its work to third-person gaze estimation and tracking methods, including smartphone-screen gaze prediction and EYEDIAP.
- 2 Related work: Generalized attention prediction differs from visual saliency because it includes cases where the subject is not looking at any object inside the scene.Real-world scenes also permit a wider range of 3D gaze directions than screen-based eye tracking.
- 2 Related work: Gaze following predicts where a person in an image is looking, providing the basis for the paper’s scene-target prediction component.The paper extends this framing to attention scenarios that include gaze targets outside the image frame.
3 Method
The method jointly models gaze direction, person-dependent scene saliency, and fixation likelihood from whole-scene, face, and face-location inputs. It trains these components across complementary datasets because no single dataset provides all required supervision.
- The model estimates gaze angle, a subject-dependent saliency heatmap, and the likelihood that the subject fixates within the visible scene.Whole-image, cropped-face, and face-location inputs support the three outputs.
- Separate scene and face pathways learn saliency and gaze features, whose final representations are combined to estimate fixation strength.The fixation label is 1 for in-frame fixation and 0 for looking outside the scene.
- Cross-domain training selectively updates task-specific branches because no single dataset covers all gaze and scene combinations.The training sources are GazeFollow, EYEDIAP, and SynHead, each contributing different supervision.
- The project-and-compare loss constrains the projected image-plane gaze direction to match the vector from the head to the gaze target.The discrepancy between these vectors is measured with cosine distance.
- GazeFollow supplies in-image gaze targets and added inside/outside labels, while EYEDIAP supplies precise 3D gaze angles and SynHead broadens angle and background variation.GazeFollow adds 14,564 images labeled as looking outside, approximately 11.6% of its training samples.
4 Evaluation
The evaluation tests person-dependent saliency, 3D gaze estimation, and generalized attention prediction across GazeFollow, EYEDIAP, and MMDB, with additional diagnostics and qualitative analyses.
- 4.1 Person-Dependent Saliency Prediction: 0.896 AUC on GazeFollow establishes state-of-the-art gaze-saliency performance.The evaluation uses ROC AUC, with ground-truth target positions as labels and heatmap values as confidence scores.
- 4.2 Gaze Angle Prediction: Less than 0.5° separates the model from the state-of-the-art appearance-based method on EYEDIAP gaze-angle estimation.The comparison uses angular errors on the chosen EYEDIAP test split.
- 4.3 Generalized Attention Prediction During Naturalistic Social Interactions: Higher precision and increased recall are reported for generalized gaze-target grid classification on MMDB.The task assigns positive labels to the target grid square for in-frame fixations and all-zero labels for out-of-frame gaze.
- 4.3 Generalized Attention Prediction During Naturalistic Social Interactions: The model’s gaze-saliency heatmap is low for out-of-frame fixations and high when an in-frame target becomes clear.Qualitative MMDB examples cover both within-frame and out-of-frame gaze behavior.
- 4.4 Alternative Model and Diagnostics: Changing architecture substantially affects GazeFollow scores, whereas omitting EYEDIAP or SynHead has little impact on attention-within-image heatmaps.Additional experiments also examine training-dataset and architectural choices.
- 4.4 Alternative Model and Diagnostics: Occlusion and missing scene-depth understanding remain qualitative failure cases for fixation prediction.The model can select salient background objects when a closer subject or object should determine fixation.
5 Conclusion
The paper introduces generalized visual attention prediction and addresses it with a multi-task architecture trained across three datasets. It reports strong single-task results, promising MMDB performance, and publicly released additional annotations.
- 5 Conclusion: The paper frames generalized visual attention prediction as a unified problem spanning previously constrained attention and gaze-estimation tasks.The task targets multiple naturalistic social scenarios.
- 5 Conclusion: A multi-task architecture trained with three datasets addresses the problem across naturalistic social scenarios.The authors supplemented the datasets with new annotations and plan to release them publicly.
- 5 Conclusion: The model achieves state-of-the-art single-task gaze-saliency performance, competes on gaze-estimation benchmarks, and shows promising MMDB generalized-attention results.The conclusion presents these as the paper’s principal evaluation outcomes.