Source-linked AI summary
Detecting Attended Visual Targets in Video
Eunji Chong, Yongxin Wang, Nataniel Ruiz, James M. Rehg
TL;DR
The paper addresses the problem of estimating each person’s gaze target in every video frame, including targets outside the frame. It introduces a spatiotemporal architecture and the VideoAttentionTarget dataset, then applies predicted attention maps to social gaze recognition. The approach achieves state-of-the-art performance across the reported benchmark and social-gaze settings.
Problem
Existing naturalistic gaze measurement is limited by wearable-eye-tracker burdens, while video gaze estimation lacks suitable datasets for complex, time-varying gaze behavior.
Method
The method uses head features to regulate scene-region learning, adds ConvLSTM temporal modeling, and introduces the annotated VideoAttentionTarget dataset.
Results
The model achieves state-of-the-art performance on GazeFollow, VideoAttentionTarget, and VideoCoAtt, while its predicted heatmaps support two social gaze recognition tasks.
Takeaways & Limitations
Dense predicted attention maps provide a useful representation for gaze behavior recognition and for studying naturalistic human interactions.
Abstract
from arXiv · showhide
We address the problem of detecting attention targets in video. Our goal is to identify where each person in each frame of a video is looking, and correctly handle the case where the gaze target is out-of-frame. Our novel architecture models the dynamic interaction between the scene and head features and infers time-varying attention targets. We introduce a new annotated dataset, VideoAttentionTarget, containing complex and dynamic patterns of real-world gaze behavior. Our experiments show that our model can effectively infer dynamic attention in videos. In addition, we apply our predicted attention maps to two social gaze behavior recognition tasks, and show that the resulting classifiers significantly outperform existing methods. We achieve state-of-the-art performance on three datasets: GazeFollow (static images), VideoAttentionTarget (videos), and VideoCoAtt (videos), and obtain the first results for automatically classifying clinically-relevant gaze behavior without wearable cameras or eye trackers.
1. Introduction
The paper targets per-frame gaze estimation in video, including out-of-frame targets, using a spatiotemporal architecture and a new dataset of complex gaze behavior. Predicted attention maps also support social gaze recognition tasks.
- Motivation: Wearable eye trackers are burdensome and introduce calibration, compliance, cost, and battery-life challenges for naturalistic gaze collection.These constraints limit large-scale measurement in face-to-face interactions and 3D object manipulation.
- Problem: Gaze target prediction in video must handle dynamic, per-frame attention and gaze targets located outside the frame.The method aims to identify where each person looks in every frame while treating out-of-frame gaze targets correctly.
- Dataset: The VideoAttentionTarget dataset contains 1,331 video sequences with annotated dynamic gaze tracks in diverse situations.It addresses the lack of ground-truth annotations for complex, time-varying gaze behavior in real-world video.
- Architecture: The model combines a head-regulated scene attention mechanism for spatial localization with ConvLSTM networks for temporal modeling.The spatial component improves GazeFollow performance, while the temporal model is evaluated on VideoAttentionTarget.
- Applications: Predicted heatmaps achieve state-of-the-art results on two social gaze behavior recognition tasks.The tasks are attention-shift detection in young children and shared-attention detection in social scenes.
2. Related Work
Prior work estimates gaze targets in images or directly classifies gaze behaviors, while this paper focuses on dense, per-frame 2D gaze representations from video. Its approach adds temporal modeling and architectural changes to improve gaze prediction and support behavior classification.
- Gaze Target Prediction: Related gaze-target methods distinguish between 2D image targets and 3D-space targets; this work addresses the 2D case.Earlier 2D methods include extensions for out-of-frame gaze targets, while 3D methods use scene assumptions or task-informed learning.
- Architectural Differences: Compared with the closest formulation, the model removes gaze-angle supervision, uses head-regulated spatial pooling, and produces fine-grained heatmaps with deconvolutions.It also replaces one-hot position vectors with head location maps and adds temporal modeling.
- Dataset: The paper introduces VideoAttentionTarget to provide dense annotations for modeling time-varying gaze in rich, real-world video.Figure 2 depicts per-frame person bounding boxes, gaze targets, and distributions of target locations, gaze directions, and head sizes.
- Gaze Behavior Recognition: Direct gaze-behavior classifiers detect predefined patterns such as mutual gaze or shared attention, whereas this method predicts dense gaze targets for each person.The dense representation is intended as a complementary mid-level representation for later behavior classification.
- Social Gaze Applications: Automated social-gaze analysis is motivated partly by clinical and research needs involving developmental conditions such as autism.The paper connects automated analysis with reducing laborious manual gaze coding and enabling finer-grained analysis.
3. VideoAttentionTarget Dataset
VideoAttentionTarget is a dataset created for modeling dynamic gaze targets in realistic video. It combines diverse clips, frame-level head tracks, gaze annotations, out-of-frame labels, and held-out shows for evaluating generalization.
- Dataset construction: VideoAttentionTarget contains 1,331 annotated dynamic gaze tracks from diverse real-world situations.The dataset was created specifically for video gaze target modeling.
- Dataset construction: Videos from 50 shows provide clips lasting 1–80 seconds without scene cuts, enabling continuous observation of dynamic gaze behavior.Sources include interviews, sitcoms, reality shows, and movie clips.
- Annotations: Annotators labeled 164,541 frame-level head bounding boxes and 109,574 in-frame plus 54,967 out-of-frame gaze targets.Each gaze target was labeled as a point per frame, with an explicit option for targets outside the video frame.
- Evaluation split: The test split holds out 10 shows, 298 tracks, and 31,978 gaze annotations, with no show overlap between training and testing.This split measures generalization to new scenarios and individuals.
- Evaluation split: Additional annotations from two raters support reporting human inter-rater reliability as an upper bound on algorithm performance.The additional raters annotated test samples beyond the original training annotations.
4. Spatiotemporal Gaze Architecture
The spatiotemporal architecture combines head-conditioned scene reasoning with recurrent temporal modeling to predict gaze heatmaps and whether attention lies inside the frame. It is trained with separate heatmap and in-frame objectives and builds on pretrained spatial components.
- Architecture: The architecture has a head conditioning branch, a main scene branch, and a recurrent attention prediction module.These components jointly process head, scene, and temporal information.
- Spatial reasoning: Head features and head-position encoding generate an attention map that regulates spatial scene features.The model uses a binary head-position image and an attention layer to weight scene features according to head information.
- Temporal prediction: A ConvLSTM integrates frame-sequence information, while four deconvolution layers upsample its output into a full-sized heatmap.The deconvolutional design produces finer details than the grid-based map used in prior work.
- In-frame modeling: A learned scalar α modulates the heatmap according to whether attention is in-frame, with higher α indicating in-frame attention.The final heatmap is formed by subtracting 1−α element-wise and clipping values at zero.
- Training objective: Training combines Gaussian-target heatmap MSE with binary-cross-entropy in-frame loss as L = w_h · L_h + w_f · L_f.The heatmap loss applies when the ground-truth target is in frame.
- Training and implementation: The model is first trained on GazeFollow to convergence and then trained on VideoAttentionTarget, with augmentations and head-position noise used during training.The implementation uses 224×224 inputs, 7×7 spatial attention weights, and 64×64 output heatmaps.
5. Experiments
Experiments evaluate the spatial and spatiotemporal gaze-target models on benchmark gaze prediction, toddler social-gaze detection, and shared-attention recognition. The results show strong benchmark performance, while toddler gaze-shift detection remains below human performance.
- Experimental overview: The experiments cover GazeFollow, VideoAttentionTarget, toddler gaze-shift detection, and VideoCoAtt shared-attention recognition.The study evaluates spatial prediction, full spatiotemporal prediction, clinically relevant social behaviors, and shared attention.
- 5.1. Spatial Module Evaluation: The attention mechanism regulates scene processing using head pose information to improve gaze-target localization.The architecture combines scene and head convolutions, with head features regulating spatial pooling through attention.
- 5.1. Spatial Module Evaluation: The spatial model achieves an AUC of 0.921 on GazeFollow, close to the human AUC of 0.924.The model’s learned attention weights use facial orientation to weight scene features.
- 5.2. Spatiotemporal Model Evaluation: The full model evaluates in-frame localization and out-of-frame detection using AUC, Distance, and Out-of-Frame AP.AUC and Distance apply to in-frame targets, while Out-of-Frame AP evaluates the scalar out-of-frame prediction score across frames.
- 5.2. Spatiotemporal Model Evaluation: Ablations show that all components contribute to performance, with the head-convolution pathway and attention mechanism contributing most.The ablations remove head position, head features, attention maps, fusion, or temporal modeling one at a time.
- 5.3. Detecting the Social Bids of Toddlers: The toddler gaze-shift detector remains below human performance, although the 2D-CNN approach generally outperforms the 3D-CNN approach.The task uses 623 toy-to-eyes shift events from 221 minutes of toddler recordings.
- 5.4. Detecting Shared Attention in Social Scenes: The method achieves state-of-the-art results on both social-gaze recognition tasks without being specifically tuned for shared-attention detection.The authors caution that differences in head detection and training data limit direct interpretation of the shared-attention comparison.
6. Conclusion
The paper presents a deep architecture and dataset for detecting time-varying attention targets in video. Its strong performance across benchmarks and a social-gaze task supports its potential for analyzing gaze behavior in naturalistic interactions.
- 6. Conclusion: The paper introduces a deep architecture and VideoAttentionTarget dataset for detecting time-varying attention targets in video.The model lets face information guide learning of gaze-relevant scene regions, while the dataset supports learning their temporal evolution.
- 6. Conclusion: Strong performance across multiple benchmark datasets and a social-gaze recognition task supports the method’s potential for understanding gaze behavior in naturalistic human interactions.