Source-linked AI summary
Multimodal Across Domains Gaze Target Detection
Francesco Tonini, Cigdem Beyan, Elisa Ricci
TL;DR
Gaze target detection asks where a person in an image is looking, but automatic gaze behavior analysis remains challenging. The paper proposes a three-pathway multimodal model and a domain-adaptation method; it reports improved target-dataset performance, including results above a state-of-the-art adaptation method.
Problem
Automatic gaze behavior analysis remains challenging, and gaze target detection focuses on locating where a person in a scene is looking.
Source: §1 · p. 1Method
A spatial model processes the person’s head image, scene image, and depth map, with an integrated domain-adaptation method to address dataset shift.
Source: §1 · p. 2Results
The proposed domain-adaptation method improves target-dataset performance and outperforms a state-of-the-art multimodal domain-adaptation method.
Source: §1 · pp. 2–3Takeaways & Limitations
The paper’s results support studying domain shift and adaptation for gaze target detection across datasets.
Source: §1 · p. 2The reported datasets were collected in unconstrained situations, so the stated evaluation scope is those settings.
Source: §5 · p. 14
Abstract
from arXiv · showhide
This paper addresses the gaze target detection problem in single images captured from the third-person perspective. We present a multimodal deep architecture to infer where a person in a scene is looking. This spatial model is trained on the head images of the person-of- interest, scene and depth maps representing rich context information. Our model, unlike several prior art, do not require supervision of the gaze angles, do not rely on head orientation information and/or location of the eyes of person-of-interest. Extensive experiments demonstrate the stronger performance of our method on multiple benchmark datasets. We also investigated several variations of our method by altering joint-learning of multimodal data. Some variations outperform a few prior art as well. First time in this paper, we inspect domain adaption for gaze target detection, and we empower our multimodal network to effectively handle the domain gap across datasets. The code of the proposed method is available at https://github.com/francescotonini/multimodal-across-domains-gaze-target-detection.
1 INTRODUCTION
The paper proposes a multimodal spatial model for detecting gaze targets in single images and investigates domain adaptation for gaze target detection. It reports improved target-dataset performance from its adaptation method, including better results than a state-of-the-art multimodal adaptation method.
1 INTRODUCTION
- Earlier gaze-target methods can produce false detections among objects at different depths along a person’s gaze direction, motivating depth-aware processing.The cited prior work also includes methods requiring head-pose and eye-feature detection or auxiliary networks for depth and gaze-orientation estimation.Source: §1 · p. 2
- The proposed three-pathway spatial model processes the person’s head image, scene image, and depth map without gaze-angle supervision or explicit head-pose and eye-location detection.The depth map is obtained through standalone monocular depth estimation from RGB images.Source: §1 · p. 2
- The model is evaluated on several benchmark datasets, where it shows improved results relative to the state of the art.The paper also reports that some model variations achieve better scores than the state of the art.Source: §1 · p. 2
- The domain-adaptation method improves target-dataset results and outperforms a state-of-the-art multimodal domain-adaptation method.The paper first diagnoses domain shift in its model and prior art, then proposes a method to address it.Source: §1 · pp. 2–3
2 RELATED WORK
Prior work studies 2D gaze targets through scene-and-head pathways, depth-aware models, and domain adaptation, while this paper extends multimodal gaze detection to cross-dataset adaptation.
2 RELATED WORK
- Gaze target detection has applications in human interaction systems, computer vision, and robotics, where understanding the object-of-interest and predicting and anticipating actions [22] [30] are important.Existing works may rely on particular sensors, address specialized settings or applications, or require constrained subject placement; targets may be in 2D images or 3D space.Source: §2.1 · p. 3
2.1 Gaze Target Detection
- Earlier 2D gaze systems combine scene and head pathways, with extensions for out-of-frame targets and temporal behavior; depth-aware alternatives can require head pose, eye locations, or pseudo-labels [5].Prior methods also struggle when candidate objects lie at different depths along the gaze direction [13].Source: §2.1 · p. 3
- The proposed approach avoids gaze-angle supervision, uses monocularly estimated depth alongside RGB, and applies head-derived attention to both scene and depth features [23].It does not use head poses or eye locations, and does not use spatio-temporal processing as in [5].Source: §2.1 · pp. 3–4
2.2 Domain Adaptation
- Prior gaze domain adaptation addresses gaze estimation, but 2D gaze-target detection had not been studied; the paper adapts modality-translation methods to its multimodal network [11].The adaptation uses a head-domain classifier and RGB-to-depth and depth-to-RGB decoders.Source: §2.2 · pp. 4–5
3 METHODOLOGY
The methodology combines head, scene, and depth pathways for gaze prediction, then adds domain-adaptation components and describes training and implementation choices.
3 METHODOLOGY
- The three-pathway model processes head, RGB scene, and monocular depth inputs to predict a 2D gaze heatmap and an inside/outside-frame probability.Head-derived attention modulates scene and depth features, which are fused to produce the outputs.Source: §3 · pp. 5–6
3.1 Multimodal Network
- Head features produce an attention map that modulates scene and depth embeddings, while fusion combines the features for heatmap and in/out prediction.The prediction module uses summation of scene and depth embeddings for the heatmap and their outer product for the in/out output.Source: §3.1 · pp. 6–7
3.2 Domain Adaptation
- Domain adaptation combines a GRL head-domain classifier with decoders for RGB-to-depth and depth-to-RGB reconstruction.These components target the head, scene, and depth networks to improve performance across datasets.Source: §3.2 · p. 7
3.3 Implementation Details
- Inputs are resized to 224 × 224 and processed with ResNet-50 backbones; training uses GazeFollow, VideoAttentionTarget, and GOO datasets.During adaptation, missing in/out annotations mean the in/out cross-entropy loss is not minimized; the adaptation objective includes domain-classification and bidirectional reconstruction losses.Source: §3.3 · pp. 7–8
4 EXPERIMENTAL ANALYSIS
Experiments evaluate the multimodal model on three benchmark datasets, compare its components and fusion strategies, and test cross-dataset performance with and without domain adaptation. The method leads the reported AUC comparisons, while adaptation improves cross-domain results.
4.1 Datasets & Evaluation Metrics
- Evaluation follows standard train/test splits on GazeFollow, VideoAttentionTarget, and GOO, using heatmap AUC and Euclidean distance to the ground-truth gaze point.The experiments use the real-environment images from GOO.Source: §4.1 · p. 8
4.2 Ablation Study & Modality Fusion
- 16.9% AUC improvement and 0.117 lower Avg.Dist. make the head network the most crucial component; adding depth to scene+head yields +0.6% AUC and -0.002 Avg.Dist. on GazeFollow.The proposed scene+depth+head model improves over scene+head by +3.4% AUC and -0.01 Avg.Dist. on VideoAttentionTarget.Source: §4.2 · pp. 9–10, Table 1 · p. 9
- Late fusion outperforms early-fusion methods on GazeFollow, while the proposed late fusion performs best among tested late-fusion methods on VideoAttentionTarget.Among early-fusion variants, grayscale depth exceeds colored depth by +0.9% AUC on GazeFollow, but colored depth is slightly better by +0.3% AUC on VideoAttentionTarget.Source: Table 1 · p. 9, §4.2 · p. 10
4.3 Comparisons with the State-of-the-art
- Our method achieves the best AUC among counterparts on all datasets, surpassing human performance on GazeFollow and VideoAttentionTarget.For Avg.Dist., it trails Fang et al. and Jin et al. but outperforms the other compared methods.Source: §4.3 · p. 10, Table 2 · p. 11
4.4 Gaze Target Detection Across Datasets
- Cross-dataset testing degrades both models by up to -36.5% AUC and +0.3% Avg.Dist. versus same-dataset testing, although ours outperforms Chong et al. [5] in every cross-domain analysis.Table 3 reports cross-dataset results and gives the drops from same-dataset evaluation in parentheses.Source: §4.4 · p. 11, Table 3 · p. 12
4.5 Domain Adaptation Results
- 1.3–15.6% more AUC and 0.017–0.086 less Avg.Dist. with proposed domain adaptation versus without; AUC gains exceed Ferreri et al. [11] in every setting.The adaptation module translates between RGB and depth with reconstruction losses; its ablation indicates that each loss function contributes to the best combined performance.Source: §4.5 · p. 12, Table 4 · p. 13
4.6 Qualitative Results
- Qualitative examples show the multimodal model producing more compact heatmaps than Chong et al. [5], and adaptation improving results for GazeFollow-to-GOO transfer.Figure 2 compares both models with ground truth; Figure 3 compares the proposed model with and without adaptation against ground truth.Source: §4.6 · p. 13, Figure 2 · p. 13, Figure 3 · p. 14
5 DISCUSSION
The discussion presents a three-pathway spatial gaze-target model using head, scene, and depth context without gaze-angle supervision, explicit head orientation, or eye-location inputs. It reports favorable evaluations and domain-adaptation gains, while noting that missing annotations preclude out-of-frame evaluation.
5 DISCUSSION
- The spatial model processes the person-of-interest’s head image, scene image, and depth maps together to supply context.Source: §5 · p. 14
- The method performs favorably against existing approaches, and some investigated joint-learning variations outperform several state-of-the-art methods.Source: §5 · p. 14
- The model does not require gaze-angle supervision, explicit head orientation, or eye-location information.Source: §5 · p. 14
- Adding domain-adaptation components improves performance on target datasets and outperforms domain-adaptation state of the art.Source: §5 · p. 14
- Missing out-of-frame annotations prevent evaluation of out-of-frame precision; proposed future work includes human-robot interaction testing and transformer integration.Source: §5 · pp. 14–15