PaperlayerView exact revision on arXiv

Source-linked AI summary

Object-aware Gaze Target Detection

Francesco Tonini, Nicola Dall'Asen, Cigdem Beyan, Elisa Ricci

arXiv:2307.09662v2cs.CV

TL;DR

Gaze-target methods have not fully modeled relationships between people and the objects they look at. This paper uses an object-aware Transformer to associate detected heads with gazed objects, reporting state-of-the-art benchmark results for gaze detection and object classification and localization.

  • Problem

    Existing gaze-target approaches have not fully decoded relationships between people and gazed objects, despite objects being typical gaze targets in interactions.

    Source: §1 · p. 2

  • Method

    An end-to-end Transformer detects scene objects including heads, builds per-person gaze cones, and models head-object interactions to predict gaze and gazed-object properties.

    Source: §1 · p. 2

  • Results

    The method reports state-of-the-art gaze-target results, including up to 2.91% gain in AUC, 50% reduction in gaze distance, and 9% gain in out-of-frame AP, with 11-13% gain in AP for gazed-object classification and localization.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Jointly predicting gaze and gazed-object class and location provides a more explainable account of gaze behavior.

    Source: §5 · p. 8

    Earlier traditional methods require carefully human-annotated head crops during training and inference, and repeated runs to handle multiple people.

    Source: §1 · p. 2

Abstract

from arXiv · show

Gaze target detection aims to predict the image location where the person is looking and the probability that a gaze is out of the scene. Several works have tackled this task by regressing a gaze heatmap centered on the gaze location, however, they overlooked decoding the relationship between the people and the gazed objects. This paper proposes a Transformer-based architecture that automatically detects objects (including heads) in the scene to build associations between every head and the gazed-head/object, resulting in a comprehensive, explainable gaze analysis composed of: gaze target area, gaze pixel point, the class and the image location of the gazed-object. Upon evaluation of the in-the-wild benchmarks, our method achieves state-of-the-art results on all metrics (up to 2.91% gain in AUC, 50% reduction in gaze distance, and 9% gain in out-of-frame average precision) for gaze target detection and 11-13% improvement in average precision for the classification and the localization of the gazed-objects. The code of the proposed method is publicly available.

1. Introduction

Gaze target detection remains challenging, and prior approaches do not fully model relationships between people and gazed objects. The paper introduces an object-aware Transformer that predicts gaze and identifies gazed objects, reporting benchmark gains across these tasks.

1. Introduction

  • Gaze is a nonverbal signal of visual attention that can indicate people's interests, intentions, or future actions.The paper notes its use in human-computer interaction, neuroscience, psychology, and social robotics.Source: §1 · p. 1
  • Prior traditional methods require carefully annotated head crops and repeated runs for multiple people, while the Transformer of Tu et al. [35] omits gazed-object contributions.The paper identifies accurate head detectors as an additional requirement for traditional methods in practical applications.Source: §1 · p. 2
  • The method predicts gaze areas and points, out-of-scene status, and the gazed object's class and bounding box, providing an explainable gaze analysis.It learns scene objects, including heads, rather than relying on head locations supplied by datasets.Source: §1 · p. 2, Table 1 · p. 2
  • An end-to-end Transformer detects objects including heads, then models head-object interactions to predict gaze heatmaps and pixel points.For each head, a predicted gaze vector defines a cone that filters objects before the Gaze Object Transformer models their interactions.Source: §1 · p. 2
  • The method reports state-of-the-art gaze-target results, with up to 2.91% gain in AUC, 50% reduction in gaze distance, and 9% gain in out-of-frame AP, plus 11-13% gain in AP for gazed-object classification and localization.The paper also reports superior performance on two large-scale benchmark datasets and strong out-of-frame prediction.Source: §1 · p. 2

2. Related Work

Related work developed gaze-target detection from scene and head-crop models toward approaches using depth, pose, and temporal cues. The paper positions its method as jointly detecting multiple people's gaze targets and the objects they look at.

2. Related Work

  • GazeFollow established a scene-image and manually annotated head-crop approach for estimating where a person looks.Its two CNN branches estimate scene saliency and process the head crop with its location.Source: §2 · p. 2
  • The proposed pipeline detects scene objects, builds a gaze cone for each head, and uses object-informed attention to produce gaze heatmaps.When no object lies in a person's cone, a skip connection allows heatmap prediction from head features alone.Source: Figure 2 · p. 3
  • Earlier methods extended scene-image and head-crop models with temporal, depth, head-pose, eye-location, or 2D-pose information.The passage notes that some added task-specific detectors or modalities, and that inaccurate head-pose or eye-location estimates can negatively affect detection.Source: §2 · p. 3
  • Tu et al. [35] introduced the first Transformer-based gaze-target approach and reported traditional methods' performance drops when using automatically detected head locations.That approach detects and localizes heads but omits the contribution of objects to gaze interactions.Source: §2 · p. 3
  • The proposed method targets simultaneous gaze detection for multiple people by learning gazed-object localization and classification and head-object gaze interactions.It is presented as an end-to-end Transformer designed to provide explainable gaze target detection.Source: §2 · p. 3

3. Method

The method detects scene objects and heads, estimates per-head gaze vectors, and uses gaze cones and object-aware attention to predict gaze heatmaps and whether gaze lies outside the frame. Its components jointly support object detection, gaze-target inference, and multi-task training.

3. Method

  • The pipeline detects scene objects including heads, predicts a gaze vector for each head, filters objects by that head’s gaze cone, and uses object relationships to predict gaze heatmaps and out-of-frame probability.When no object lies inside a person’s cone, a separate head-feature pathway predicts the heatmap.Source: §3 · pp. 3–4

3.1. Object Detector Transformer

  • The Object Detector Transformer encodes backbone image features and decodes learnable object queries into bounding boxes and class labels.The maximum number of predicted objects is N.Source: §3.1 · p. 4

3.2. Gaze Cone Predictor

  • For each detected head, an MLP estimates a 3D gaze vector that defines a cone of angle α; object scores sample cone intensity at object centers, indicating cone membership and closeness to the predicted gaze vector.Source: §3.2 · pp. 4–5

3.3. Gaze Object Transformer

  • The Gaze Object Transformer adds the object-score matrix as an attention bias, masks head and non-head features by role, and predicts heatmaps from the resulting features.A separate heatmap pathway handles heads with no object inside the cone, while another head-feature MLP predicts out-of-frame probability.Source: §3.3 · p. 5

3.4. Training objective

  • The training objective is a weighted sum of object localization and classification losses, gaze-vector and heatmap losses, and an out-of-frame loss.Localization combines weighted L1 distance and GIoU; vector and heatmap losses use L2, while the out-of-frame loss uses binary cross-entropy.Source: §3.4 · pp. 5–6

4. Experiments

Experiments on GazeFollow and VideoAttentionTarget show strong gaze-target results and effective gazed-object classification and localization, with ablations and annotation-variance analyses examining the method’s components and robustness.

4.1. Datasets and Evaluation metrics

  • GazeFollow has over 122K images and more than 130K people, while VideoAttentionTarget has in- and out-of-frame gaze annotations; evaluation uses gaze-target and gazed-object metrics.VideoAttentionTarget clips are subsampled to one image per five consecutive frames because the model does not use temporal information.Source: §4.1 · p. 6

4.2. Implementation details

  • Training uses Adam for 80 epochs followed by 20 epochs at a learning rate reduced tenfold, with a 120° gaze cone and confidence filtering for GOT queries, keys, and values.Depth maps are used only to build gaze cones, without learning additional depth features.Source: §4.2 · pp. 6–7

4.3. Comparison with State-of-the-Art

  • Using RGB alone, the method outperforms prior state of the art across datasets and metrics, with minimum and average distance and mAP best in both Head GT and Real settings.Adding depth slightly improves average performance; minimum and average distance and mAP remain best with or without it.Source: §4.3 · p. 7

4.4. Ablation Study

  • Across both datasets, GOT improves all gaze-target metrics, and the full gaze-cone plus no cone-object skip design performs best consistently.Object masking also improves metrics and yields interpretable head-object attention; the skip and gating mechanism supplies a heatmap when the cone contains no meaningful object.Source: §4.4 · p. 7, Table 3 · p. 8

4.5. Gazed-object class detection and localization

  • On the GazeFollow COCO subset, the proposed model is relatively effective for gazed-object prediction and has the fewest parameters, despite not being trained on COCO object classes and locations.The comparison includes DETR trained on the full COCO dataset, which the authors note is advantageous to DETR.Source: §4.5 · pp. 7–8, Table 4 · p. 8

4.6. The effect of variance in gaze annotations

  • Across increasing gaze-annotation variance, both 2D and 3D versions consistently outperform Tu et al. [35], while all methods perform better at lower variance.The authors speculate that 3D performance is lower than 2D because annotations were collected on 2D images.Source: Table 5 · p. 8, §4.6 · p. 8

4.7. Qualitative Results

  • Qualitative gaze heatmaps on GazeFollow are more accurate than Tu et al. [35] in the illustrated comparison.Figure 4 shows one person’s gaze, with the ground truth above the Tu et al. [35] and proposed-method results.Source: Figure 4 · p. 8, §4.7 · p. 8

5. Discussion & Conclusion

The paper presents an end-to-end Transformer gaze-target detector that predicts gazed-object classes and locations, improving explainability; it also notes slow convergence and privacy considerations for real-world use.

5. Discussion & Conclusion

  • The end-to-end Transformer detector jointly predicts gazed-object class and location, which the authors identify as advantageous for explainability.They report better gaze behavior understanding and potential usefulness in real-world human interaction analysis, while noting that face processing may warrant privacy-policy review and that training convergence is slow.Source: §5 · p. 8

A. The effect of variance in gaze annotations

The appendix examines how multiple, sometimes divergent GazeFollow test annotations affect evaluation, using AUC across variance levels to compare methods.

A. The effect of variance in gaze annotations

  • GazeFollow test images can have up to 10 gaze-point annotations per person, and the standard AUC metric does not account for variation among them.The training split has one gaze-point annotation per person.Source: A. The effect of variance in gaze… · p. 11
  • The variance analysis computes each annotation’s distance from the person’s average gaze point, retains points by distance decile, and reports AUC for each level.Figure 6 plots model performance across those retained-variance levels.Source: A. The effect of variance in gaze… · p. 11, Figure 6 · p. 11
  • Both proposed versions consistently outperform Tu et al. [35] across variance levels; performance falls faster in the highest-noise annotations, and 3D trails 2D.The authors speculate that the 3D-versus-2D difference may reflect that human annotations were collected on 2D images.Source: A. The effect of variance in gaze… · p. 11

B. Additional evaluation on GazeFollow [30] and VideoAttentionTarget [7]

The additional evaluation reports angular error for gaze vectors on GazeFollow and compares the proposed method with state of the art.

B. Additional evaluation on GazeFollow [30] and VideoAttentionTarget [7]

C. Implementation Details

The implementation adapts DETR for object and head detection, using dataset and automatically generated annotations to train it. It also uses monocular depth estimates for depth images.

C. Implementation Details

  • DETR’s object-classification and detection heads were replaced with larger MLPs to predict object locations and classes, including the head class.The number of object classes was adapted to include heads.Source: C. Implementation Details · p. 12
  • YOLOv8 pseudo-annotated objects in images without object annotations, while RetinaFace supplied additional head detections.The datasets lacked object annotations except for the COCO subset of GazeFollow.Source: C. Implementation Details · p. 12
  • Head annotations for every head in an image were necessary for convergence, using dataset locations and RetinaFace detections.The authors note that other head detectors could also be adapted.Source: C. Implementation Details · p. 12
  • MIDAS monocular depth estimation was used to obtain depth images for both datasets.Source: C. Implementation Details · p. 12

D. Qualitative Results

Qualitative comparisons show the method predicting gaze more precisely than Tu et al. [35] in many scenes, while also illustrating difficult cases and the role of 3D gaze cones. The examples cover GazeFollow and VideoAttentionTarget [7].

D. Qualitative Results

  • The qualitative-results section compares gaze heatmaps and source-head boxes with ground truth, including examples where the method has AUC below 70%.Source: D. Qualitative Results · p. 12
  • The method predicts gaze precisely in many scenes where Tu et al. [35] does not, although both methods’ predictions fall within the person’s field of view.The comparisons cover GazeFollow and VideoAttentionTarget [7].Source: D. Qualitative Results · p. 12, Figure 7 · p. 13, Figure 8 · p. 14
  • Tu et al. [35] favors image regions closer to the gaze source, the person’s head.Source: D. Qualitative Results · p. 12
  • Challenging head poses make gaze prediction difficult, while partially visible faces produce scattered heatmaps that still cover the gaze point.The examples include cases where the method performs relatively worse and Tu et al. [35] also often underperforms.Source: D. Qualitative Results · p. 12, Figure 9 · p. 15
  • In complex scenes, the 3D gaze vector and cone help identify the looked-at object; they also further center heatmaps toward objects when 2D predictions are already strong.The 3D-cone comparison highlights its importance particularly in challenging scenes.Source: D. Qualitative Results · p. 12, Figure 10 · p. 16
Loading 2307.09662v2…