Source-linked AI summary
End-to-End Human-Gaze-Target Detection with Transformers
Danyang Tu, Xiongkuo Min, Huiyu Duan, Guodong Guo, Guangtao Zhai, Wei Shen
TL;DR
Existing gaze-following approaches require head locations and predict gaze targets separately, often one person at a time. HGTTR jointly detects all heads and gaze targets from scene-wide context; it outperforms state-of-the-art methods on both benchmark datasets, including a 6.4 mAP gain on GazeFollowing.
Problem
Existing methods require head crops and separate gaze-target prediction from head detection, limiting them to sequential predictions and lacking contextual reasoning between gaze and saliency.
Source: §1 · pp. 1–2Method
HGTTR uses set-based Transformer prediction to detect all people’s head locations and gaze targets together from a single scene image.
Source: §1 · p. 2Results
HGTTR outperforms existing state-of-the-art methods on GazeFollowing and VideoAttentionTarget, with a 6.4 mAP gain on GazeFollowing.
Source: §1 · p. 2Takeaways & Limitations
HGTTR predicts head positions and gaze targets simultaneously without requiring head images as input.
Source: §5 · p. 8HGTTR has slow convergence because learning attention weights to focus on sparse meaningful locations takes long training epochs.
Source: §4.6 · p. 8
Abstract
from arXiv · showhide
In this paper, we propose an effective and efficient method for Human-Gaze-Target (HGT) detection, i.e., gaze following. Current approaches decouple the HGT detection task into separate branches of salient object detection and human gaze prediction, employing a two-stage framework where human head locations must first be detected and then be fed into the next gaze target prediction sub-network. In contrast, we redefine the HGT detection task as detecting human head locations and their gaze targets, simultaneously. By this way, our method, named Human-Gaze-Target detection TRansformer or HGTTR, streamlines the HGT detection pipeline by eliminating all other additional components. HGTTR reasons about the relations of salient objects and human gaze from the global image context. Moreover, unlike existing two-stage methods that require human head locations as input and can predict only one human's gaze target at a time, HGTTR can directly predict the locations of all people and their gaze targets at one time in an end-to-end manner. The effectiveness and robustness of our proposed method are verified with extensive experiments on the two standard benchmark datasets, GazeFollowing and VideoAttentionTarget. Without bells and whistles, HGTTR outperforms existing state-of-the-art methods by large margins (6.4 mAP gain on GazeFollowing and 10.3 mAP gain on VideoAttentionTarget) with a much simpler architecture.
1. Introduction
Gaze following predicts where each person in an image or video frame is looking, but existing approaches require head crops and separate gaze and saliency predictions. HGTTR instead detects all heads and gaze targets together from scene-wide context.
1. Introduction
- Existing methods require both scene and head images, use an additional head detector, predict targets sequentially, and lack joint contextual reasoning between gaze and saliency.Detector precision can affect final results, and multi-person scenes require repeated prediction and post-processing.Source: §1 · pp. 1–2
- HGTTR detects all people’s head locations and gaze targets simultaneously from a single scene image using image-wide contextual modeling.It represents outputs as human-gaze-target instances pairing a head location with an attention target.Source: §1 · p. 2
- HGTTR uses set-based prediction with learnable queries, a CNN backbone, Transformer global-context modeling, decoder embeddings, and a quaternate matching loss.The model predicts multiple human-gaze-target instances from one image.Source: §1 · p. 2
2. Related Work
Prior gaze-following work estimates what a person is looking at in images and video, with methods using auxiliary cues and datasets addressing video dynamics and out-of-frame targets. Transformer-based vision methods provide related context for HGTTR’s approach.
2. Related Work
- Gaze following estimates what a person is looking at in a picture or video frame, with prior work incorporating gaze angle, body pose, sight lines, and video data.The task is distinguished from eye tracking and saliency detection.Source: §2 · p. 2
- VideoAttentionTarget models gaze dynamics from video and infers per-frame gaze targets, while other work addresses out-of-frame targets and 3D gaze inference.GazeFollowing introduced a large dataset and an accompanying algorithm.Source: §2 · p. 2
- Vision Transformers have been applied to object detection, image classification, semantic segmentation, and low-level vision tasks.DETR reformulated object detection as set prediction using a Transformer.Source: §2 · p. 2
3. Method
The method reformulates gaze following as predicting paired head locations and gaze targets for all people in one image. HGTTR uses Transformer-based global context and set matching to produce and train these predictions.
3.1. Problem Reformulation
- The method predicts all human head locations and gaze locations jointly from one image, representing head positions as bounding boxes and gaze locations as Gaussian heatmaps.The objective maximizes the joint posterior of head-location and gaze-location pairs; the number of pairs corresponds to the people in the image.Source: §3.1 · p. 2
3.2. Network Architecture
- HGTTR combines a CNN backbone and Transformer encoder-decoder with learnable HGT queries, then uses MLP branches to predict head boxes, human and watch-in/out confidence, and gaze heatmaps.The encoder produces contextual features; decoder self-attention and cross-attention transform the query embeddings into output embeddings.Source: §3.2 · pp. 3–4
3.3. Loss Calculation
- Training matches fixed-size prediction sets to padded ground-truth sets with Hungarian bipartite assignment, using costs for head boxes, headness, watch-in/out status, and gaze heatmaps.Head-box regression combines weighted GIoU and L1 losses, and the matched pairs determine the training loss.Source: §3.3 · p. 4
4.1. Datasets and Evaluation Metrics
VideoAttentionTarget training samples one of every five similar consecutive frames, while testing uses all test images. The evaluation combines gaze-target measures with human-and-target detection mAP.
4.1. Datasets and Evaluation Metrics
- Training uses one randomly selected frame from each five-frame group with no obvious appearance differences; testing uses every image in the test set.The authors omit temporal information during training because these frames have almost the same gaze target and to avoid overfitting.Source: §4.1 · p. 4
- Images are annotated together in a COCO-style format so the model can predict all subjects in a scene at once.Head locations serve as ground truth rather than model inputs; previous work assigned each person in an image a separate annotation.Source: §4.1 · p. 4
- Gaze-target prediction is evaluated with AUC and L2 distance, with both average and minimum ground-truth distances reported for multimodal GazeFollowing targets.The distance uses image dimensions normalized to 1; gaze targets outside the scene are evaluated with AP.Source: §4.1 · pp. 4–5
- A human-gaze-target detection counts as correct only when head-box IoU exceeds 0.5 and gaze-target L2 distance is below 0.15; mAP evaluates detection.The table compares GazeFollowing and VideoAttentionTarget results under Default and Real settings, using labeled or detector-predicted head locations, respectively.Source: §4.1 · p. 5, Table 1 · p. 5
4.2. Implementation Details
HGTTR uses ResNet backbones and a six-layer Transformer encoder and decoder, with DETR initialization and a 150-epoch AdamW training schedule.
4.2. Implementation Details
- Experiments use ResNet-50 or ResNet-101 backbones, and six-layer Transformer encoders and decoders with eight attention heads.The network is initialized from DETR trained on COCO.Source: §4.2 · p. 5
- Training runs for 150 epochs with AdamW and batch size 16; the backbone and other components start at learning rates 10^-5 and 10^-4, respectively.Both learning rates decay after 80 epochs; weight decay is 10^-4.Source: §4.2 · p. 5
- The model uses 20 HGT instance queries per dataset and is trained on eight NVIDIA GTX 2080TI GPUs.Loss weights β1–β4 are 2, 1, 1, and 2, while α1 and α2 are 1.0 and 2.5.Source: §4.2 · p. 5
4.3. Comparison to State-of-the-Art
HGTTR outperforms existing state-of-the-art methods on both datasets, with reported gains in mAP on GazeFollowing and AUC on VideoAttentionTarget.
4.3. Comparison to State-of-the-Art
- 6.4 mAP over VideoAttention [4] and 7.8 mAP over GazeFollow on GazeFollowing are reported for HGTTR with ResNet-50.The paper reports that HGTTR outperforms existing state-of-the-art methods on all datasets.Source: §4.3 · p. 5
- 8 AUC over VideoAttention [4] is reported for gaze-target prediction on VideoAttentionTarget in the Real setting.The Real setting uses head positions generated by an additional head detector for existing methods.Source: §4.3 · p. 5
- The paper attributes VideoAttentionTarget’s stronger performance partly to its larger size and multiple people per image; fewer people per image can increase false positives for fixed-count outputs.GazeFollowing images contain only one person, whereas VideoAttentionTarget includes images with more than one person.Source: §4.3 · p. 5
4.4. Ablation Study
The ablations examine HGTTR’s contextual modeling and matching strategy, then compare pairwise detection with methods that depend on detected head locations. They report robustness and efficiency advantages for pairwise detection.
4.4.1 Model components
- HGTTR performs relatively better than existing methods on distant gaze targets, where gaze-target detection becomes more difficult.Instances are binned by normalized L1 distance between head center and gaze target; the relative performance gaps grow with distance.Source: Figure 3 · p. 6, §4.4.1 · p. 6
4.4. Ablation Study
- The ablation experiments use a ResNet-50 backbone and the VideoAttentionTarget dataset.Source: §4.4 · pp. 5–6
4.4.1 Model components
- More decoder layers improve performance, especially on AUC and L2 Dist., with the figure showing normalized metric trends.The authors suggest additional self-attention may help the decoder learn relationships among HGT instances.Source: Figure 4 · p. 6, §4.4.1 · p. 6
- Matching performs best with β1 = β4 = 2 and β2 = β3 = 1, indicating localization matters more than classification in the matching costs.The costs cover head-box location, head presence, whether the target is in-frame, and gaze-heatmap localization.Source: §4.4.1 · p. 6, Table 2 · p. 6
4.4.2 Importance of pairwise detection
- Decoder attention maps highlight both human heads and gaze targets, while different instances can show little shared pattern.The authors interpret the maps as evidence of global-context reasoning and the need for a unique matching process.Source: §4.4.1 · p. 6, Figure 5 · p. 7
- HGTTR shows better robustness and performance under image degradations than methods using different head detectors.The authors attribute the comparison partly to pairwise detection being less sensitive to degradations than two-stage methods.Source: Table 3 · p. 7, §4.4.2 · p. 7
- As human count rises, existing methods’ inference time grows sharply, while HGTTR’s is almost unaffected.The implementation handles up to 20 people; increasing Nq is the stated way to raise that maximum.Source: §4.4.2 · pp. 7–8, Figure 6 · p. 8
4.5. Qualitative Analysis
The paper reports practical-application results for HGTTR and evaluates its potential for detecting shared attention in social scenes.
Practical application
- HGTTR outperforms all state-of-the-art methods by a significant margin in the practical-application evaluation.The models were trained only on VideoAttentionTarget, and existing methods also used an additional head detector.Source: §4.5 · p. 8
- The practical-application evaluation uses 15,000 DL Gaze pictures, with ground truth annotated by observers in videos.The dataset records daily activities of 16 volunteers in four scenes.Source: §4.5 · p. 8
Shared attention detection
- On VideoCoAtt, HGTTR shows potential for recognizing higher-level social gaze through shared-attention detection.The 113,810 test frames mark target locations attended by at least two people; interval detection uses accuracy and location prediction uses L2 distance.Source: §4.5 · p. 8
4.6. Discussion
The discussion presents joint head-and-target detection as better suited to practical use, while noting slow Transformer convergence and privacy concerns.
Discussion
- HGTTR's Transformer-based training converges slowly because learning attention weights to focus on sparse meaningful locations takes many epochs.The authors suggest more flexible self-attention, such as deformable DETR, as a possible solution for future work.Source: §4.6 · p. 8
- Table 4 defines its gaze-direction measure as the angle between ground-truth and predicted gaze directions.Source: Table 4 · p. 8
- Table 5 evaluates shared-attention interval detection by prediction accuracy and location prediction by L2 distance.Source: Table 5 · p. 8
Broader impacts
- Human-centric gaze-target prediction in real-world applications raises privacy concerns and warrants further policy review.The passage frames this as a consideration for practical application, not as a measured outcome.Source: §4.6 · p. 8
5. Conclusion
The conclusion presents HGTTR as a Transformer-based gaze-following method that detects head positions and gaze targets simultaneously. It reports higher effectiveness and efficiency without head-image input, and points to potential for understanding gaze behavior and human activity.
Conclusion
- HGTTR detects all people's head positions and gaze targets simultaneously for gaze-following detection.Source: §5 · p. 8
- Without head-image input, HGTTR achieves higher effectiveness and efficiency, according to the conclusion.Source: §5 · p. 8
- The authors report potential for understanding gaze behavior in naturalistic human interactions and for human activity understanding research.Source: §5 · p. 8