ViTGaze: Gaze Following with Interaction Features in Vision Transformers

1 Introduction

• Prior multimodal methods rely on earlier modality predictions, while query-based methods use heavy decoders that increase network complexity.Multimodal methods add inputs such as depth and pose; query-based methods also use object bounding boxes.Source: §1 · pp. 1–2
• The paper investigates whether pretrained ViTs can provide effective interactive representations between people and scenes for gaze following.Its central premise is that self-attention token interactions can represent human-scene interactions.Source: §1 · p. 2
• ViTGaze uses pretrained plain ViTs to extract human-scene interactions from self-attention maps with a 4D interaction encoder and 2D spatial guidance module.The framework is mainly encoder-based, with decoder parameters accounting for less than 1%.Source: §1 · p. 2
• Self-supervised pretrained ViTs show stronger ability to understand token interactions, and ViTGaze uses attention maps alongside feature maps.Source: §1 · p. 2
• 3.4% higher AUC on GazeFollow and 5.1% higher AP on VideoAttentionTarget than the previous single-modality SOTA; ViTGaze also has 59% fewer parameters than multimodal methods.The figure reports SOTA AUC among all methods on both benchmarks; the paper reports comparable multimodal performance, including 1.8% lower distance error and 2.7% higher AUC.Source: Figure 1 · p. 2, §1 · p. 2

2.1 Gaze Following

• Single-modality gaze-following methods use RGB images and varied interaction cues, while multimodality methods add depth, pose, temporal information, masks, or object-level supervision.Many multimodality approaches use two stages and depend on the first-stage predictions; concise, high-performing single-modality gaze following remains unresolved.Source: §2.1 · p. 3

2.2 Pre-training of Vision Transformers

• Self-supervised ViT pretraining research includes masked image modeling, CLIP integration, and online tokenizers.Source: §2.2 · p. 3
• Pretrained ViTs have supported downstream tasks including detection, pose estimation, matting, cell segmentation, and weakly supervised segmentation.Source: §2.2 · p. 3
• Whether pretrained ViT representations contain enough interaction information to facilitate gaze following remains unexplored.Source: §2.2 · p. 3

3.1 Preliminary: Self-Attention

• Self-attention uses query-key dot products to capture token correlations, which are normalized into weights for aggregating values.For image patches, the attention map represents relationships among patch pairs.Source: §3.1 · pp. 3–4

3.2 Overall Structure

• The model takes an image and a person’s head bounding box, then predicts a gaze-target heatmap and the probability that gaze falls outside the scene.Its three components are a pre-trained ViT encoder, 2D spatial guidance, and heatmap and in-out prediction heads.Source: §3.2 · p. 4

3.3 Multi-level 4D Interaction Encoder

• The encoder represents patch interactions as 4D tensors and combines attention maps from four transformer layers and six heads to capture local and global correlations.Each 4D tensor has dimensions h × w × h × w, where h and w are image dimensions divided by the ViT patch size.Source: §3.3 · p. 4

3.4 2D Spatial Guidance

• A two-layer MLP uses head position and features to weight patches, aggregating 4D interactions into person-specific 2D interaction features.Background patches are masked before softmax so the weights are specific to the target person; image-token features guided by the same weights support in-out prediction.Source: §3.4 · pp. 4–5

3.5 Prediction Heads

• The gaze heatmap head uses bilinear interpolation and convolutional layers to predict a logit map, while a two-layer MLP with sigmoid predicts the probability of looking outside.The highest activation in the logit map is the predicted gaze target.Source: §3.5 · p. 5

3.6 Training objective

• End-to-end training combines gaze heatmap, gaze in-out, and auxiliary head regression losses as a weighted sum.The heatmap loss uses mean squared error against a Gaussian target; the in-out loss uses focal loss, and the auxiliary loss uses binary cross-entropy for head occurrence.Source: §3.6 · p. 6

4.1 Datasets and Evaluation Metrics

• GazeFollow has over 130 K annotations in 122 K images and evaluates gaze heatmap regression; VideoAttentionTarget has 1331 clips and 165 K annotations and evaluates heatmap regression and watching-outside prediction.AUC measures gaze-heatmap prediction confidence; distance is relative Euclidean error, with minimum and average distances reported for GazeFollow, while AP evaluates watching-outside prediction.Source: §4.1 · p. 7

4.2 Implementation Details

• ViTGaze uses a 12-block ViT-S encoder and extracts attention maps from blocks 3, 6, 9, and 12 to build multi-level 4D interactions.For GazeFollow, training uses DINOv2 initialization; VideoAttentionTarget fine-tunes weights learned from GazeFollow.Source: §4.2 · p. 7

4.3 Comparison with State-of-the-Art

• 3.4% higher AUC on GazeFollow and 5.1% higher AP on VideoAttentionTarget than the previous single-modality SOTA; Table 1 also reports comparable multi-modality performance with 59% fewer parameters and 25% lower computational cost.The comparison to the previous multi-modality SOTA on GazeFollow is 1.8% lower distance and 2.7% higher AUC.Source: §4.3 · p. 7, Table 1 · p. 8

4.4 Ablation Study

• Multi-level 4D features outperform 2D token features and single-level 4D features; 2D spatial guidance improves AUC by 0.7%.The study also reports that auxiliary head prediction supervises spatial guidance, while direct pooling of head-region interactions achieves an acceptable result.Source: §4.4 · pp. 8–9
• DINOv2 pre-training improves AUC by 6.8% over DeiT pre-training, and a 100 : 1 : 1 loss-weight ratio achieves the best reported performance.The authors report that self-supervised pre-training improves performance and that the loss ratio keeps task losses at the same scale.Source: §4.4 · p. 9

4.5 Visualization Results

• Visualizations show spatial guidance selecting patches overlapping the face, with layer 6 interactions reflecting local regions and layer 12 interactions indicating global relations and roughly locating gaze targets.The paper also reports VideoAttentionTarget visualizations in Appendix C.Source: Figure 5 · p. 10, §4.5 · p. 10

• ViTGaze combines a 4D interaction encoder, 2D spatial guidance, and lightweight prediction heads, with relative decoder parameters below 1%.Source: §5 · p. 10
• The paper presents ViTGaze as a concise single-modality framework that achieves high gaze-following performance with pre-trained vision transformers.Source: §5 · p. 11
• Predictions may become ambiguous in complex environments with occlusions; integrating gaze dynamics and global saliency is left for future work.Source: §5 · p. 11, Figure 6 · p. 11

• GazeFollow images are increased to 518 × 518 in the last epoch.Source: Appendix A · p. 11
• GazeFollow training uses head-box jittering, color jittering, resizing and cropping, horizontal flips, rotations, and random scene-patch masking.Source: Appendix A · p. 11
• VideoAttentionTarget is fine-tuned from the GazeFollow-pretrained model for one epoch at a fixed learning rate of 1.0 × 10−4, without a scheduler.Source: Appendix A · p. 11

• Self-supervised pre-trained ViTs produce more accurate and finer-grained gaze predictions than supervised pre-training on GazeFollow.Source: Appendix B · p. 11
• The supervised pre-trained model tends to produce more false-positive masks in salient regions outside the person’s field of view.Source: Appendix B · p. 11
• Prediction performance consistently aligns with the models’ representation-learning outcomes on ImageNet.Source: Appendix B · p. 11
• Among the tested pre-training techniques, DINOv2 yields the highest precision and granularity in gaze-target predictions.Source: Appendix B · p. 11

• For each video clip, the visualization samples one image every eight frames and presents five consecutive sampled images.Source: Appendix C · p. 11
• ViTGaze performs effective gaze following in video sequences despite using no explicit time-series modeling.Source: Appendix C · p. 11
• Predicted heatmap responses reflect confidence in the person’s gaze direction and show no discernible activation when the target is outside the frame.Source: Appendix C · p. 12

Declarations

• The authors’ code is available on GitHub, and the analyzed datasets are GazeFollow and VideoAttentionTarget [20].The passage provides the repository URL: https://github.com/hustvl/ViTGaze.Source: Declarations · p. 12
• The authors declare no competing interests.Source: Declarations · p. 12
• The work was supported by the National Science and Technology Major Project (No. 2022YFB4500602).Source: Declarations · p. 12
• The listed authors contributed to ideas, experiments, manuscript revision, research framework, and writing, and all approved the final manuscript.YS conceived the initial ideas, conducted experiments, and drafted the paper; XW, JY, WL, JZ, and XX contributed revisions, framework refinement, experimental design, and writing guidance.Source: Declarations · p. 12