Patch-level Gaze Distribution Prediction for Gaze Following

1. Introduction

• Gaze following locates a person's target in a scene or indicates that it lies outside the image.Source: §1 · p. 1
• Pixel-wise MSE regression to a Gaussian heatmap from one coordinate imposes a strict constraint despite annotator disagreement and uncertain gaze targets.Source: §1 · p. 1, Figure 1 · p. 1
• Prior approaches train heatmap and in/out outputs separately, overlooking their relationship; the paper instead treats outside gaze as a special target.Source: §1 · pp. 1–2
• PDP replaces the in/out branch with a patch-level gaze distribution, softening the pixel-level constraint and permitting variable patterns, including an outside token.Source: §1 · p. 2, Figure 2 · p. 2
• The model achieves super-human AUC on GazeFollow and significantly improves in/out prediction, with simultaneous training causing no loss on either task.Source: §1 · p. 2

2. Related Work

• Earlier gaze-following methods used one-hot patch classification or heatmap regression, and prior models treated in/out prediction as a separate task.One-hot patch classification can incur higher distance errors [7], while heatmap regression methods commonly used MSE [8] [37].Source: §2 · pp. 2–3

3.1. Overall Structure

• The model extracts features from the scene image, head-position mask, depth map, and cropped head, then uses inside tokens for heatmaps and all tokens for PDP.Its feature-extraction design generally follows VideoAtt [8], with an additional depth-map input based on DualAtt.Source: §3.1 · pp. 3–4, Figure 3 · p. 3

3.2. Gaze Distribution Feature Computation

• Patch attention gives each token global scene context, while temporal attention aggregates gaze-distribution features across video frames.The model aggregates the original input features with attention-weighted values and a residual connection.Source: §3.2 · p. 4

3.3. Gaze Prediction

• PDP replaces the separate in/out head with token-level gaze probabilities, whose inside-token scores sum to the in-image probability.Training uses MSE for the heatmap and KL-divergence for the predicted and ground-truth patch distributions.Source: §3.3 · pp. 4–5

3.4. Ground Truth Patch Distribution Creation

• The ground-truth patch distribution takes each patch’s maximum heatmap score, normalizes across patches, and assigns outside targets entirely to the outside token.Its variable patterns can place high responses in multiple patches, encouraging multimodal heatmap clusters for ambiguous targets.Source: §3.4 · p. 5

4.1. Datasets

• GazeFollow provides over 122K images and ten test annotations per person, whereas VideoAttentionTarget contains gaze-following videos and single annotations per person.Evaluation uses AUC and distance metrics for heatmaps, and average precision for in/out prediction.Source: §4.1 · p. 5, §4.2 · p. 5

4.3. Results

• On GazeFollow, PDP outperforms VideoAtt depth across all metrics and newer models on AUC, achieving super-human AUC without depth input.The authors note a smaller advantage on distance metrics, which they suggest may reflect PDP's emphasis on gaze distributions rather than a specific coordinate.Source: Table 1 · p. 6, §4.3 · p. 6
• On VideoAttentionTarget, the full model improves target-prediction metrics and achieves significantly higher in/out AP than all SOTA models, closer to human performance.The authors interpret this as evidence that PDP bridges the target-prediction and in/out-prediction subtasks.Source: Table 2 · p. 6, §4.3 · p. 6

4.4.1 Ablation Study

• Removing patch attention causes an obvious VideoAttentionTarget performance drop; replacing KL divergence with MSE or BCE also lowers performance, with BCE outperforming MSE.Replacing PDP with the original in/out head causes a significant drop, even below VideoAtt depth; extra 2D gaze-direction prediction brings no GazeFollow improvement.Source: §4.4.1 · pp. 6–7, Table 3 · p. 7

4.4.2 Performance Regarding Annotation Variance

• PDP's AUC advantage appears above the 50% annotation-variance quantile, while performance is highly close to VideoAtt below it.The authors report that depth-related gains are distributed evenly across upper and lower quantiles, supporting their attribution of PDP's gain to images with larger annotation variance.Source: §4.4.2 · pp. 7–8, Figure 7 · p. 8
• For high-variance examples, PDP predicts multimodal heatmaps that are more aligned with group-level annotations than VideoAtt depth's unimodal Gaussian predictions.The variance score is the mean distance of annotations from their average coordinate; the test set is split into ten quantile groups.Source: §4.4.2 · pp. 7–8, Figure 8 · p. 8

4.4.3 Comparison w/ Original In/Out Prediction Task

• Unlike the prior separate-task setup, PDP integrates target and in/out prediction without performance loss, allowing one model to be trained instead of separate dataset-specific versions.The prior VideoAtt setup omitted BCE during GazeFollow training to obtain SOTA heatmap performance; jointly training both losses caused a large heatmap-performance drop in their experiments.Source: §4.4.3 · p. 8, Table 4 · p. 8

5. Conclusions

• PDP predicts a patch-level gaze distribution using inside feature tokens and an outside token, regularizing heatmap regression and improving performance, especially on images with larger annotation variance.The conclusion also reports significantly higher AP and describes PDP as a simpler training approach for in-the-wild images and videos.Source: §5 · p. 8

Patch-level Gaze Distribution Prediction for Gaze Following: Supplementary Material

• The supplement provides implementation details and explains the feature-extraction module's structure.These topics are covered in Sections S1 and S2.Source: Patch-level Gaze Distribution… · p. 11
• It reports alternatives for creating patch distributions and analyzes consistency between predicted heatmaps and patch distributions.These topics are covered in Sections S3 and S4.Source: Patch-level Gaze Distribution… · p. 11
• It compares the model with VideoAtt with and without depth input and presents example failure cases.These topics are covered in Sections S5 and S6.Source: Patch-level Gaze Distribution… · p. 11

S1. Implementation Details

• Inputs are resized to 224×224, and ResNet-50 scene and head backbones produce 1024 × 7 × 7 features; the encoder reduces channels from 2048 to 512, yielding 50 patch tokens.The head and scene backbones use Eyediap and Places pretrained weights, respectively; the ground-truth heatmap uses σ = 3.Source: S1. Implementation Details · p. 11
• Training first runs on GazeFollow until convergence, then finetunes on VideoAttentionTarget with five-frame sequences and the network after the patch-attention module trainable.The weights through the patch-attention module are frozen during finetuning.Source: S1. Implementation Details · p. 11
• Adam is used with a 2.5e-4 learning rate on GazeFollow and 1e-4 on VideoAttentionTarget, with dataset-specific decay schedules and batch sizes.Batch sizes are 80 for GazeFollow and 16 for VideoAttentionTarget.Source: S1. Implementation Details · p. 11

• The scene branch extracts scene features from the image, head-position mask, and normalized depth map, while the head branch extracts features from the person’s head crop.Source: S2. Structure of the Feature… · p. 11
• An attention layer uses the pooled head feature, downsampled head mask, and depth map to weight scene features before concatenating them with the head feature.Source: S2. Structure of the Feature… · p. 11
• Unlike VideoAtt’s separate encoders, the model uses one shared encoder after feature concatenation to serve both prediction tasks.Source: S2. Structure of the Feature… · p. 11

• A 4 × 4 patch grid substantially reduced GazeFollow performance, while a 14 × 14 grid reduced VideoAttentionTarget performance.The authors suggest the coarser encoding makes fine-grained target estimation difficult; with 14 × 14 patches, the larger number of inside tokens may make prediction difficult alongside the outside token.Source: S3. Ablations of the Ground Truth… · p. 12
• One-hot patch classification caused a significant performance drop, with little improvement over the VideoAtt depth model.Its hard label cannot cover adjacent patches when an annotation lies near a patch boundary, limiting multi-modal heatmaps.Source: S3. Ablations of the Ground Truth… · p. 12, Figure S2 · p. 13
• Average pooling retained good performance but caused a slight drop across all metrics compared with max pooling.Source: S3. Ablations of the Ground Truth… · p. 12
• Max pooling creates higher confidences in multiple patches that align with group-level human annotations and show the best potential for multi-modal outputs.Source: Table S1 · p. 12, S3. Ablations of the Ground Truth… · p. 13

• For comparison, the authors derive a patch distribution from each predicted heatmap and compare it with the patch head’s output using Bhattacharyya coefficient and Jensen-Shannon divergence.They use the symmetric Jensen-Shannon divergence rather than KL divergence because there is no ground-truth patch distribution for this comparison.Source: S4. Consistency between the Heatmap… · p. 13
• The patch-head distribution and the distribution derived from the predicted heatmap show high consistency on GazeFollow and VideoAttentionTarget.The reported pattern is high Bhattacharyya coefficients and low Jensen-Shannon divergence.Source: S4. Consistency between the Heatmap… · p. 13, Table S2 · p. 13

• With or without a depth map, the model produces heatmaps better aligned with human annotations than VideoAtt.Source: S5. Model Performance with and without… · p. 14
• Without depth, the model can assign confidence to objects inconsistent with human gaze or miss potential gazed objects.Source: S5. Model Performance with and without… · p. 14
• Depth provides scene-structure information that makes it easier for the model to infer potential gazed objects.Source: S5. Model Performance with and without… · p. 14

S6. Example Failure Cases

• Subtle eye orientation inconsistent with head pose can lead to incorrect targets, while the model may favor a person’s head over the actual target.The authors suggest frequent head targets in the dataset may contribute; using cropped heads without cropped eye regions also sacrifices some accuracy.Source: S6. Example Failure Cases · p. 14
• Under input uncertainty, multiple heatmap clusters can place the maximum-response target away from the annotation even when responses remain in annotated regions.The authors suggest performance may improve with more annotations.Source: S6. Example Failure Cases · p. 14
• Figure S4 shows examples from GazeFollow in rows 1–2 and VideoAttentionTarget in rows 3–4, with predictions in yellow and ground truth in red.GazeFollow average annotations are shown in blue.Source: Figure S4 · p. 16