GazeDETR: Gaze Detection using Disentangled Head and Gaze Representations

• End-to-end gaze detection can support applications including human-robot interaction, social interaction analysis, and digital phenotyping.Source: §1 · p. 1
• A single decoder learns joint representations for head localization and gaze prediction, although the tasks require different information: local for heads, and local plus global for gaze.Source: §1 · p. 1
• GazeDETR uses parallel decoding, assigning head localization to the first decoder and gaze prediction and interaction classification to the second.Source: §1 · p. 2
• The head predictor uses local information, while the gaze decoder uses scene context; their attention maps are designed to localize heads and highlight heads and salient areas for gaze prediction.Source: §1 · p. 2, Figure 2 · p. 2
• GazeDETR reports new state-of-the-art results across different metrics on three evaluation benchmarks.Source: §1 · p. 2

• Gaze target detection predicts a specific person's focus of attention, represented as a heatmap or exact location, with an inside-or-outside-frame classification extension.Source: §2 · p. 2
• Unlike multi-branch approaches that require head locations as input, end-to-end models need only an input image.Source: §2 · p. 2
• Multi-branch methods use the whole image to find salient regions and a head location to infer gaze, requiring head locations and multiple forward passes.Source: §2 · p. 2
• End-to-end approaches jointly predict head and gaze locations from an image, but prior models use one decoder whose unified representation produces inconsistent attentive fields across subtasks.Source: §2 · p. 2
• GazeDETR uses two disentangled decoders specialized for head localization and gaze prediction, and reports superior performance across benchmarks.Source: §2 · p. 2

3. GazeDETR Architecture

• GazeDETR combines a human head predictor, a global scene feature encoder, and a gaze decoder conditioned on learned head queries.The encoder combines head and scene features for identifying gaze directions.Source: §3 · p. 3

3.1. Motivation

• A single decoder must represent both local head information and global scene information, while DETR-like attention maps highlight object extremities.This motivates separate decoders for head localization and gaze prediction.Source: §3.1 · p. 3, Figure 4 · p. 3

3.2. Human head predictor

• The human head predictor uses a backbone, transformer encoder and decoder, and feed-forward networks to classify heads and regress their bounding boxes.Its learned spatial queries encode head-location priors and are reused by the gaze decoder.Source: §3.2 · p. 3

3.3. Global scene feature encoder

• A separate scene encoder enriches two feature maps and cross-projects them before concatenating head and scene features into global memory.The memory supplies features to the gaze decoder.Source: §3.3 · p. 3

3.4. Gaze decoder

• The gaze decoder reuses the head predictor’s spatial queries in self- and cross-attention to leverage learned head-location priors.The first cross-attention layer adds the head query to the self-attention output and uses global memory with positional encoding as its key.Source: §3.4 · p. 4
• Its cross-attention separately interacts with head and scene features and also uses previous decoder embeddings to identify gaze locations.The head and scene features are concatenated rather than added to preserve their separate interactions.Source: §3.4 · p. 4

3.5. Loss Calculation

• Hungarian matching uses head-class and box-location costs; after matching, training computes the loss using binary cross-entropy, gaze L1, and heatmap L2 terms.The box term combines L1 and generalized intersection-over-union losses.Source: §3.5 · p. 4
• Adding in/out, gaze, and heatmap losses to matching produced overlapping head predictions despite different predicted gaze directions.This finding explains why the matching and optimization cost functions differ.Source: §3.5 · p. 4

4.1. Datasets and Evaluation Metrics

• The three benchmarks provide gaze instances with head boxes, gaze points, and in/out labels; GazeFollow has 122k training images, VideoAttentionTarget 71k frames, and ChildPlay 120k frames.Head localization uses mAP with IoU above 0.5 and gaze L2 distance below 0.15 for a true positive; gaze detection uses AUC, distance measures, and AP for in/out classification.Source: §4.1 · pp. 4–5

4.2. Implementation Details

• The architecture combines a head predictor, global scene feature encoder, and gaze decoder; training uses GazeFollow followed by fine-tuning on VideoAttentionTarget and ChildPlay.The head predictor supplies head embeddings to the scene encoder, which forms global memory for the gaze decoder.Source: §4.2 · p. 5

4.3. Comparison with the State-of-the-Art

• GazeDETR achieved state-of-the-art results across all metrics on GazeFollow, VideoAttentionTarget, and ChildPlay, with especially notable gains among end-to-end gaze-detection models.Source: §4.3 · p. 5, Table 1 · p. 6, Table 2 · p. 6
• Against multi-branch models, GazeDETR improved AUC by almost 1% and reduced GazeFollow Avg. Distance (0.101 vs 0.113) and Min. Distance (0.055 vs 0.056).Source: §4.3 · p. 5, Table 1 · p. 6
• Compared with HGTTR, GazeDETR improved GazeFollow AUC by almost 4%; on VideoAttentionTarget, its L2 distance was 0.098 vs 0.137 and mAP gained 17.3.Source: §4.3 · p. 5, Table 1 · p. 6

• Removing the global scene feature encoder produced the lowest performance across metrics; adding it improved gaze detection, supporting the need for global information.Source: Table 3 · p. 6, §4.4 · p. 6
• Using a unique learnable gaze query reduced mAP by 2%, suggesting head queries encode spatial information useful for locating corresponding gaze targets.Source: Table 3 · p. 6, §4.4 · p. 6
• Heatmap-only variants performed worst; adding heatmap prediction yielded the highest AUC, while BM1 performed best overall and matched heads without gaze outputs.Adding gaze outputs to bipartite matching produced different gaze predictions for the same head and reduced mAP in BM4, BM5, and BM6.Source: Table 4 · p. 7, §4.4 · p. 7
• Increasing head-query count improved all metrics, especially mAP; increasing decoder layers had a similar effect.Source: Table 5 · p. 7, §4.4 · p. 7
• ResNet-101 brought only minimal gains, with AUC 0.953, Avg. Dist. 0.099, Min. Dist. 0.054, and mAP 0.549.Source: §4.4 · p. 7

4.5.1. Cross-attention maps of the decoders

• Removing GSFE left variants A and B less able to use relevant visual context, producing incorrect gaze predictions and lower mAP.With GSFE, variants C and D more effectively used long-range dependencies to identify gaze targets.Source: §4.5.1 · p. 7
• Variant D’s head predictor localized attention within predicted head boxes, while its gaze decoder highlighted candidate salient regions and localized attention near the eye region.The authors attribute the gaze decoder’s use of head queries to its cross-attention reformulation.Source: §4.5.1 · p. 8

4.5.2. Bounding box and gaze prediction

• Qualitative examples show predicted bounding boxes and gaze points on images containing different numbers of people.Source: §4.5.2 · p. 8

4.5.3. Limitations

• Gaze direction predictions occasionally fail with slightly occluded eyes, conflicting eye and head directions, or small heads.The authors hypothesize that the model implicitly encodes both eye and head information, and suggest explicit eye supervision as a possible improvement.Source: §4.5.3 · p. 8

• GazeDETR achieves new state-of-the-art results across datasets and outperforms models that rely on head annotations and off-the-shelf head detectors.Source: §5 · p. 8
• The authors attribute long training schedules in prior transformer gaze models to unified decoders learning entangled representations for head localization and gaze prediction.They characterize head localization as requiring local information, while gaze prediction requires both local and global information.Source: §5 · p. 8
• The paper reports extensive ablation studies to examine the contributions of key model components.Source: §5 · p. 8

• The supplementary material enlarges or rearranges Figures 1–5 to improve their visibility and readability.Source: Supplementary Material · p. 11
• The architecture comparison contrasts unified representations in existing approaches with GazeDETR’s disentangled representations and shared query between decoders.Source: Figure 1 · p. 11
• The overview depicts a head predictor for localization and a scene feature encoder supplying contextual information to the gaze decoder.Source: Figure 3 · p. 13
• Attention-map figures show predictions and ground truth, including DETR and conditionalDETR maps highlighting detected-object extremities and maps for GazeDETR variants.Source: Figure 2 · p. 12, Figure 4 · p. 14, Figure 5 · p. 15

• GazeDETR is initially trained on GazeFollow for 50 epochs.Source: §1 · p. 16
• The model is fine-tuned for 20 epochs on either VideoAttentionTarget or ChildPlay.Source: §1 · p. 16
• The authors state that they will release training and inference code and model checkpoints.Source: §1 · p. 16

2. GazeDETR variants

• Variants A and B omit the GSFE, whereas C and D use it to fuse head-predictor and scene-encoder features into gaze-decoder memory.For cross-attention, the fused memory is projected during Value formation, and the self-attention output is duplicated during Query formation.Source: §2 · p. 16
• Variants A and D use head-predictor queries, while B and C learn new gaze queries and require fewer parameters.Source: §2 · p. 16
• Variants A and B perform worse than C and D, which the authors attribute to the effectiveness of the GSFE module.The head predictors’ cross-attention maps localize around predicted heads across variants, while gaze-decoder maps differ substantially and affect final gaze predictions.Source: §2 · p. 16

3. Quantitative Results

• Variant D visualizations show positive predictions, with bounding boxes marking heads and lines indicating directions to predicted gaze locations.Source: §3 · p. 16

3.1. More Results on GazeFollow

• Figures 9–11 show extensive variant D predictions on GazeFollow, with head-predictor and gaze-query attention maps, predictions, and ground truth.Source: §3.1 · p. 16, Figure 9 · p. 20, Figure 10 · p. 21

3.2. More Results on VideoAttentionTarget

• Figures 12–14 show extensive variant D predictions on VideoAttentionTarget, with head-predictor and gaze-query attention maps, predictions, and ground truth.Source: §3.2 · p. 16, Figure 12 · p. 23, Figure 13 · p. 24

3.3. More Results on ChildPlay

• Figure 6 compares GazeDETR variants, while Figures 15–17 show variant D predictions on ChildPlay with attention maps, predictions, and ground truth.Source: §3.3 · p. 16, Figure 6 · p. 17, Figure 15 · p. 26