Source-linked AI summary
Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders
Fiona Ryan, Ajay Bati, Sangmin Lee, Daniel Bolya, Judy Hoffman, James M. Rehg
TL;DR
Gaze target estimation must combine a person’s appearance with scene context, while prior approaches often use complex multi-branch pipelines and small-scale annotated datasets. Gaze-LLE decodes gaze from a frozen DINOv2 encoder using a person-specific prompt, and the paper reports state-of-the-art results across four benchmarks.
Problem
Prior gaze systems use complex, specialized multi-branch architectures and rely on small-scale annotated datasets, motivating whether foundation models can benefit gaze estimation.
Source: §1 · p. 1Method
Results
Gaze-LLE achieves state-of-the-art results across four benchmarks.
Source: §5 · p. 8Takeaways & Limitations
The paper concludes that frozen foundational encoders can support gaze target estimation with a streamlined, adaptable alternative to complex multi-branch approaches.
Source: §5 · p. 8For comparisons with detection methods, inference-time matching uses ground-truth gaze labels and the metrics do not penalize extra incorrect predictions or assess precision.
Source: §8 · p. 15
Abstract
from arXiv · showhide
We address the problem of gaze target estimation, which aims to predict where a person is looking in a scene. Predicting a person's gaze target requires reasoning both about the person's appearance and the contents of the scene. Prior works have developed increasingly complex, hand-crafted pipelines for gaze target estimation that carefully fuse features from separate scene encoders, head encoders, and auxiliary models for signals like depth and pose. Motivated by the success of general-purpose feature extractors on a variety of visual tasks, we propose Gaze-LLE, a novel transformer framework that streamlines gaze target estimation by leveraging features from a frozen DINOv2 encoder. We extract a single feature representation for the scene, and apply a person-specific positional prompt to decode gaze with a lightweight module. We demonstrate state-of-the-art performance across several gaze benchmarks and provide extensive analysis to validate our design choices. Our code is available at: http://github.com/fkryan/gazelle .
1. Introduction
Gaze target estimation matters for understanding human behavior, yet prior systems rely on specialized, often multi-branch models trained on small-scale annotated datasets. Gaze-LLE uses a frozen foundation-model encoder and a person-specific prompt, aiming to simplify that design while achieving strong benchmark performance.
1. Introduction
- Prior gaze systems commonly fuse head and scene branches, sometimes adding depth, pose, or other cues, and are limited to small-scale annotated datasets.The multi-branch paradigm traces to Recasens et al., while later work incorporated additional modalities.Source: §1 · p. 1, §2 · p. 2
- Gaze-LLE uses a frozen foundation encoder and a novel decoder, demonstrating that pretrained visual features can support state-of-the-art gaze estimation.The authors identify this as the first demonstration of leveraging frozen foundational feature encoders for gaze target estimation.Source: §1 · pp. 1–2
- Gaze-LLE achieves state-of-the-art performance across three main gaze benchmarks with approximately 5% of the trainable parameters used by most prior methods.The paper also reports strong cross-dataset performance without finetuning and state-of-the-art results in under 1.5 GPU-hours.Source: §1 · p. 2
2. Related work
- Replacing existing gaze models’ scene encoders with DINOv2 does not improve performance, whether its parameters are frozen or trainable.The comparison replaces the scene encoder in three open-source methods and evaluates on GazeFollow.Source: Table 1 · p. 2, §1 · p. 2
- Gaze-LLE prompts its unified decoder with head position, extracting scene cues there instead of using a separate head branch, fusion modules, and multitask objectives.Prior multi-branch systems use the head representation as input to the scene branch and carefully fuse representations across branches.Source: §2 · pp. 2–3
- Compared with Tafasca et al., Gaze-LLE removes the separate head branch and achieves higher accuracy on all datasets with two orders of magnitude fewer learned parameters.Tafasca et al. train their transformer backbone end-to-end alongside a head branch and large DPT decoder.Source: §2 · p. 3
3. Gaze-LLE
Gaze-LLE uses a frozen visual encoder and a lightweight decoder to predict gaze targets from a scene and a person’s head location. Its design analysis finds that adapting frozen DINOv2 features with late head integration and transformer decoding works better than a direct backbone swap into prior architectures.
3.1. Model Architecture
- Given an RGB image and head box, Gaze-LLE extracts frozen scene features, adds a learned head-position embedding after encoding, and uses transformer layers to predict a gaze heatmap and optionally in/out status.The optional in/out task token is used for benchmarks that require classifying whether the target is in frame.Source: §3 · p. 3, §3.1 · p. 3, Figure 2 · p. 4
- Training uses pixel-wise binary cross-entropy for the heatmap, with a Gaussian target around each annotated gaze point; in/out benchmarks add a weighted binary cross-entropy loss.The backbone remains frozen, and the ViT-B model has about 2.8M learnable parameters.Source: §3.1 · p. 4
3.2. Key Design Decisions for Foundation Models
- A direct DINOv2 swap underperformed the original backbones, while late head integration and transformer decoding improved frozen-feature use; with transformer decoding, a head branch made nearly no difference.The transformer decoder performed better than a same-parameter convolutional stack, and the head-branch comparison was nearly unchanged only with transformer decoding.Source: §3.2 · pp. 4–5, Table 2 · p. 5
4. Experiments
Across four gaze benchmarks, Gaze-LLE reports strong performance, while the analysis examines backbone portability and head-prompt design choices.
4. Experiments
- Experiments cover GazeFollow, VideoAttentionTarget, ChildPlay, and GOO-Real, using heatmap AUC and L2, plus in/out-of-frame AP on two benchmarks.GazeFollow reports both average and closest-annotation L2; benchmark-specific protocols are used for the other datasets.Source: §4 · p. 5
4.1. Main Results
- Gaze-LLE achieves state-of-the-art AUC and L2 on GazeFollow and VideoAttentionTarget; ViT-B beats Tafasca et al. using ≈2% of its learned parameters.The method also achieves strong results in fewer GPU hours than prior approaches.Source: §4.1 · p. 5, Table 3 · p. 6, Figure 4 · p. 7
- Both ViT-B and ViT-L achieve state-of-the-art ChildPlay results across AUC, L2, and AP; P.Head may diverge because detections can be inaccurate.The authors report that P.Head tends not to correlate with the other metrics.Source: Table 4 · p. 6, §4.1 · p. 6
- Without finetuning, Gaze-LLE generalizes across VAT, GOO-Real, and ChildPlay, surpassing Tonini et al.’s [64] method with unsupervised domain adaptation.The largest performance drop occurs on GOO-Real, where both the retail setting and annotation scheme differ.Source: Table 5 · p. 6, §4.1 · pp. 6–7, Figure 3 · p. 7
4.2. Analysis
- Gaze-LLE works with different pretrained backbones; DINOv2 performs best, CLIP is also strong, and the framework can adapt to future encoders.The backbone comparison is evaluated on GazeFollow.Source: Table 6 · p. 7, §4.2 · p. 7
- A head-position token can approach the default position embedding with suitable attention and decoding choices, and may support multiple queried heads as separate tokens.The authors note this token design may benefit future heavier gaze decoders.Source: Table 7 · p. 7, §4.2 · pp. 7–8
- Injecting the prompt in later decoder layers causes only small performance drops; YOLOv5 head detections also cause almost no degradation.The reported setup confines person-specific computation to the final transformer layer and two convolutional layers.Source: Table 8 · p. 8, §4.2 · p. 8
- Without a head prompt, the model can predict gaze in single-person scenes, but the prompt identifies whose gaze to decode in multi-person scenes.The authors infer that the scene representation implicitly detects heads and reasons about potential gaze targets.Source: §4.2 · p. 8, Figure 5 · p. 8
5. Discussion
Gaze-LLE uses a frozen foundation encoder with a learned head-prompting decoder, but its performance and efficiency remain tied to encoder choice and size.
5. Discussion
- Because the encoder is frozen, Gaze-LLE’s performance is inherently tied to encoder quality, favoring large, diverse encoders trained with dense objectives.The method is reported to exceed 50 fps on an Nvidia RTX4090, but reliance on a large encoder may challenge embedded systems.Source: §5 · p. 8
- The paper demonstrates frozen foundational feature encoders for gaze estimation and reports state-of-the-art results across four benchmarks.Its proposed architecture uses a single frozen DINOv2 encoder with a learned gaze decoder and head prompting.Source: §5 · p. 8
6. Integration of DINOv2 into Existing Methods
Replacing the scene encoders of three existing gaze methods with DINOv2 did not surpass their original backbones, even when DINOv2 was fully trained. The section also tests larger scene inputs and finds no clear overall gain.
Chong et al.
- Chong et al.’s integration tested RGB-only frozen DINOv2, a learned patch projection that also receives head position, and full encoder training.The DINOv2 feature map was pooled and projected to the size expected by the existing method.Source: §6 · p. 12
DINOv2 integration results
- Across Chong, Miao, and Gupta integrations, fully training DINOv2 performed best among tested configurations but still underperformed each method’s original backbone.For Chong et al., regular training also outperformed LoRA.Source: §6 · pp. 12–13, Table 9 · p. 12, Table 10 · p. 13
Miao et al.
- Miao et al.’s integration retained depth as an input and changed DINOv2’s patch projection to accept five channels.Full DINOv2 training gave the best tested result, though it remained below the original ResNet50-based method.Source: §6 · pp. 12–13
Gupta et al.
- Gupta et al.’s image-only integration replaced EfficientNet-B1 with DINOv2 and added a projection to the model’s 64-dimensional internal representation.Training DINOv2 performed better than freezing it but still underperformed Gupta et al.’s original method.Source: §6 · p. 13
Input Size
- Increasing scene input size to 448 × 448 showed no clear gains across the tested original methods and DINOv2 variants.The paper uses the larger input because its main method lacks a separate, higher-resolution head crop.Source: §6 · p. 13, Table 12 · p. 13
7. Experiment Details for Section 3.2
These experiments compare early versus late head-position integration, scene-only versus head-and-scene features, and transformer versus convolutional decoding. Both decoders produce a 64 × 64 gaze heatmap, and the models are trained under shared GazeFollow settings.
Scene & Head Backbones
- The experiments use frozen DINOv2 ViT-Base features at 32 × 32 resolution, with a 448 × 448 scene input and a 224 × 224 head input when using both branches.The head feature map is upsampled before concatenation with scene features; the scene-only variant uses scene features alone.Source: §7 · p. 14
Head Position Integration
- Early head-position integration learns a four-channel patch projection, while late integration concatenates a downsampled head-position map with the encoded features.Early integration keeps the rest of DINOv2 frozen; late integration leaves its projection unchanged.Source: §7 · p. 14
Decoder
- Both decoder variants produce a 64 × 64 gaze heatmap from the integrated features.The convolutional decoder uses six convolutional layers; the transformer decoder uses one transformer layer followed by two shallow convolutional layers.Source: §7 · p. 14
Training Details
- Models are trained on GazeFollow for 15 epochs with Adam, cosine scheduling, an initial learning rate of 1e-3, and batch size 60.Training uses random crops, flips, and bounding-box jitter.Source: §7 · p. 14
8. Comparison to Detection Methods
Detection-based gaze methods predict head boxes and gaze targets jointly, unlike the traditional setting that takes a person’s head box as input. Their reported gaze metrics use ground-truth gaze during inference-time matching, so the paper re-evaluates Tonini et al. without that information and observes a significant performance drop.
Formulation
- Detection-based methods predict a fixed set of head-and-gaze instances, whereas the traditional formulation receives a particular person’s head box and predicts that person’s gaze heatmap.HGTTR uses one-to-one Hungarian matching during training; Tonini et al.’s matching cost also includes a gaze-vector term.Source: §8 · pp. 14–15
Evaluation Setting
- Ground-truth-gaze matching can select among multiple predicted gaze targets for a person, so those metrics do not penalize extra incorrect predictions.The altered matching experiment was not an exhaustive adaptation of Tonini et al.’s method to the paper’s setting.Source: §8 · pp. 15–16
- The detection-based metrics are not a direct comparison with methods following the traditional formulation because ground-truth gaze is used to retrieve predictions at inference.The matching may favor gaze similarity without ensuring that the predicted person box corresponds to the ground-truth person.Source: §8 · p. 15
Quantitative Results
- Removing ground-truth gaze from Tonini et al.’s inference-time matching cost causes a significant performance drop on GazeFollow.The altered cost uses bounding-box overlap and class similarity; the table distinguishes this evaluation from ground-truth-gaze matching.Source: §8 · pp. 15–16, Table 14 · p. 15
9. Runtime Analysis
The ViT-Base model runs in 15 ms on an RTX 4090, and its runtime increases only modestly when estimating gaze for more people. The paper compares this speed with existing methods while accounting for auxiliary-model and head-detector runtimes where specified.
Inference Speed
- 15 ms (about 66 fps) is the ViT-Base model’s inference time on one NVIDIA RTX 4090 GPU.It is faster than Miao et al.’s approach, while Chong et al.’s approach is faster but has a significant performance drop relative to Gaze-LLE.Source: §9 · p. 16
Multi-Person Scaling
- Runtime rises from 15 ms for one person to 19 ms for ten people because the DINOv2 scene encoder accounts for over 95% of computation and runs once per image.The cross-attention token variant scales even better, with a slight performance decrease.Source: §9 · p. 16
- The default method is faster than Tonini et al.’s 2D method for up to seven people, while the token variant is faster for every tested count.The reported runtimes include a YOLOv5 head detector for Gaze-LLE and a DPT-Hybrid depth estimator for Tonini et al.’s 3D variant.Source: §9 · p. 16
10. Comparison to ViTGaze
Gaze-LLE uses frozen DINOv2 features, while ViTGaze trains a smaller ViT-S backbone end-to-end. The comparison also highlights matching and duplicate-instance issues in Tonini et al.’s model.
10. Comparison to ViTGaze
- Tonini et al.’s matching can select a gaze-close instance whose predicted box belongs to the wrong person, and can produce multiple instances for one person.Without ground-truth gaze, the model cannot determine which duplicate instance is best.Source: Figure 6 · p. 17
- Gaze-LLE uses frozen DINOv2 features out-of-the-box; ViTGaze trains a smaller ViT-S backbone end-to-end and may better suit on-device use.Gaze-LLE may suit settings where general-purpose features are precomputed for several downstream tasks; ViTGaze uses 512 input resolution and additional postprocessing for L2 predictions.Source: §10 · pp. 16–17
11. Performance with Estimated Head Bounding Boxes
Gaze-LLE shows almost no performance degradation with YOLOv5 head detections rather than ground-truth boxes. Its coarse head-position map makes it less dependent on an exact head crop.
11. Performance with Estimated Head Bounding Boxes
- Gaze-LLE achieves consistent results with out-of-the-box YOLOv5 head detections instead of ground-truth boxes.The reported comparison pairs a YOLOv5 detector trained on CrowdHuman with Gaze-LLE; the authors contrast this with reported performance drops for 2-stream methods.Source: §11 · p. 17, Table 15 · p. 17
- Gaze-LLE’s coarse, down-sampled head-position map is less dependent on an exact head crop and works with out-of-the-box detections.The authors suggest future work could integrate head detection into the pipeline using the same frozen DINOv2 features.Source: §11 · p. 17
12. Reimplementation of Horanyi et al.
The authors reimplement Horanyi et al. because its reported metrics appear imbalanced and implementation details and code are limited. Their reproduction uses a FoV-map pipeline with auxiliary depth and 3D gaze-angle models, and training the gaze-angle model performs better in their experiments.
12. Reimplementation of Horanyi et al.
- The authors reimplement Horanyi et al. because reported metrics appear imbalanced and the method’s large gains are difficult to explain from its shared components.The original work has no published code and limited implementation details, so the authors follow some choices from Chong et al.’s codebase.Source: §12 · pp. 17–18
- The runtime analysis compares inference time with performance and examines multi-person scaling against detection methods on one RTX 4090 GPU.The supplied caption states the analysis scope but does not provide specific outcomes.Source: Figure 7 · p. 18
- The reimplementation builds a person-specific 3D FoV map from estimated depth and gaze angle, then encodes the map, depth, and image with a ResNet50-based scene encoder and gaze decoder.The auxiliary models are Gaze360 and Monodepth2; the FoV construction assumes the k-coordinate comes from normalized estimated depth.Source: §12 · p. 18
- Training the auxiliary gaze-angle model performs better than freezing it in the authors’ experiments on GazeFollow and VideoAttentionTarget.They report the trained-angle setting in the main paper.Source: §12 · p. 18, Table 16 · p. 18
13. Additional Ablation Studies
The ViT-Base ablations favor a 256-dimensional gaze module with three transformer layers: smaller dimensions reduce parameters and benefit performance, while deeper modules yield diminishing returns.
13. Additional Ablation Studies
- A 256-dimensional gaze module with 3 transformer layers is the selected default, balancing performance and learnable parameters.The caption reports diminishing returns as model dimension and layer count increase.Source: §13 · p. 18, Table 17 · p. 19
- Increasing the gaze module from 1 to 3 transformer layers yields sizeable performance gains, with performance largely stable beyond 3 layers.The model is competitive with prior work using only one transformer layer.Source: §13 · p. 18
- A smaller internal dimension benefits performance while reducing the number of learnable parameters.The architecture illustration identifies the FoV Map Generator, auxiliary gaze-angle and depth models, and the ResNet50-based encoder and decoder.Source: §13 · p. 18, Figure 8 · p. 19
14. LoRA Backbones
LoRA tuning brings limited improvement on GazeFollow, which the authors attribute to strong frozen-encoder features and performance already near the human inter-rater level. Additional visualizations show error modes involving turned-away, occluded, or blurred faces.
14. LoRA Backbones
- LoRA tuning yields limited improvements on GazeFollow.The authors update DINOv2 query and value projections using rank 16, and attribute the limited gains to strong frozen features and performance near the human inter-rater level.Source: §14 · p. 19, Table 19 · p. 19
15. Additional Visualizations & Failure Modes
- Predicted heatmaps and ground-truth gaze annotations are shown for GazeFollow, VideoAttentionTarget, ChildPlay, and GOO-Real.The visualizations pair each prediction’s heatmap and maximum point with the ground-truth annotations.Source: §15 · p. 19, Figure 9 · p. 20
- Errors are more likely when the person faces away, is heavily occluded, or has a blurred face or eyes.In turned-away or occluded cases, the model may select a potential target in the person’s field of view without refining it to the ground-truth target; these cases can also be inherently ambiguous. Face blur is more common in VideoAttentionTarget and ChildPlay, and temporal information is suggested as future work.Source: §15 · p. 21, Figure 10 · p. 21