Source-linked AI summary
Gaze Target Estimation Anywhere with Concepts
Xu Cao, Houze Yang, Vipin Gunda, Zhongyi Zhou, Tianyu Xu, Adarsh Kowdle, Inki Kim, James M. Rehg
TL;DR
In-the-wild gaze estimation often relies on explicit intermediate inputs rather than flexible prompts to identify whom to analyze. This paper introduces prompt-conditioned end-to-end gaze estimation with GazeAnywhere, which achieves state-of-the-art results on Gaze-Co and is evaluated on an out-of-domain child social-communication dataset.
Problem
In-the-wild gaze estimation typically depends on explicit detections and pose or depth inputs in multi-stage pipelines, while flexible concept prompts for selecting the subject are lacking.
Source: §1 · pp. 1–2Method
The paper introduces PGE, conditioning end-to-end gaze-heatmap prediction on a text or visual prompt, and presents GazeAnywhere as its concept-driven model.
Source: §3.1 · pp. 2–3, §1 · p. 2Results
Takeaways & Limitations
The concept-based approach is a step toward more flexible, nonexpert-friendly gaze assessment in clinical applications.
Source: §6 · p. 8Two-stage gaze pipelines can fail in crowded scenes when upstream detection selects the wrong person or a box spanning multiple people, preventing reliable disambiguation.
Source: §14 · p. 16
Abstract
from arXiv · showhide
Estimating human gaze targets from images in-the-wild is an important and formidable task. Existing approaches primarily employ brittle, multi-stage pipelines that require explicit inputs, like head bounding boxes and human pose, in order to identify the subject of gaze analysis. As a result, detection errors can cascade and lead to failure. Moreover, these prior works lack the flexibility of specifying the gaze analysis task via natural language prompting, an approach which has been shown to have significant benefits in convenience and scalability for other image analysis tasks. To overcome these limitations, we introduce the Promptable Gaze Target Estimation (PGE) task, a new end-to-end, concept-driven paradigm for gaze analysis. PGE conditions gaze prediction on flexible user text or visual prompts (e.g., "the boy in the red shirt" or "person in point [0.52, 0.48]") to identify a specific subject for gaze analysis. This approach integrates subject localization with gaze estimation, and eliminates the rigid dependency on intermediate analysis stages. We develop a scalable data engine to generate Gaze-Co (Gaze Estimation with Concepts), a dataset and benchmark of 120K high-quality, prompt-annotated image pairs. We also propose GazeAnywhere, the first model designed for PGE. GazeAnywhere uses a transformer-based detector to fuse features from frozen encoders and simultaneously solves subject localization, in/out-of-frame presence, and gaze target heatmap estimation. GazeAnywhere achieves state-of-the-art performance on multiple PGE benchmarks, setting a strong baseline for this new problem even on a difficult out-of-domain, real-world clinical dataset. GazeAnywhere is open-sourced in github.com/IrohXu/GazeAnywhere.
1. Introduction · 2. Related Work · 3. Method
GazeAnywhere introduces promptable, end-to-end gaze estimation that identifies a subject from text or visual cues and predicts their gaze target without relying on intermediate head boxes or pose. The paper defines the task, provides 120K prompt-annotated training pairs, and reports state-of-the-art results across benchmarks, including an out-of-domain clinical dataset.
1. Introduction
- Prior gaze pipelines depend on intermediate head or face boxes and pose cues, so errors in crowded, poorly lit, or challenging scenes can cascade into system failure.The proposed concept-conditioned, end-to-end approach aims to remove these dependencies and identify the gaze subject from semantic prompts.Source: §1 · pp. 1–2
- The paper defines Promptable Gaze Target Estimation and introduces a scalable data engine producing 120K high-quality subject-text-to-gaze alignment pairs, alongside GazeAnywhere.The task extends gaze estimation to unconstrained text prompting, and the model is presented as the first promptable concept-driven gaze estimator.Source: §1 · p. 2
- GazeAnywhere achieves state-of-the-art performance on several benchmarks, including an out-of-domain private dataset for autism children’s gaze target estimation.The supplied passages do not provide benchmark values or a results table.Source: §1 · p. 2
2. Related Work
- Prior gaze-following systems combine explicit cues such as head position, pose, expression, and depth, while existing end-to-end models cannot specify the subject at inference [58] [60] [45] [65] [18] [66] [68].Promptable detection and segmentation motivate using conceptual cues, while pretrained vision foundation models have also been adapted for gaze estimation [25] [42] [83] [9] [36] [60] [58].Source: §2 · p. 2
3.1. Promptable Gaze Target Estimation
- PGE estimates a selected person’s gaze heatmap from either a natural-language noun phrase or a visual spatial cue, jointly requiring subject identification, localization, and gaze estimation.Text descriptions can combine appearance, location, pose, and action to reduce ambiguity; the formulation excludes auxiliary subject boxes, pose keypoints, and depth maps.Source: §3.1 · pp. 2–3
3.2. GazeAnywhere Architecture
- GazeAnywhere fuses frozen image and text encoder features with a transformer detector, whose specialized tokens and decoders predict gaze heatmaps, head boxes, and in/out-of-frame presence.Separate head and presence tokens decouple localization from global in/out prediction; the gaze heatmap decoder upsamples spatial features to a 64 × 64 output.Source: §3.2 · pp. 3–4
3.3. Learning Objective
- Training uses a weighted multitask objective combining pixel-wise BCE for gaze heatmaps, focal loss for gaze presence, and L1 plus generalized IoU losses for head localization.The heatmap target places a 2D Gaussian with σ = 3 at the ground-truth gaze location.Source: §3.3 · pp. 4–5
4. Gaze with Concept (Gaze-Co) Dataset
The paper introduces Gaze-Co, a large-scale concept-annotated dataset and benchmark for promptable gaze estimation, built with a scalable human-in-the-loop data engine.
4.1. Data Engine
- The data engine aligns and filters source annotations, generates structured concept phrases, and verifies them through an iterative human-in-the-loop process.It uses head-box size, area, image-ratio, and focus thresholds; concepts encode attributes, position, action, and pose. Review checks consistency, completeness, and privacy, with iteration until the observed error rate is ≤1%; Child-SC annotations are created manually by authorized annotators.Source: §4.1 · p. 5
4.2. Gaze-Co Dataset and Benchmarks
- Gaze-Co is the first large-scale promptable gaze dataset, with about 120K training samples from GazeFollow, VAT, and ChildPlay and converted concept test benchmarks.Records include a target head box, normalized gaze point, in/out-of-frame label, and concept phrase; the benchmark also evaluates out-of-domain Child-SC data, whose test prompts are fully human-verified.Source: §4.2 · pp. 5–6, §4 · p. 5
4.3. Metrics
- Evaluation uses heatmap AUC and pixelwise L2, with Avg L2 and Min L2 additionally reported for GazeFollow-Concept and AP for VAT-Concept, ChildPlay-Concept, and Child-SC.AUC compares predicted heatmaps with binary gaze target maps; pixelwise L2 measures distance from the heatmap peak to the ground-truth gaze point, while AP evaluates localization and in/out classification.Source: §4.3 · p. 6
5. Experiments
The experiments evaluate GazeAnywhere against two-stage gaze pipelines and other models, examine prompting, losses, and encoders, and test qualitative generalization and an AR agent in real-world videos.
5.1. Implementation Details
- Models were trained for 25 epochs with Adam, a cosine schedule starting at 1e-3, and batch size 128, then trained for five more epochs at 1e-5.Training used four NVIDIA H100 GPUs; inference used one NVIDIA L40S GPU, and the AR agent ran on DigiLens ARGO smartglasses.Source: §5.1 · p. 6
5.2. Main Results
- GazeAnywhere achieved state-of-the-art performance on all metrics across three public datasets and a challenging out-of-domain clinical dataset, and surpassed the compared VLMs in zero-shot gaze-point prediction.The VLM comparison included Qwen3-VL-8B and Gemini 2.5 Flash.Source: Table 2 · p. 7, §5.2 · p. 7, §5.3 · p. 7
5.3. Ablation Study
- Text prompting performed on par with visual prompting, and subject appearance and pose descriptions were the most critical text-prompt components.GazeAnywhere supports coordinate-based visual prompts as well as natural-language prompts.Source: Table 3 · p. 7, §5.3 · p. 7
- The head loss improved both gaze-target estimation and target-presence prediction, while the presence loss supported only auxiliary in/out prediction and did not help gaze estimation.The ablation retained the gaze heatmap loss in every model.Source: Table 4 · p. 7, §5.3 · p. 7
- With encoders frozen and only the projection layer, transformer detector, and decoder heads fine-tuned, DINOv3 achieved the best performance on nearly all metrics among the tested encoders.The comparison included CLIP, SigLIP 2, MetaCLIP 2, and DINOv3 with dino.txt.Source: Table 5 · p. 7, §5.3 · p. 7
5.4. Visualization
- Qualitative results showed robust gaze estimation in scenes with two or three people and crowded scenes with four or more, including successful examples from unseen Child-SC videos.The prompts described the subject’s appearance only.Source: §5.4 · p. 7, Figure 4 · p. 8
5.5. GazeAnywhere as Agent in AR
- On 10 real-world AR-glass videos, the GazeAnywhere Agent outperformed a raw single-MLLM solution on gaze-shift and eye-contact calculations measured by MAE per minute.The agent uses Gemini 2.5 to reason over spoken queries, calls Whisper v3 for transcription, and invokes GazeAnywhere for gaze tracing before analyzing the processed video.Source: §5.5 · pp. 7–8, Table 6 · p. 8
6. Discussion
Concept-based prompting makes gaze assessment more flexible for nonexperts, including potential clinical analysis of non-verbal behaviors implicated in developmental conditions such as autism. The unified end-to-end approach also aims to improve robustness by avoiding brittle staged subject identification.
6. Discussion
- Gaze target estimation could support healthcare analysis of non-verbal communication behaviors implicated in diagnosing and treating developmental conditions such as autism.The paper identifies clinical use as a practical application area.Source: §6 · p. 8
- Concept-based prompts identify subjects through natural-language attributes, offering flexible specification of behavioral analysis tasks beyond gaze assessment.The paper presents this as a first step toward convenient, broad behavioral-analysis applications.Source: §6 · p. 8
- The unified end-to-end architecture eliminates brittle staged subject identification, while OWLv2 achieves only a 70% detection accurate rate for child head and face detection in Child-SC.The paper frames robustness and ease of use by nonexperts as requirements for clinical AI applications.Source: §6 · p. 8
7. Conclusion
GazeAnywhere enables interactive gaze-target estimation with flexible open-vocabulary text prompts. The work introduces the PGE task and Gaze-Co benchmark, a tailored detector and learning objective, and a human-and-AI-in-the-loop data engine.
7. Conclusion
- GazeAnywhere achieves state-of-the-art results on Gaze-Co, with robustness validated on an out-of-domain dataset of child social communication videos.The dataset is described as challenging.Source: §7 · p. 8
- The work introduces Promptable Gaze Target Estimation and the Gaze-Co benchmark for open-vocabulary, prompt-based gaze analysis.Gaze-Co is presented as a benchmark for the new task.Source: §7 · p. 8
- GazeAnywhere combines a tailored transformer-based detector and learning objective with a human-and-AI-in-the-loop data engine to adapt existing datasets.Source: §7 · p. 8
Supplementary Material … 10. Dataset & Benchmark
The supplementary sections position GazeAnywhere as a promptable alternative to gaze pipelines without identity association, describe an agent for natural-language gaze analysis, and document Gaze-Co’s training and evaluation datasets. The benchmark includes human-verified concept-based in-domain splits and a manually annotated, privacy-protected out-of-domain clinical dataset.
8.1. Toward End-to-end Gaze Target Estimation
- GazeAnywhere addresses the identity-association gap in prior end-to-end gaze methods by jointly identifying a prompted person and estimating that person’s gaze target.Earlier methods could detect gaze for everyone in a scene but required separate modules or post-processing to associate a gaze with a specific individual, leaving cascaded detection errors possible.Source: §8.1 · p. 13
8.2. Future Application
- Promptable gaze estimation could let pediatricians or parents track a child’s gaze longitudinally without manually drawing head boxes, supporting home-based monitoring and potential earlier intervention.The authors also propose using GazeAnywhere Agent to post-process target-tracking videos and summarize high-level gaze behavior such as gaze shifts.Source: §8.2 · p. 13
9. GazeAnywhere Agent
- The GazeAnywhere Agent uses an MLLM to plan natural-language gaze requests, invoking Whisper-large-v3 for audio transcription and GazeAnywhere for target prediction, then revising analysis using visual feedback stored in memory.This workflow supports queries more complex than noun phrases, including deeper analysis of gaze behavior in video streams.Source: §9 · p. 13, Figure 6 · p. 14
10.1. Training Set
- Gaze-Co’s 119,525 training samples combine three gaze datasets and include target head boxes, normalized gaze points, in/out-of-frame labels, and compact concept phrases.GazeFollow contributes 69.6% (83,148 samples), VideoAttentionTarget 19.6% (23,481), and ChildPlay 10.8% (12,896); construction applies image-quality filtering, MLLM concept generation, and human-in-loop MLLM verification.Source: §10.1 · p. 13, Figure 7 · p. 14
- Gaze-Co’s training annotations label 13.9% of gaze targets out of frame and 86.1% in frame, while apparent subject categories are visual perceptions rather than verified identities.The largest categories are man (51.0%), woman (32.2%), and boy (8.2%).Source: §10.1 · p. 13, Figure 7 · p. 14
10.2. Concept-based In-domain Test Set
- Gaze-Co’s three concept-augmented test splits use a unified PGE schema and human-verified concepts, with additional Avg L2 and Min L2 metrics for GazeFollow’s multiple gaze annotations.The splits contain 2,436 GazeFollow-Concept records, 5,301 VAT-Concept records, and 1,238 ChildPlay-Concept records. Avg L2 measures distance to the mean human gaze point, while Min L2 measures distance to the nearest human point; apparent subject categories reflect perceived attributes, not verified identities.Source: §10.2 · pp. 14–15
10.3. Concept-based Out-of-domain Test set
- The Child-SC out-of-domain dataset comprises 326 videos from 40 children, sampled at 5 fps to yield 151,533 images, with target-person concepts manually annotated under privacy constraints.It captures child-clinician interactions in which toys elicit frequent, structured gaze shifts; privacy regulations preclude cloud-based MLLM processing.Source: §10.3 · p. 15
11. Baseline Details
The baselines first localize the prompted person with open-vocabulary detectors, then use the localized region with gaze-following models to predict a point of regard. Detector outputs are selected by confidence or text-match ranking, while gaze models produce heatmaps whose peaks give predicted locations.
11.1. Open-Vocabulary Detector (OVD)
- OVD baselines select the highest-confidence box for a text-prompted person, and comparisons include RexSeek, a 3B referring-expression-comprehension model [31].OVD models match prompts to visual regions in a shared vision–language space; the selected box supplies the subject localization for gaze prediction.Source: §11.1 · p. 15
- GroundingDINO-B [42] uses language-guided query selection in a dual-encoder, single-decoder detector, then selects the highest-confidence box for the referring phrase.Its architecture deeply fuses image and text features and refines candidate boxes with prediction scores.Source: §11.1 · p. 15
- LLMDet-L [25] uses multimodal co-training with LLM-generated captions, but removes the LLM at test time and retrieves the top-ranked text-matched region.At inference, it generates region candidates and matches them against the prompt’s text embedding.Source: §11.1 · p. 15
- OWLv2-L [49] scales OWL-ViT through self-training on over one billion weakly supervised examples and selects the best text-similarity candidate box.A Vision Transformer predicts boxes and text-alignment scores directly from image tokens.Source: §11.1 · p. 15
11.2. Gaze Model
- Gaze-following baselines take the full scene and localized person region, predict a 2D gaze heatmap, and use its peak coordinates as the gaze location.The model output is a probability distribution over image locations.Source: §11.2 · p. 15
- ViTGaze [60] uses RGB and a head box with pretrained ViT self-attention to model human–scene cues, predicting a gaze heatmap and in/out-of-frame score.It avoids additional input modalities.Source: §11.2 · p. 15
- Sharingan fuses head-crop features and normalized box coordinates into a Person Gaze Token, then uses a ViT encoder with scene tokens to predict gaze.Its outputs include a 2D heatmap and in/out-of-frame score.Source: §11.2 · p. 15
- Gaze-LLE [58] injects the head location as a positional prompt into frozen DINOv2 scene features and uses a lightweight decoder to predict gaze.It predicts a 2D heatmap and in-/out-of-frame score without traditional multi-branch head/scene architecture.Source: §11.2 · p. 15
12. Experimental Protocol
The protocol uses DigiLens ARGO glasses for real-world video and audio capture, and specifies the deployed GazeAnywhere-DINOv3-L model and its training augmentations.
12.1. AR Device for GazeAnywhere Agent
- DigiLens ARGO captures real-world video with a 48 MP camera and audio with a five-microphone beamforming array.The camera includes autofocus, optical and electronic stabilization, 4×4 pixel binning, and low-light support; the array supports spatial recordings and wearer-voice pickup in noisy environments.Source: §12.1 · pp. 15–16, Figure 8 · p. 16
12.2. Implementation Details of GazeAnywhere-DINOv3-L
- GazeAnywhere-DINOv3-L uses a three-layer, D = 256 detector transformer, jointly trains visual and text prompts, and trains at 512 × 512 resolution.Visual prompting uses bounding-box jittering, color jitter, resizing, cropping, flipping, rotation, and scene-patch masking; text prompting reorders appearance, location, pose, and action attributes, while visual augmentation is limited to scene-patch masking.Source: §12.2 · p. 16
13. More Results
The section evaluates GazeAnywhere’s encoder-freezing and detector-transformer design choices. It reports that frozen encoders perform better, detector performance plateaus at D = 128 while D = 256 offers the best accuracy–cost trade-off, and performance stabilizes with three transformer layers.
13.1. Impact of Frozen Encoder.
- Fine-tuning either the DINOv3 image encoder or text encoder reduces performance relative to keeping both encoders frozen.The authors attribute this to the robustness and generalizability of DINOv3’s pretrained features, and suggest fine-tuning may cause overfitting or harmful feature drift.Source: §13.1 · p. 16, Table 7 · p. 16
13.2. Impact of Detector Dimension.
- D = 128 marks a performance plateau, while D = 256 is selected for the best accuracy–computational-cost trade-off.Increasing the dimension beyond 128 brought no significant performance gain.Source: §13.2 · p. 16, Table 8 · p. 16
13.3. Ablation on Detector’s Transformer Layer Number
- Increasing the detector transformer’s layer count to three yields stable model performance.The layer-number ablation is reported in Table 9.Source: §13.3 · p. 16, Table 9 · p. 16
14. Qualitative Analysis
Figure 9 shows GazeAnywhere producing sharper, more accurate heatmaps around true gaze targets than Gaze-LLE. Gaze-LLE works well in sparse scenes but degrades as crowds grow, exposing weaknesses in two-stage pipelines.
14. Qualitative Analysis
- Gaze-LLE performs well with one or two people but degrades noticeably as crowd density increases.The qualitative comparison attributes this decline to unreliable upstream OVD detection in complex scenes.Source: §14 · p. 16
- In crowded scenes, OVD may localize the wrong person or produce a box covering multiple people, preventing reliable gaze conditioning and disambiguation.These failures expose a key limitation of two-stage gaze-estimation pipelines in real-world social scenes.Source: §14 · p. 16
- GazeAnywhere produces sharper, more accurate heatmaps around the true gaze targets than Gaze-LLE in the qualitative comparison.Each Figure 9 column is a sample, with methods shown in separate rows.Source: Figure 9 · p. 17
15. Related Prompts
The appendix provides the prompts used for the concept-generation data engine, MLLM-only gaze prediction, and video-based social gaze analysis with the GazeAnywhere Agent. The prompts specify how concepts are generated and checked, gaze predictions are formatted, and agent-assisted video analysis is conducted.
15. Related Prompts
- The appendix gives the prompts verbatim as used in batch API calls for reproducibility across three pipeline components.Those components are the concept-generation data engine, the MLLM-only gaze prediction baseline, and the GazeAnywhere Agent for video-based social gaze analysis.Source: §15 · p. 17
15.1. Data Engine
- The data-engine attribute prompt asks the MLLM to describe a subject person’s appearance, position, action, pose, and the number of people.The description is intended to be compact.Source: §15.1 · p. 17
- The verification prompt checks candidate concepts field by field, returns JSON flags and an overall pass/fail, and works with human spot-checks in the human-in-the-loop data engine.It checks attribute, position, action, and pose.Source: §15.1 · p. 18
15.2. MLLM Baseline
- The MLLM-only baseline prompt asks Gemini-2.5-flash and Qwen3-VL-8b to predict gaze in/out-of-frame status and a normalized 2D target point from an image and textual concept.Responses must use strict JSON format.Source: §15.2 · p. 18
15.3. GazeAnywhere Agent
- For raw-video analysis, the MLLM is prompted to infer social gaze behavior directly from an unannotated smart-glasses recording.The prompt uses the original AR recording.Source: §15 · p. 17, §15.3 · p. 18
- The GazeAnywhere Agent prompt uses the same video with overlays for the subject head box, gaze point, and out-of-frame indications, then asks for social-partner gaze-shift and overall gaze-shift counts.Source: §15 · p. 17, §15.3 · p. 18
16. Notations
This section records the prompts used to generate and verify subject concepts, predict gaze targets, and analyze gaze shifts, alongside notation for the model’s inputs, embeddings, detector, and losses.
Concept Generation Prompt
- Concept generation asks for a concise (<30-word) attribute phrase describing the person marked by a green bounding box on the head.The prompt specifies a natural attribute phrase for the target person.Source: §16 · p. 19, Figure 10 · p. 19
Concept Verification Prompt
- Concept verification checks whether a candidate description’s attribute, position, action, and pose are consistent with the boxed subject.The prompt also includes a count field in the candidate description.Source: §16 · p. 20, Figure 11 · p. 20
Gaze Target Prediction Prompts
- Gemini 2.5 Flash and Qwen3-VL prompts request an in-frame gaze flag and a normalized gaze point, with coordinates rounded or formatted to three decimals.The Gemini prompt defines image corners as (0, 0) and (1, 1); Qwen3-VL likewise normalizes image coordinates to 0.000–1.000.Source: §16 · pp. 21–22, Figure 12 · p. 21, Figure 13 · p. 22
Gaze Shift Analysis Prompts
- Both gaze-shift analysis prompts ask for analysis of a short video, while the GazeAnywhere-agent prompt overlays the detected subject and gaze target and uses blue boxes to mark out-of-frame gaze.The Single MLLM figure describes counting gaze shifts and eye-contact events; the agent prompt specifies a green box and point over the raw video.Source: §16 · pp. 22–23, Figure 14 · p. 22, Figure 15 · p. 23
Data and Embedding Notation
- The notation defines image dimensions and RGB input, prompt and text inputs, output heatmap dimensions, and visual and text embedding tokens and dimensions.It identifies [CLS] as the classification token, [EOS] as the end-of-sentence token, and distinguishes initial text embeddings from their dimension.Source: §16 · p. 24
Detector and Loss Notation
- Detector notation defines head and target-presence tokens, projected and refined visual/text sequences, head-box coordinates, gaze-point coordinates, and gaze and presence losses.The listed loss labels are gaze heatmap BCE loss and target presence focal loss.Source: §16 · pp. 24–25