PaperlayerView exact revision on arXiv

Source-linked AI summary

Gaze Target Estimation Anywhere with Concepts

Xu Cao, Houze Yang, Vipin Gunda, Zhongyi Zhou, Tianyu Xu, Adarsh Kowdle, Inki Kim, James M. Rehg

arXiv:2608.11367v1cs.CVcs.AI

TL;DR

Gaze target estimation in-the-wild often depends on brittle pipelines and explicit subject inputs. This paper introduces promptable, end-to-end gaze estimation with Gaze-Co and GazeAnywhere, achieving state-of-the-art results across benchmarks and on a challenging out-of-domain dataset.

  • Problem

    In-the-wild gaze estimation typically requires explicit subject inputs and multi-stage pipelines whose upstream errors can cascade into system failure.

  • Method

    The paper defines Promptable Gaze Target Estimation, builds the 120K-pair Gaze-Co benchmark, and introduces GazeAnywhere for text- or coordinate-prompted end-to-end gaze estimation.

  • Results

    GazeAnywhere achieves state-of-the-art performance across several PGE benchmarks, including a challenging out-of-domain dataset of child social communication videos.

  • Takeaways & Limitations

    The concept-based, end-to-end approach supports flexible subject specification and is presented as a step toward robust, usable gaze assessment for nonexperts.

  • Takeaways & Limitations

    The paper’s qualitative analysis highlights that two-stage pipelines can fail in crowded scenes by selecting the wrong person or grouping multiple people together.

Abstract

from arXiv · show

Estimating human gaze targets from images in-the-wild is an important and formidable task. Existing approaches primarily employ brittle, multi-stage pipelines that require explicit inputs, like head bounding boxes and human pose, in order to identify the subject of gaze analysis. As a result, detection errors can cascade and lead to failure. Moreover, these prior works lack the flexibility of specifying the gaze analysis task via natural language prompting, an approach which has been shown to have significant benefits in convenience and scalability for other image analysis tasks. To overcome these limitations, we introduce the Promptable Gaze Target Estimation (PGE) task, a new end-to-end, concept-driven paradigm for gaze analysis. PGE conditions gaze prediction on flexible user text or visual prompts (e.g., "the boy in the red shirt" or "person in point [0.52, 0.48]") to identify a specific subject for gaze analysis. This approach integrates subject localization with gaze estimation, and eliminates the rigid dependency on intermediate analysis stages. We develop a scalable data engine to generate Gaze-Co (Gaze Estimation with Concepts), a dataset and benchmark of 120K high-quality, prompt-annotated image pairs. We also propose GazeAnywhere, the first model designed for PGE. GazeAnywhere uses a transformer-based detector to fuse features from frozen encoders and simultaneously solves subject localization, in/out-of-frame presence, and gaze target heatmap estimation. GazeAnywhere achieves state-of-the-art performance on multiple PGE benchmarks, setting a strong baseline for this new problem even on a difficult out-of-domain, real-world clinical dataset. GazeAnywhere is open-sourced in github.com/IrohXu/GazeAnywhere.

1 University of Illinois Urbana-Champaign 2 Google · 1. Introduction · 2. Related Work

The paper introduces Promptable Gaze Target Estimation (PGE), an end-to-end paradigm that uses text or visual concepts to identify a gaze-analysis subject while estimating the gaze target. It addresses failures caused by auxiliary-input dependencies, contributes 120K annotated pairs and GazeAnywhere, and reports state-of-the-art benchmark performance.

  • 1 University of Illinois Urbana-Champaign 2 Google: The work is affiliated with the University of Illinois Urbana-Champaign and Google.The listed contact is {xucao2,jrehg}@illinois.edu.
  • 1. Introduction: Human gaze conveys social and cognitive information and supports interaction, attention, turn-taking, intimacy regulation, and inference of intentions, preferences, and emotions.Its study spans psychology, human-computer interaction, and clinical research including autism spectrum disorder.
  • 1. Introduction: In-the-wild gaze-target estimation is difficult because current methods require explicit head, face, pose, and depth information in sequential processing stages.Initial-stage errors can arise in crowded scenes, poor lighting, and challenging cases such as detecting children’s faces.
  • 1. Introduction: These sequential dependencies can cascade, causing inaccuracies in initial analysis stages to fail the entire gaze-estimation system.The paper identifies this cascading failure as a critical bottleneck.
  • 1. Introduction: The proposed solution is an end-to-end, concept-based framework inspired by semantic text- and visual-prompt capabilities in open-vocabulary detectors and Segment Anything models.The framework is intended to improve convenience and flexibility in specifying gaze-analysis tasks.
  • 1. Introduction: The paper defines PGE, builds a scalable engine producing 120K high-quality prompt-annotated training pairs, and introduces GazeAnywhere as the first promptable concept-driven gaze-target model.PGE extends gaze-target estimation to an unconstrained, text-promptable end-to-end paradigm.
  • 1. Introduction: GazeAnywhere achieves state-of-the-art performance across several benchmarks, including an out-of-domain private dataset for autism children’s gaze-target estimation.The result establishes the model as a baseline for the proposed promptable task.
  • 2. Related Work: Prior gaze-following research includes GazeFollow, VideoAttentionTarget, GOO, ChildPlay, multi-view gaze-target estimation, and GazeHOI, but commonly depends on precise head boxes or pose estimates.Promptable and open-vocabulary perception methods motivate removing these unreliable or unavailable auxiliary priors in realistic settings.

3. Method

The method defines Promptable Gaze Target Estimation as end-to-end gaze heatmap prediction for a user-specified subject identified by text or visual prompts. GazeAnywhere fuses frozen visual and text features with a trainable transformer detector and jointly predicts gaze, subject localization, and gaze-target presence.

  • PGE Task: PGE predicts a gaze heatmap for a specific subject identified by either a natural-language query or a coordinate-based visual cue.Each heatmap element represents the probability that the prompted subject gazes at a spatial location, without requiring auxiliary inputs common to classic gaze estimation.
  • Prompt Design: Text prompts combine appearance, location, and other descriptive categories to reduce ambiguity when identifying the target subject.Appearance descriptions can include identity and modifiers such as clothing, hair, glasses, or hats, while location specifies spatial position.
  • Architecture: GazeAnywhere uses frozen image and text encoders, trainable projection layers, and a transformer-based detector to fuse multimodal features with specialized task tokens.The projections map visual and textual features into a shared dimension, while head and target-presence tokens model localization and gaze-presence subproblems.
  • Decoders: Three prediction heads produce the gaze heatmap, prompted-subject head box, and target-presence prediction from refined detector tokens.The gaze tracker reassembles visual tokens into a spatial grid and upsamples them with two transposed convolutional layers to a 64 × 64 heatmap.
  • Training Objective: The model is trained end-to-end with a weighted sum of gaze heatmap, gaze presence, and auxiliary head-localization losses.The heatmap uses pixel-wise BCE with a 2D Gaussian target, presence uses focal loss, and head localization combines L1 and generalized IoU losses.

4. Gaze with Concept (Gaze-Co) Dataset

Gaze-Co is a large-scale promptable gaze dataset created through a scalable human-in-the-loop data engine. It unifies three source datasets with concept annotations, standardized gaze records, quality filtering, and concept-conditioned benchmark settings.

  • Data Engine: The data engine aligns heterogeneous annotations, filters low-quality frames, generates concise concept phrases, and supports human feedback.Its workflow comprises data alignment and filtering, concept generation, and verification.
  • Annotation Schema: Annotations use explicit pixel-coordinate head boxes, normalized gaze points, and concepts covering attributes, position, action, and pose.Concept generation uses Gemini 2.5 Pro with fixed prompts and prioritizes stable visual cues; indeterminate fields are recorded as “none.”
  • Dataset: Gaze-Co contains about 120K training images from GazeFollow, VAT, and ChildPlay, making it the first large-scale dataset for promptable gaze target estimation.Each record includes the target head box, normalized gaze point, in/out-of-frame label, and compact concept phrase, with image-quality filtering across varied viewpoints, poses, scales, and interactions.
  • Benchmark Settings: The benchmark converts official test splits of GazeFollow, VAT, and ChildPlay into the Gaze-Co format for concept-conditioned gaze prediction.The Child-SC dataset is an IRB-approved out-of-domain set used for evaluation.
  • Evaluation: Evaluation reports heatmap AUC for GazeFollow and pixelwise L2 across datasets, plus average precision for localization and in/out classification where applicable.Pixelwise L2 measures Euclidean distance between the heatmap peak and ground-truth gaze point; AP is additionally reported for VAT-Concept, ChildPlay-Concept, and Child-SC.

5. Experiments

Experiments show that GazeAnywhere achieves state-of-the-art promptable gaze estimation across public and challenging real-world datasets, outperforming general-purpose VLMs and benefiting from DINOv3-based encoders. Its qualitative robustness and AR-based GazeAnywhere Agent further demonstrate applicability to crowded scenes and real-world gaze analysis.

  • PGE benchmarks: GazeAnywhere achieves SOTA performance on all metrics across three public PGE datasets and a challenging real-world clinical OOD dataset.The OOD dataset quantifies children’s social communication skills through expert assessment.
  • VLM comparison: GazeAnywhere surpasses Qwen3-VL-8B and Gemini 2.5 Flash in zero-shot gaze point prediction, supporting a PGE-specific model design.The comparison evaluates general-purpose VLMs against GazeAnywhere on gaze point prediction.
  • Encoder ablation: DINOv3 achieves the best performance on nearly all metrics among the evaluated frozen encoder backbones.The ablation compares CLIP, SigLIP 2, MetaCLIP 2, and DINOv3, fine-tuning only the projection layer, transformer detector, and decoder heads.
  • Qualitative analysis: Qualitative results show robust gaze estimation from appearance-only text prompts in both simple scenes with 2–3 people and crowded scenes with four or more individuals.The model detects the prompted subject’s head and tracks the gaze target.
  • Real-world application: The GazeAnywhere Agent uses an AR glass, Gemini 2.5, Whisper v3, and GazeAnywhere to analyze gaze shifts and eye contact, outperforming a raw single-MLLM solution in MAE per minute.The evaluation used 10 real-world videos recorded with DigiLens ARGO glasses.

6. Discussion

Gaze target estimation has diverse applications, including healthcare analysis of non-verbal communication relevant to developmental conditions such as autism. The concept-based approach advances clinical usability by supporting robust, nonexpert-friendly subject identification through natural-language attributes.

  • Clinical applications: Gaze target estimation can support healthcare analysis of non-verbal communication behaviors implicated in diagnosing and treating developmental conditions such as autism.The passage identifies healthcare as one practical application of gaze target estimation.
  • Clinical applications: Clinical AI applications require models that are sufficiently robust and easy for nonexperts to use.The discussion frames robustness and usability as prerequisites for clinical deployment.
  • Clinical applications: The concept-based approach identifies gaze-analysis subjects by their attributes in natural language, advancing usability for gaze assessment.This approach is presented as a significant step toward practical clinical use of gaze assessment.

7. Conclusion

GazeAnywhere enables interactive human gaze target estimation with flexible open-vocabulary text prompts. The paper introduces PGE and Gaze-Co, a tailored transformer-based detector and learning objective, and a human-and-AI-in-the-loop data engine.

  • Conclusion: GazeAnywhere enables interactive gaze target estimation by using flexible, open-vocabulary text prompts to identify the subject.The system is designed for interactive human gaze target estimation.
  • Conclusion: The work introduces the Promptable Gaze Target Estimation task and the Gaze-Co benchmark.These are presented as principal contributions of the paper.
  • Conclusion: The method proposes a tailored transformer-based detector and learning objective for promptable gaze estimation.The detector and objective are listed among the paper’s principal contributions.
  • Conclusion: A human-and-AI-in-the-loop data engine adapts existing datasets for the proposed gaze estimation framework.The data engine is another principal contribution identified in the conclusion.

Supplementary Material … 10. Dataset & Benchmark

The supplementary material motivates concept-driven, end-to-end gaze analysis, presents an agent for natural-language requests, and details the Gaze-Co training and benchmark datasets. These resources span concept-annotated in-domain splits and a private clinical out-of-domain evaluation set.

  • 8.1. Toward End-to-end Gaze Target Estimation: Prior gaze estimators increasingly use streamlined head-box inputs and end-to-end bottom-up detection, but still lack identity association for selecting a specific gaze subject.This limitation motivates promptable gaze target estimation, which identifies the subject through flexible concepts rather than relying only on intermediate detections.
  • 8.2. Future Application: Concept-based prompting may support earlier screening of atypical joint attention behaviors associated with Autism Spectrum Disorder.The passage connects delayed joint attention in children with ASD to the potential of gaze target estimation models for screening and detection.
  • 9. GazeAnywhere Agent: GazeAnywhere Agent combines an MLLM planner with Whisper-large-v3 and GazeAnywhere to process natural-language gaze estimation and post-analysis requests.The initial agent uses Whisper-large-v3 for audio-to-text conversion and GazeAnywhere for promptable gaze target prediction.
  • 9. GazeAnywhere Agent: The agent converts audio to text, plans tool calls from scene context, visualizes gaze targets as feedback, and stores that feedback to revise subsequent analysis steps.This feedback loop enables the agent to determine the next actions after each invocation of GazeAnywhere.
  • 10.1. Training Set: 119,525 samples comprise the Gaze-Co training set, with each record containing a target head box, normalized gaze point, in/out-of-frame label, and compact concept phrase.The records are assembled from three published gaze datasets using image-quality filtering, MLLM concept generation, and human-in-the-loop verification.
  • 10.1. Training Set: 60,983 samples (51.0%) are labeled man, while 16,671 samples (13.9%) contain out-of-frame gaze annotations.The apparent subject categories represent perceived visual categories rather than verified identity attributes; 102,854 samples (86.1%) are in-frame.
  • 10.2. Concept-based In-domain Test Set: Three human-verified concept-augmented test splits—GazeFollow-Concept, VAT-Concept, and ChildPlay-Concept—convert official test sets into a unified PGE schema after image-quality filtering.The schema includes an image, head box, normalized gaze point, in/out-of-frame label, and concept phrase.
  • 10.3. Concept-based Out-of-domain Test set: Child-SC provides out-of-domain evaluation through 326 video clips from 40 children, sampled at 5 fps to produce 151,533 images of child–clinician interactions.The private dataset is IRB-protected and captures structured gaze shifts elicited by clinicians using toys; privacy regulations prevent cloud processing.

11. Baseline Details

The baselines use open-vocabulary detectors to localize a text-prompted target person, then apply gaze-following models to the full scene and localized region to predict a gaze heatmap and final gaze location.

  • Subject localization: Open-vocabulary detectors localize the prompted person by ranking candidate boxes using text–image similarity or semantic alignment.The highest-confidence or best-matching box becomes the subject-person localization for gaze prediction.
  • Subject localization: GroundingDINO-B uses language-guided query selection and iterative box refinement to identify the highest-confidence box for the referring phrase.Its dual-encoder single-decoder Transformer deeply fuses image and text features.
  • Subject localization: LLMDet-L uses multimodal co-training with caption generation to enrich feature alignment, then ranks image regions against the prompt embedding at test time.The large language model is removed during testing, leaving the detector to retrieve the top-ranked localized subject.
  • Subject localization: OWLv2-L scales OWL-ViT through self-training on over one billion weakly supervised examples and selects the candidate with the highest text-alignment score.A Vision Transformer predicts bounding boxes and text-alignment scores directly from image tokens.
  • Gaze following: Gaze-following models consume the full scene and localized person region, output a 2D gaze heatmap, and use its peak coordinates as the predicted gaze location.ViTGaze uses RGB-only human–scene interaction cues, Sharingan fuses head features with normalized coordinates into a Person Gaze Token, and Gaze-LLE injects head position into frozen DINOv2 scene features.

12. Experimental Protocol · 13. More Results

The experimental protocol uses DigiLens ARGO for real-world video, audio, and on-device feedback, while GazeAnywhere-DINOv3-L jointly trains visual and text prompting with a compact detector transformer. Ablations support frozen encoders, detector dimension D = 256, and three detector layers as effective design choices.

  • 12.1. AR Device for GazeAnywhere Agent: ARGO’s five-microphone beamforming array records wearer speech spatially in noisy environments.The array is designed to isolate the wearer’s voice and provide recordings suitable for analysis.
  • 12.2. Implementation Details of GazeAnywhere-DINOv3-L: GazeAnywhere-DINOv3-L uses a three-layer detector transformer with dimension D = 256 and jointly trains visual and text prompts.Visual prompting uses bounding-box, color, geometric, resizing, cropping, flipping, rotation, and masking augmentations; text prompting has fixed subject-position information.
  • 13.1. Impact of Frozen Encoder: Freezing both image and text encoders outperforms fine-tuning either encoder.The ablation attributes this to the robustness and generalizability of DINOv3’s pre-trained features, while fine-tuning can cause overfitting or harmful feature drift.
  • 13.2. Impact of Detector Dimension: Detector performance plateaus at D = 128, but D = 256 provides the best accuracy–computational-cost trade-off.Increasing the dimension beyond D = 128 produced no significant performance gain.
  • 13.3. Ablation on Detector’s Transformer Layer Number: The detector uses three transformer layers because performance becomes stable after increasing the layer number to 3.This layer-count ablation is reported in Table 9.

14. Qualitative Analysis

The qualitative analysis compares GazeAnywhere with Gaze-LLE under appearance prompts and shows stronger robustness in crowded scenes. Gaze-LLE degrades with crowd density because its upstream localization can select the wrong person or produce unreliable outputs, whereas GazeAnywhere yields sharper, more accurate heatmaps.

  • Qualitative comparison: Figure 9 qualitatively compares GazeAnywhere and Gaze-LLE for gaze-target localization conditioned on appearance prompts.Each column represents a sample, and each row represents a method.
  • Crowded-scene robustness: Gaze-LLE performs well in sparse scenes with one or two individuals but degrades noticeably as crowd density increases.The degradation is associated with increasingly complex multi-person settings.
  • Crowded-scene robustness: In complex crowded scenes, Gaze-LLE’s upstream OVD module becomes unreliable and may localize the wrong person, causing gaze estimation for an incorrect target.This exposes the vulnerability of its upstream localization stage.
  • Qualitative comparison: GazeAnywhere produces sharper and more accurate heatmaps around the true gaze targets than the compared method.The comparison is shown across samples with appearance-based prompts.

15. Related Prompts

This section records the verbatim natural-language prompts used across the concept-generation data engine, MLLM-only gaze prediction baselines, and GazeAnywhere Agent video analysis. The prompts define concept annotation, structured gaze prediction, and social-gaze analysis with or without model overlays.

  • 15. Related Prompts: The appendix provides exact prompts used in batch API calls for reproducibility across three gaze-analysis components.These components are the concept-generation data engine, MLLM-only gaze prediction baseline, and GazeAnywhere Agent.
  • 15.1. Data Engine: The attribute prompt asks the MLLM to describe each marked person’s appearance, position, action, pose, and people count.It is used to construct concept-level annotations for subject persons.
  • 15.1. Data Engine: The concept verification prompt checks candidate concepts field by field and returns JSON flags plus an overall pass/fail decision.The checked fields are attribute, position, action, and pose, with human-annotator spot-checks completing the human-in-the-loop data engine.
  • 15.2. MLLM Baseline: The baseline prompt requests an in/out-frame flag and normalized 2D gaze target point in strict JSON format.The model receives an image and a textual concept description.
  • 15.3. GazeAnywhere Agent: For smart-glasses recordings, the raw-video prompt asks an MLLM to infer social gaze behavior directly from unannotated AR video.This supports comparison between an MLLM alone and an MLLM assisted by the GazeAnywhere Agent.
  • 15.3. GazeAnywhere Agent: The GazeAnywhere-agent prompt adds subject head boxes, gaze points, and out-of-frame indications as overlays on the same video.It instructs the model to count gaze shifts to social partners and overall gaze shifts.

16. Notations

This section defines the prompts used for concept generation, verification, gaze prediction, and gaze-shift analysis, then specifies notation for inputs, encodings, detector variables, and losses. It also establishes normalized image coordinates and core gaze, localization, and presence symbols.

  • Prompt definitions: Concept generation describes a green-boxed person using a concise natural attribute phrase under 30 words.The prompt targets the person with a green bounding box on the head.
  • Prompt definitions: Concept verification checks candidate descriptions against subject attributes, position, action, pose, and count.The verification prompt uses a green-boxed human head and a candidate description from concept generation.
  • Gaze prediction prompts: Gemini 2.5 Flash and Qwen3-VL prompts represent gaze points in normalized image coordinates from the top-left (0, 0) to bottom-right (1, 1).Qwen3-VL rounds normalized pixel values to three decimal places, while Gemini uses image width and height for normalization.
  • Gaze-shift analysis: Gaze-shift prompts analyze videos by identifying gaze shifts and eye-contact events using subject boxes, out-of-frame color changes, and gaze points.For GazeAnywhere, green marks the detected child, blue indicates looking out of frame, and a green point marks the gaze target.
  • Mathematical notation: Notation covers RGB inputs, prompts, text, gaze heatmaps, visual and text embeddings, detector tokens, refined sequences, head boxes, and loss components.Symbols include I ∈R3×H×W, ˆH ∈RHout×Wout, projected tokens, target-presence variables, and head-box and gaze-loss terms.
Loading 2608.11367v1…