PaperlayerView exact revision on arXiv

Source-linked AI summary

TransGaze-Object: Transformer Based Driver Gaze Object Prediction Framework in Real Driving

Pavan Kumar Sharma, Ayush Pande, Pranamesh Chakraborty

arXiv:2609.10139v1cs.CV

TL;DR

Existing driver-gaze methods commonly estimate zones or points of gaze, whereas object-level attention prediction from real driving data remains limited. The paper proposes TransGaze-Object, a Transformer framework trained with the UD-FSG dataset, and reports higher gaze-object accuracy than point-of-gaze association.

  • Problem

    Existing gaze representations use zones or points of gaze, while prior gaze-object work largely performs point-to-object association in simulated or controlled settings.

  • Method

    TransGaze-Object predicts gaze objects from driver-face and traffic-scene information using Transformer-based feature integration, trained on the UD-FSG real-driving dataset.

  • Results

    60% overall accuracy was achieved for gaze-object prediction, compared with 51% for associating estimated point-of-gaze with traffic objects.

  • Takeaways & Limitations

    The results support directly predicting gaze objects from driver-face and traffic-scene information rather than using an intermediate point of gaze.

Abstract

from arXiv · show

Driver gaze provides information regarding driver visual attention and situational awareness to the surrounding traffic. Existing driver gaze estimation studies represent gaze in terms of gaze zone or gaze vector/point-of-gaze (PoG). However, object-level gaze information provides a more semantically meaningful representation of visual attention by identifying attended objects, such as vehicles, pedestrians, or traffic signals. In this study, we propose an end-to-end driver gaze object prediction framework, TransGaze-Object, Transformer-based Gaze Object prediction model. The proposed framework first extracts facial features, including face and iris-weighted eye features, along with trafficobject spatial features. A transformer based cross-attention mechanism is then used to compute similarity scores and attention weights for predicting the drivers gaze object. To train this model, we propose a benchmark driver gaze dataset, Urban Driving-Face Scene Gaze (UD-FSG), comprising synchronized driver-face and traffic-scene images, scene objects bounding boxes, and gaze labels in terms of 2D gaze coordinate and gaze object. The TransGaze-Object model achieves an overall accuracy of 60% for gaze-object prediction, compared to 51% accuracy obtained from associating the estimated Point-of-Gaze to traffic objects. The error analysis reveals that TransGaze-Object reduces confusion between traffic objects (predicted) and the background (ground-truth), achieving an error rate of 11.68%, a 49.7% relative reduction compared with 23.21% error obtained from PoG-based gaze-object association. Overall, the results demonstrate the effectiveness of directly predicting gaze objects from driver-face and traffic-scene information, rather than estimating an intermediate Point-of-Gaze and subsequently associating it with traffic objects.

1. Introduction

Driver gaze estimation can be represented by gaze zones, point-of-gaze, or gaze objects, but object-level prediction is intended to identify the traffic object receiving attention. The paper introduces an end-to-end Transformer framework and the UD-FSG dataset to predict gaze objects from driver-face and traffic-scene information.

  • Human factors including visual attention and situational awareness are identified as major contributors to road crashes.
  • Object-level gaze representation addresses the limitation that gaze zones and gaze points do not directly identify attended traffic objects.
  • TransGaze-Object directly predicts the gazed object from the driver’s face image and traffic-scene object information.
  • The model integrates face features with traffic-object information through a Transformer-based cross-attention mechanism.
  • UD-FSG contains synchronized driver-face and scene images, traffic-object bounding boxes, and gaze labels expressed as 2D coordinates and gaze objects.

2. Literature review

Driver gaze research has often represented attention through zones or gaze points, while gaze-object prediction remains limited and has largely relied on controlled or simulated settings. The literature review motivates real-world, object-aware prediction using visual cues rather than post hoc gaze-point association.

  • Non-intrusive gaze estimation uses vehicle-mounted cameras to capture the driver’s face for continuous in-vehicle monitoring.
  • Zone-based methods identify broad viewing regions but do not specify which scene objects receive the driver’s attention.
  • Prior gaze-object studies map known or simulated gaze points to scene bounding boxes rather than predicting objects directly from visual cues.
  • Transformer-based gaze-object models are used in non-driving applications to model relationships between gaze-related features and multiple scene objects or regions.
  • Real-world driving data introduce conditions such as sunglasses reflections, facial glare, and low illumination that simulated settings may omit or inadequately represent.

3. Dataset

UD-FSG is a large real-driving dataset combining synchronized driver-face and traffic-scene imagery with gaze and object annotations. It captures heterogeneous traffic, varied lighting, calibrated gaze coordinates, and detected scene objects for gaze-object prediction.

  • Annotations: The dataset provides 2D gaze coordinates, detected traffic-object bounding boxes, and gaze-object labels alongside face–scene image pairs.Gaze coordinates were transformed into the fixed dashboard-camera reference frame before object assignment.
  • Dataset diversity: UD-FSG was collected in high-density heterogeneous traffic with variation in lighting, making gaze-object prediction more challenging than in lower-density benchmark settings.The scenes include diverse dynamic traffic agents such as vehicles and pedestrians.
  • Gaze labels: 37.98% of the 373,488 frame pairs contain gaze directed at a detected traffic object, while the remaining frames correspond to background gaze.Background denotes scene regions without detected object bounding boxes.
  • Dataset collection: UD-FSG contains 373,488 synchronized driver face–scene image pairs collected from 35 drivers in real urban driving conditions.Data were collected in Kanpur across varied traffic density and sunlight conditions using professional drivers.
  • Training data: A balanced training subset contains 189,850 synchronized face–scene pairs with object bounding boxes and ground-truth gaze-object labels.Background samples were subsampled because background gaze labels outnumbered gaze-on-object labels.

4. Methodology

TransGaze-Object predicts which traffic object or background the driver is gazing at by combining facial and scene-object information through a Transformer-based pipeline.

  • Problem formulation: The task predicts one of N traffic objects or the background from a driver face image and scene objects.The background is included as an additional candidate.
  • Problem formulation: Scene images are represented with a fixed number of object slots because traffic-object counts vary across images.Images with fewer objects are padded with virtual objects.
  • Overall framework: The framework detects facial geometry and scene objects, extracts features, encodes both modalities, applies cross-attention, and predicts the gaze object.These are the five major components of the proposed pipeline.

4.1. Facial geometry and object detection module

The detection module identifies the driver’s face, eyes, and irises from the face camera and traffic objects from the forward scene camera before gaze-object prediction.

  • Facial geometry detection: A custom face-eye-iris detector identifies the driver’s face, eyes, and irises from face-camera images.It was developed using pretrained YOLOv8 and trained on 2,200 annotated face images from 481 drivers.
  • Facial geometry detection: The detector achieved 95.7% mAP at IoU 0.5 and 93.0% recall on face, eye, and iris detection.The face-eye-iris detector was trained separately from the gaze-object prediction model.
  • Facial geometry detection: A validity-aware gating strategy deactivates iris representations when iris coordinates are unavailable because of occlusion, blur, head rotation, or illumination.This prevents unreliable iris information from influencing feature fusion.
  • Traffic-object detection: A YOLOv8 traffic-object detector supplies scene-object information, while missed detections are manually annotated for gaze-object training data.The traffic detector categorizes objects into 10 classes.

4.2. Feature extraction module

The feature-extraction module builds separate representations for facial cues, iris-weighted eyes, and scene-object geometry, aligning them for later multimodal attention.

  • Facial feature extraction: A pretrained ResNet-18 extracts four hierarchical facial representations, each projected and pooled into a 256-dimensional vector.The representations capture complementary information from fine-grained texture to high-level facial structure.
  • Eye feature extraction: Eye features use Layer 4 of ResNet-18 and Gaussian weighting centered on the iris to emphasize gaze-relevant regions.The weighting suppresses peripheral regions while retaining contextual eye features.
  • Multimodal facial representation: The facial representation contains six tokens: four face-feature tokens and two eye-feature tokens.The face tokens represent multiple ResNet-18 stages, while the eye tokens represent the left and right eyes.
  • Scene-object feature extraction: Scenes use 20 object slots comprising up to 19 traffic objects and one background object, with virtual padding when needed.Approximately 99.78% of frames contain 20 or fewer objects.
  • Scene-object feature extraction: Each scene object is represented by eight spatial features: center coordinates, width, height, area, center offsets, and Euclidean distance from the image center.These features encode object size and relative position, reflecting center bias in driver gaze.
  • Scene-object feature extraction: A linear projection maps spatial features into the embedding space used for interaction with facial features.The projection enables attention-based interaction and task-specific combinations of position, size, and distance.

4.3. Transformer-based encoding of face and scene features

Transformer encoders contextualize facial and scene-object features separately, after which projected representations support cross-modal similarity matching for gaze-object prediction.

  • Feature encoding: The model jointly represents multi-level facial cues, iris-weighted eye features, and scene-object spatial and geometric features.The design is based on jointly modeling facial cues and surrounding scene objects.
  • Self-attention encoding: Self-attention contextualizes facial tokens and scene objects by modeling relationships within each modality.For faces, this includes eye alignment, head orientation, and facial structure; for scenes, it includes object relationships and grouping.
  • Cross-modal matching: Projection layers transform encoded face and scene features into a task-specific space for cross-modal dot-product attention.The learned projections align gaze-relevant facial patterns with corresponding scene-object features.
  • Cross-modal matching: Attention scores function as learned similarity measures between facial gaze cues and scene objects.The cross-modal matching stage supplies the basis for selecting the predicted gaze object.

4.4. Attention between encoded facial and scene features

The model computes cross-attention between facial queries and scene-object keys to assign attention weights across detected and zero-padded objects. Eye- and face-based attention are combined using a learnable weighting parameter.

  • Facial queries and scene-object keys produce similarity scores through scaled dot-product attention.The attention score matrix contains pairwise query-key similarities, normalized by the feature dimension.
  • Zero-padded virtual objects are masked so their attention weights become zero and cannot affect the final output.Virtual objects maintain a fixed input size when fewer than 19 traffic objects are detected.
  • Temperature scaling with τ = 0.5 controls attention sharpness before the softmax operation.The scaling prevents excessively large similarity scores from producing overly peaked attention distributions.
  • A learnable parameter λ combines eye- and face-based attention, increasing the eye contribution as λ increases.The resulting attention matrix represents the model’s attention assigned to each scene object.
  • The normalized attention weights form the final object-level representation used for gaze-object prediction.The attention weights sum to one across the scene-object outputs.

4.5. Gaze estimation head

The gaze estimation head selects the scene object with the highest final attention weight and trains this prediction with classification, alignment, confusion-aware, and hard-negative objectives. Accuracy measures exact agreement between predicted and ground-truth gaze-object indices.

  • The object with the highest attention weight is selected as the predicted gaze object.The prediction is represented by the index of the highest-weight object among N + 1 outputs.
  • The model treats gaze-object estimation as multiclass classification over traffic objects and background.Classification loss supplies the primary supervision signal for assigning the highest probability to the correct object.
  • The confusion-aware loss assigns higher target probabilities to spatially nearby objects using distance-based soft labels and KL divergence.Object-center distances are converted into a probability distribution before comparing it with predicted eye attention.
  • Hard-negative margin loss separates the ground-truth object from the top-k confusing objects, using k = 2 and margin m = 0.5.The margin term is active when the ground-truth attention does not exceed the hard-negative average by the required amount.
  • The final loss combines classification, margin, consistency, and attention terms with uncertainty-based weighting for selected components.This weighting strategy avoids relying entirely on manually fixed balances during optimization.
  • Accuracy is the proportion of test samples whose predicted gaze-object index matches the ground-truth index.The metric is computed over M testing samples using an indicator of exact index agreement.

4.6. Training details

The model is trained with PyTorch on a four-GPU NVIDIA RTX 3080 server using the AdamW optimizer. The dataset is divided into training, validation, and testing sets.

  • The training, validation, and testing sets contain 165,969, 9,208, and 14,673 images, respectively.
  • Training uses PyTorch with four NVIDIA GeForce RTX 3080 GPUs and CUDA 11.4.
  • The model is optimized with AdamW using an initial learning rate of 1 × 10^-5.

5. Results

TransGaze-Object was evaluated on held-out driving data, compared with PoG-to-object association, and analyzed by error type and object scale. It achieved strong overall accuracy, with particular improvement in background-versus-traffic-object confusion.

  • Overall accuracy: 59.45% overall gaze-object prediction accuracy was achieved on 14,673 test samples, with 8,723 correct predictions.The test set covered three drivers and was separate from training and validation data.
  • Performance comparison: TransGaze-Object was compared with SGAP-Gaze by associating estimated gaze points with detected object bounding boxes.A gaze point inside a ground-truth object bounding box was assigned as the corresponding gaze object.
  • Error analysis: 11.68% of samples incorrectly predicted background when the ground truth was a traffic object, versus 23.21% for SGAP-Gaze association.For traffic-object-to-traffic-object errors, the reported rates were nearly identical: 16.23% for TransGaze-Object and 16.55% for SGAP-Gaze.
  • Error analysis: 353 overlapping traffic-object errors represented 2.40% of samples, compared with 331 errors representing 2.25% for SGAP-Gaze.These errors mainly occurred when nearby traffic objects had overlapping bounding boxes.
  • Object-scale analysis: Larger predicted bounding boxes were associated with lower normalized gaze error, with Pearson = -0.286 and Spearman = -0.622.Small objects showed significantly higher normalized error than medium and large objects.

6. Conclusions

The conclusion presents TransGaze-Object as an end-to-end framework for representing driver gaze as traffic objects or background. It combines face and scene inputs with facial, iris-weighted eye, and traffic-object features for gaze-object prediction.

  • Conclusions: TransGaze-Object directly represents driver gaze as a traffic object or the background rather than only as a gaze zone or point.The framework is positioned as an end-to-end gaze-object prediction approach.
  • Conclusions: The framework uses face and scene images to extract facial and traffic-scene features.Face-Eye-Iris and traffic objects are detected separately before feature extraction.
  • Conclusions: The proposed framework is intended to provide semantically meaningful cues for assessing driver visual attention and situational awareness.The stated application scope includes gaze-based driver monitoring systems aimed at improving driver safety.

CRediT authorship contribution statement

The authorship statement assigns contributions across conceptualization, investigation, methodology, data curation, software, validation, supervision, and writing. Sharma is credited with the broadest implementation and analysis responsibilities, while Pande and Chakraborty contribute to conception, investigation, and manuscript work.

  • Contributions: Pavan Kumar Sharma contributed to conceptualization, data curation, formal analysis, investigation, methodology, software, validation, and original-draft writing.
  • Contributions: Ayush Pande contributed to conceptualization, investigation, and writing-review and editing.
  • Contributions: Pranamesh Chakraborty contributed to conceptualization, investigation, methodology, resources, supervision, validation, and writing-review and editing.
Loading 2609.10139v1…