TransGaze-Object: Transformer Based Driver Gaze Object Prediction Framework in Real Driving

1. Introduction

• Driver gaze serves as an alternative measure of visual attention and situational awareness, with applications in driver monitoring systems and advanced driver assistance systems.Source: §1 · p. 1
• Prior gaze-estimation outputs include zones and PoG (Sharma & Chakraborty, 2024b); PoG requires object-association postprocessing, motivating direct prediction of a scene object or background from face and traffic inputs.Source: §1 · p. 2
• TransGaze-Object fuses driver-face features as queries with traffic-object bounding-box size and location as keys through cross-attention to predict the gazed-at object or background.Source: §1 · p. 2
• UD-FSG provides synchronized driver-face and scene images, object bounding boxes, and 2D gaze-coordinate and gaze-object labels collected in high-density heterogeneous urban traffic.The introduction notes that the existing open-source LBW dataset was mostly collected in low-density traffic, averaging four objects per image.Source: §1 · pp. 2–3

2. Literature review

• In practical driver monitoring, non-intrusive gaze estimation using dashboard or windshield cameras is considered more suitable than intrusive methods requiring a head-mounted eye tracker.The review consequently focuses on non-intrusive studies.Source: §2 · p. 3

2.1. Literature on gaze zone classification and point-of-gaze estimation

• Zone-based methods classify predefined regions but do not identify the scene objects attracting attention; scene-based methods instead represent gaze as direction, point, or saliency.Examples include Kasahara et al. (2022), which estimates 3D gaze direction and visual saliency, and Hu et al. (2021), whose saliency method was evaluated in a driving simulator, limiting demonstrated real-world generalizability.Source: §2.1 · p. 3

2.2. Simulated study on gaze object prediction by associating point of gaze on traffic object

• Driving gaze-object work described in the review is limited to simulated or controlled settings, where gaze points are associated with traffic objects rather than objects being predicted directly from visual cues.Dua et al. (2020) projected gaze points onto video and checked object bounding boxes; Deng et al. (2026) manually placed gaze points on BDD100K images before testing object-identification models.Source: §2.2 · pp. 3–4

2.3. Recent development in transformer based gaze estimation in non driving applications

• Transformer gaze research beyond driving includes gaze estimation and object-interaction models, with architectures combining facial, head, scene, gaze-cone, and object features.Examples include GazeTR-Pure and GazeTR-Hybrid (Cheng & Lu, 2022), Tonini et al. (2023)’s Gaze Transformer, and retail TransGOP (Wang et al., 2024).Source: §2.3 · p. 4

2.4. Summary

• Existing driving gaze-object studies mainly map a known or simulated gaze point to scene objects, and the review identifies no prior real-world model that explicitly predicts gaze objects from facial and traffic-object cues.Dua et al. (2020) use gaze-point association with object bounding boxes, while Deng et al. (2026) simulate gaze coordinates on BDD100K images; the proposed study addresses the gap with direct prediction in real-world driving.Source: §2.2 · pp. 3–4, §2.4 · p. 4

3.1.1. Sensors setup and synchronization

• The instrumented SUV carried multiple sensors, but this study used the face camera, forward-facing scene camera, and eye tracker; cameras were synchronized with GStreamer and timestamps.The eye tracker provided ground-truth gaze coordinates, while a stopwatch supplied the timestamp reference for synchronization between cameras and the eye tracker.Source: §3.1.1 · p. 5

3.1.2. Participants /Drivers

• The final dataset includes 35 male professional drivers after data from six of the 41 recruited participants were lost; mean age was 35.77 years and mean driving experience was 13.70 years.The authors note the extremely low percentage of professional female drivers in the data-collection region.Source: §3.1.2 · p. 5

3.1.3. Driver gaze ground truth creation

• Eye-tracker gaze coordinates were transformed from the moving eye-tracker scene frame to the fixed dashboard-camera frame and manually checked against the eye-tracker image.Data collection covered arterial and sub-arterial roads at different times of day; drivers traveled about 30–35 km each for approximately 1 hour, and frames were extracted at 5 fps. Gaze points were concentrated in the forward scene but also reached lateral edges (x < 200 or x > 1000 pixels).Source: §3.1.3 · pp. 5, 7

3.1.4. Gaze object ground truth creation

• Ground-truth gaze objects are assigned when a gaze point falls inside a detected object box or within 10 pixels of its nearest boundary; otherwise, the label is background.Across 373,488 frames, 141,884 (37.98%) gaze points fell on traffic objects; the remaining frames were labeled background.Source: §3.1.4 · pp. 7–8

3.1.5. Dataset details

• UD-FSG contains 373,488 synchronized driver face–scene pairs from 35 drivers and is the first heterogeneous-traffic dataset to provide detected object boxes and gaze-object labels.The authors describe it as the second dataset with synchronized face and traffic-scene images and 2D gaze labels, and as larger than LBW.Source: §3.1.5 · pp. 8–9

3.1.6. Dataset characteristics

• 176,612 frames contained 1–5 objects, approximately 152,785 contained 6–10, and approximately 99.78% of frames contained 20 or fewer traffic objects.Source: §3.1.6 · pp. 9–10, Figure 5 · p. 10

3.2. Training dataset

• 189,850 synchronized face–scene pairs form the 11-class training dataset, with 141,884 traffic-object and 47,966 background samples, split across 26/5/3 training, validation, and test drivers.The splits contain 165,969 training, 9,208 validation, and 14,673 test samples; the 10 traffic classes are pedestrian, rider, bicycle, motorcycle, auto-rickshaw, car, bus, truck, traffic sign, and traffic light.Source: §3.2 · p. 11, Table 2 · p. 11

4. Methodology

• The model predicts one of N traffic objects or the background as the driver's gaze object.The prediction is an object index, with the background included as the additional class.Source: §4 · pp. 11–12
• The scene representation contains N traffic-object slots plus background; scenes with fewer than N objects are padded with virtual objects.N is fixed based on the maximum number of traffic objects present in most of the dataset, and each object's bounding box is represented by center coordinates, width, and height.Source: §4 · pp. 11–12
• The five-stage framework detects facial geometry and scene objects, extracts facial and spatial-geometric features, encodes them, applies cross-attention, and predicts the gaze object.The overall pipeline is illustrated in Figure 8.Source: §4 · p. 12, Figure 8 · p. 12

4.1.1. Facial geometry detection

• 95.7% mAP at IoU threshold 0.5 and 93.0% recall were achieved by the face-eye-iris detector.The detector was trained on 2,200 annotated face images from 481 drivers and localizes the face, eyes, and irises.Source: §4.1.1 · pp. 12–13
• The facial geometry module returns face and eye bounding boxes and iris-center coordinates relative to each eye region.When only one iris is detected, the other location is estimated using conjugate eye movement.Source: §4.1.1 · p. 13
• When iris coordinates are unavailable, validity-aware gating deactivates the iris representation and shifts feature fusion toward remaining visual cues.The stated causes include occlusion, motion blur, large head rotations, and varying illumination.Source: §4.1.1 · p. 13

4.1.2. Tra

• 79.1% mAP at confidence threshold 0.5 (50%) was achieved by the YOLOv8 traffic-object detector trained on 10,981 scene images.It detects 10 traffic-object classes; missing category annotations were added manually to source datasets.Source: §4.1.2 · p. 13
• Undetected scene objects are manually annotated for preparation of the gaze-object prediction data.The face and scene detection outputs are illustrated in Figure 9.Source: §4.1 · p. 12, Figure 9 · p. 13, §4.1.2 · p. 14

4.2.1. Face features extraction

• ResNet-18 extracts four 256-dimensional face embeddings from progressively deeper layers, using 1 × 1 projections and global average pooling to capture complementary facial information.The face input is resized to 224×224 before passing through the convolutional stem and four residual stages.Source: §4.2.1 · p. 14, Figure 10 · p. 15

4.2.2. Gaussian weighted eye feature extraction

• For eye features, ResNet-18 layer 4 processes resized, gray-padded eye crops; Gaussian weighting centered on the detected iris emphasizes nearby features, with σ = 1.2.Padding uses intensity 114 to preserve aspect ratio and reduce artificial boundary effects; the Gaussian-weighted representation retains contextual eye information.Source: §4.2.2 · pp. 14–16, Figure 11 · p. 16
• The facial token set combines four 256-dimensional face embeddings with separate 256-dimensional left- and right-eye embeddings, yielding six facial tokens.Face embeddings represent local and global facial geometry; eye embeddings are global features extracted from layer 4.Source: §4.2.2 · p. 16

4.2.3. Scene object spatial features

• Each scene contains 20 object slots: at most 19 traffic objects plus background, retaining the 19 largest by area when more are detected.This fixed set reflects the finding that approximately 99.78% of frames contain 20 or fewer objects.Source: §4.2.3 · p. 16
• Each object is represented by eight spatial attributes—center, width, height, area, center offsets, and distance—then projected to align with facial-feature embeddings.The background receives a fixed out-of-range spatial representation to distinguish it from traffic-object boxes; spatial features also encode size and center-relative position.Source: §4.2.3 · p. 17

4.3. Transformer-based encoding of face and scene features

• Separate ResNet-18 networks extract face and iris-weighted eye features, while an MLP maps scene-object spatial and geometric features into the compatible embedding dimension.The model represents facial and scene features independently before attention-based fusion.Source: §4.3 · p. 17
• Encoded facial features serve as queries and scene-object features as keys; learned linear projections place them in a task-specific space for dot-product attention matching.The learned similarity is intended to align gaze-relevant facial patterns with objects, including visually similar or spatially proximate candidates.Source: §4.3 · pp. 17, 19, 20
• Two separate Transformer encoders use self-attention to contextualize facial regions and scene objects, capturing relationships that independent raw tokens would miss.Facial tokens can incorporate cues across four face and two eye tokens; scene tokens capture spatial relationships, grouping, and relative importance.Source: §4.3 · pp. 18–19

4.4. Attention between encoded facial and scene features

• Scaled dot-product attention computes similarity scores between facial-feature queries and scene-object keys, forming the attention score matrix.Each score is the dot product of a query and key, scaled by the square root of the feature dimension.Source: §4.4 · p. 20
• Zero-padded virtual objects receive −∞ scores before softmax, yielding zero attention weights so padded detections do not contribute to the attention output.Source: §4.4 · p. 20
• With τ = 0.5, temperature scaling is applied before softmax to control attention sharpness and avoid overly peaked weights that can destabilize gradients and generalization.Source: §4.4 · p. 20
• Separate face and eye attention captures coarse head-orientation and precise iris cues, then learnable λ combines their object weights, with larger λ favoring eyes.Eye cues can be sensitive to noise and occlusion, whereas face features provide more stable but coarser gaze cues.Source: §4.4 · pp. 20–21

4.5.1. Gaze object prediction

• The predicted gaze object is the index of the scene object with the highest computed attention weight.The prediction is selected from N traffic objects and background.Source: §4.5.1 · p. 21

4.5.2. Loss functions

• Training combines classification, consistency, confusion-aware attention, and hard-negative margin losses to supervise gaze-object prediction and distinguish competing objects.Classification uses cross-entropy; consistency encourages agreement between eye- and face-based attention. Confusion-aware supervision uses spatially weighted soft targets and KL divergence, while the margin loss separates the ground-truth object from the top two hard negatives by a minimum margin of 0.5.Source: §4.5.2 · pp. 21–22
• Learnable uncertainty parameters weight the consistency and confusion-aware attention losses, with regularization to prevent their weights from collapsing toward zero.The loss description sets α = 0.03 for consistency and uses uncertainty terms σ1 and σ2 to balance the two losses during optimization.Source: §4.5.2 · pp. 21–23

4.5.3. Evaluation metric

• Accuracy is the proportion of test samples whose predicted gaze-object index matches the ground-truth index.It is computed across M test samples using an indicator for whether each predicted index equals its ground-truth index.Source: §4.5.3 · p. 23

4.6. Training details

• The training, validation, and testing sets contain 165,969, 9,208, and 14,673 images, respectively.Dataset details are referenced in Table 2.Source: §4.6 · p. 23
• Training uses AdamW with an initial learning rate of 1 × 10−5, weight decay, and cosine annealing to gradually reduce the learning rate.The scheduler is used to support stable convergence and weight decay to improve generalization.Source: §4.6 · p. 23
• The model is trained in minibatches of 64 with gradient clipping at a maximum norm of 1.0 on four NVIDIA GeForce RTX 3080 GPUs.The GPU server has 10 GB VRAM per GPU and uses CUDA 11.4; training is implemented in PyTorch.Source: §4.6 · p. 23

5.1. Overall accuracy

• 59.45% accuracy: TransGaze-Object correctly predicted 8,723 of 14,673 test samples across three drivers.The test samples were completely separate from the training and validation data.Source: §5.1 · p. 23

5.2. Performance comparison of TransGaze-Object and SGAP-Gaze

• 59.45% vs 50.98% accuracy: TransGaze-Object improved on SGAP-Gaze PoG-to-object association by 8.47% on the same test set (Sharma & Chakraborty, 2026).The comparison associates estimated gaze points with detected object bounding boxes; the authors note that TransGaze-Object still correctly predicts only about 60% of gaze objects.Source: §5.2 · p. 24

5.3. Error analysis of gaze object predictions

• 11.68% vs 23.21%: TransGaze-Object made fewer background predictions when the ground-truth gaze object was traffic, compared with SGAP-Gaze PoG-to-object association.For traffic-object-to-traffic-object errors, rates were 16.23% for TransGaze-Object and 16.55% for SGAP-Gaze; for background ground truth but traffic-object predictions, rates were 12.62% and 9.24%, respectively. The authors attribute the overall improvement primarily to fewer traffic-object-as-background errors.Source: §5.3 · pp. 24–25, Table 3 · p. 25
• 353 (2.40%) vs 331 (2.25%): overlapping traffic-object errors were slightly higher for TransGaze-Object than SGAP-Gaze, while non-overlapping errors were 2029 (13.82%) vs 2098 (14.29%).Overlapping-box errors often occurred when nearby objects had overlapping boxes; overall, traffic-object-to-traffic-object error rates were comparable between methods.Source: §5.3 · p. 25, Table 4 · p. 27

5.4. Error analysis of background and traffic object

• Pearson = -0.286 and Spearman = -0.622: bounding-box area was negatively correlated with normalized gaze error, and small objects had higher error than medium and large objects.Normalized error was the shortest distance from the ground-truth point of gaze to the predicted box edge divided by the square root of box area; size groups were quantile-based thirds.Source: §5.4 · p. 27

6. Conclusions

• TransGaze-Object predicts traffic objects or background directly from face and scene images using detected face, iris-weighted eye, and object features with transformer cross-attention.The UD-FSG dataset provides synchronized driver-face and traffic-scene images, scene-object bounding boxes, and 2D gaze-coordinate and gaze-object labels; training uses classification, consistency, confusion-aware attention, and hard-negative margin losses.Source: §6 · p. 28
• 60% overall gaze-object prediction accuracy versus 51% for PoG-based association; background-confusion error was 11.68% versus 23.21%, and accuracy improved 17.5% relative to SGAP-Gaze.The authors note that distinguishing traffic objects from the background and predicting gaze on smaller objects remain challenging.Source: §6 · p. 28
• Future work proposes improving accuracy, robustness, and generalization, incorporating camera parameters and placements, expanding beyond the forward scene, and modeling gaze across consecutive frames.The proposed scope includes side windows and more of the vehicle interior, with temporal modeling intended to stabilize predictions in challenging real-world driving conditions.Source: §6 · p. 28

• Pavan Kumar Sharma contributed to Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Software, Validation, and Writing-original draft.Source: CRediT authorship contribution statement · p. 28
• Ayush Pande contributed to Conceptualization, Investigation, and Writing-Review & Editing.Source: CRediT authorship contribution statement · p. 28
• Pranamesh Chakraborty contributed to Conceptualization, Investigation, Methodology, Resources, Supervision, Validation, and Writing-Review & Editing.Source: CRediT authorship contribution statement · p. 28